🧠 Cognitive Memory Evaluation & Benchmarks¶

As formalized in the Memory Fundamentals Specification (MF-001), a database returns what was written; a memory engine reconstructs what is reachable from a cue at this moment — under decay, association, and tier physics — without losing a live trace because the first index was the wrong one.

To rigorously validate how Spector's closed recall algebra, 6-phase scoring pathway, off-heap storage substrate, and multi-layer associative graphs perform under lifelong conversational demands without succumbing to the truncation trap, we evaluate Spector across three standard and longitudinal benchmarks:

1. MindSpan — 20-Year Longitudinal Cognitive Memory Benchmark (19,512 records, 4-generation kinship tree, 17 cognitive tracks)
2. LongMemEval — Long-Horizon Multi-Session Conversational Needle-in-a-Haystack (10,866 turns, 500 queries)
3. LoCoMo — Multi-Turn Conversational Dialogue Memory with Grounding (5,882 turns, 1,986 queries)

🏆 Headline Benchmark Leaderboard¶

1. MindSpan: 20-Year Longitudinal Cognitive Benchmark¶

MindSpan is a 20-year longitudinal cognitive benchmark designed to evaluate AI memory systems against the full depth of human autobiographical memory, lifelong temporal decay, and multi-hop kinship reasoning. It is maintained in the spector-datasets repository under mindspan/.

1.1. Why MindSpan? The Need for True Lifelong Evaluation¶

Academic chat benchmarks like LongMemEval test short-term buffer retrieval under scraped internet noise: their target facts average only 16.4 days old, 88% are under a month old, and sessions are artificially compressed into 16 sessions per calendar day.

MindSpan evaluates true lifelong episodic and semantic recall:

• Can an agent recall a 20-year-old childhood memory subjected to power-law decay?
• Can it disambiguate between soccer practices in May 2024 vs. September 2025?
• Can it traverse a 4-generation family tree to deduce that an antique heirloom was inherited through marriage?

1.2. The Persona: Mike Thompson¶

The entire corpus follows a single, longitudinal persona:

• Mike Thompson: 36→38 years old (2024–2026), Principal Product Manager at Vertex Health in Frisco, Texas.
• Family: Wife Sarah (Lead UX Designer), son Ethan (8→10), daughter Lily (3→5), Golden Retriever Cooper.
• AI Companion: Jarvis (powered by Spector Memory).

1.3. Corpus Scale & Temporal Structure¶

timeline
    title MindSpan 20-Year Longitudinal Memory Architecture
    1990 - 2023 : Autobiographical Foundations : ~468 Historical Milestones : Childhood, UIUC college dorm, meeting Sarah in Chicago (2011), wedding at Lake Geneva (2016), Ethan's birth (2017), family heirlooms
    2024 - 2026 : Continuous Daily AI Companion : 1,095 Continuous Days (4 sessions/day) : Morning calendar briefing (7:30 AM), Midday task check-in (12:30 PM), Evening family debrief (6:45 PM), Night journal (10:15 PM)

1.4. The 17 Cognitive Reasoning Tracks¶

MindSpan queries test 17 specialized cognitive reasoning tracks. Spector Memory achieved 100% QA accuracy across all 17 tracks:

1.5. MindSpan Empirical Results¶

Evaluated with natural-ingest-v1 on Google gemini-3.1-flash-lite.

1.6. Head-to-Head: MindSpan vs. LongMemEval¶

2. LongMemEval Benchmark¶

LongMemEval (ICLR 2025 / UCLA & Tencent AI Lab) is the standard academic benchmark for multi-session conversational memory, long-horizon state tracking, and needle-in-a-haystack retrieval across 10,866 turns and 500 evaluation questions.

2.1. Empirical Accuracy & Cognitive Breakdown¶

On the full LongMemEval suite, Spector Memory achieved 91.0% – 94.0% Overall Benchmark Accuracy while injecting an ultra-compact context of only ~1,500 tokens per query (an 88% reduction in LLM prompt token consumption).

2.2. The 1,500-Token Context Advantage¶

Dumping dozens of conversational turns (12,000+ tokens) into large context windows causes the well-documented "Lost in the Middle" failure mode, where dates, names, and numbers are drowned in conversational chatter.

Empirical citation analysis showed that 91.9% of all winning evidence resides in the Top 15 retrieved candidates. By focusing synthesis on these Top-15 high-confidence candidates:

• LLM prompt context is constrained to ~1,500 tokens.
• Token consumption and API billing drop by ~88%.
• Hallucinations from distractor noise are virtually eliminated.
• Retrieval executes in 3.0 ms \(p_{50}\) (with 0.13 ms raw SIMD core; 92% of queries finish in \(\le 5\text{ ms}\)).

2.3. Single-Persona Longitudinal Slice (Sam Okonkwo)¶

When evaluated on an isolated single-persona history (Sam Okonkwo: 51 sessions, 514 turns, 10 queries) with Google gemini-embedding-001 and zero cross-tenant contamination:

3. LoCoMo Benchmark¶

LoCoMo (Long-Context Memory for Multi-Turn Dialogues, ACL 2024) evaluates agentic memory systems across 5,882 turns and 1,986 questions from multi-party conversational dialogues.

3.1. Headline Accuracy & Baseline Comparison¶

Spector Memory achieves 83.42% – 85.0% Overall Accuracy (J-Score) on LoCoMo, delivering higher recall precision than competing graph/vector architectures while injecting 3.1× fewer context tokens and running >50× faster:

3.2. Cognitive Subsystem Reasoning Breakdown¶

Spector's multi-layer cognitive architecture dynamically activates specialized subsystems based on the reasoning demands of the dialogue:

3.3. Raw Information Retrieval (IR) Metrics Under Warmup¶

Evaluated with cold-start warmup pass to eliminate JIT compilation skew:

4. Large-Corpus Synthetic Stress Tests¶

4.1. Balanced-Baseline (50,041 Records / 517 Queries)¶

Stress-testing the off-heap engine against 50,000 synthetic life-history records and noise distractors:

• Zero GC Overhead: 50K off-heap memory segments scanned on a single core in \(p_{50} = \mathbf{53.38\text{ ms}}\) and \(p_{99} = \mathbf{58.51\text{ ms}}\).
• Throughput: Sustained 20.8 QPS single-threaded with zero heap allocation.

4.2. Interest-Diversified 365-Day Evaluation¶

• Persona Context: Mike Thompson with enriched hobby graphs (astrophotography, local LLM hacking, smart home APIs).
• Scale: 12,879 records, 115 entity relations, 1,824 temporal chains, 4,576 Hebbian edges.
• Results with Specialized Profiles:
  • HYPERFOCUS Profile: 79.30% nDCG@10, 84.50% MRR@10, 78.10% Recall@10 (time decay clamped to zero for active focus domains).
  • CRITICAL Profile: 79.30% nDCG@10, 84.50% MRR@10 (exponential boost for high-importance episodic milestones).

5. Comparative Architecture Matrix¶

6. Methodology Deep-Dive: IR Metrics vs. LLM-as-a-Judge¶

flowchart LR
    subgraph Tier1["Tier 1: Standalone Memory Engine (Spector Benchmark Harness)"]
        Q1[Query] --> M1[Spector Panama Off-Heap Memory]
        M1 --> R1[Top-10 Candidates]
        R1 --> MTR[IR Metrics: nDCG@10, Recall@10, MRR@10 evaluated on Gold Turn QRELS]
    end

    subgraph Tier2["Tier 2: End-to-End Generative QA (Zep, Mem0, Spector Evaluation)"]
        Q2[Query] --> M2[Memory Substrate]
        M2 --> R2[Retrieved Context: Top 10–15 Candidates]
        R2 --> LLM[LLM Generator: gemini-3.1-flash-lite / gpt-4o-mini]
        LLM --> ANS[Generated Answer Text]
        ANS --> JUG[LLM Judge: Grade Answer 0 or 1 -> J-Score %]
    end

6.1. Why Standalone IR Recall (57%–61%) Yields 83%–85%+ Downstream Accuracy¶

1. Partial Evidence Sufficiency: Dialogue turns frequently contain conversational redundancy. Retrieving 1 out of 2 ground-truth turns is almost always 100% sufficient for an LLM to formulate the correct answer.
2. LLM Deductive Reasoning: Modern models (gemini-3.1-flash-lite, gpt-4o-mini) synthesize subtle clues across retrieved context chunks.
3. Extreme Token Efficiency: Injecting ~1,250 to 1,730 tokens keeps context dense and high-signal, preventing distraction and hallucinations.

6.2. Cold-Start Elimination Protocol¶

A mandatory 20-query JIT and memory warmup pass precedes official metric capture. Without warmup, initial Tiered JIT compilation and operating system page faults cause the first few queries to register 350–800 ms, skewing the reported mean. With warmup, steady-state latency is strictly predictable within \(\pm 0.8\text{ ms}\).

7. Community Discussions & Verification Artifacts¶

For complete raw logs, per-query CSV dumps, and community discussions:

• 📖 MindSpan: A 20-Year Longitudinal Cognitive Memory Benchmark and Why It Supersedes LongMemEval (Discussion #753)
• 🚀 LongMemEval Benchmark Results: 91% Memory Accuracy at 1,500 Tokens with Spector (Discussion #696)
• ⚡ Benchmark Spotlight: Spector Memory Achieves 83.42% Accuracy & 12ms Recall Latency on LoCoMo (Discussion #695)
• 📁 Datasets repository: spectrayan/spector-datasets (mindspan/, longmemeval/, locomo/)