Cognitive Memory Evaluation & Methodology¶

Spector Memory is designed to behave like a biological memory system rather than a static database. To measure how successfully this cognitive approach mimics human recall and improves agentic intelligence, we evaluate it using a 3-Way Cognitive Benchmark.

This document explains our evaluation methodology, the synthetic dataset representing a rich family-oriented persona, the results of our runs, and a deep comparison showing how Spector’s cognitive approach differs from vanilla semantic search.

1. Key Results & Performance Summary¶

We evaluated Spector Memory across two distinct benchmark datasets (Balanced Family and Interest-Diversified) to measure how the scoring pipeline and specialized cognitive profiles respond under different narrative and cognitive constraints.

In all runs, we compare three experimental conditions: 1. Baseline: Pure vector similarity search (nearest-neighbor cosine/L2 distance), similar to standard vector databases. 2. Similarity: The full Spector query pipeline (Bloom filter tag gating, valence range checks, and active-project pre-screening) but ranked using pure cosine similarity. 3. Cognitive: The full Spector pipeline, ranking candidates using fused scoring (similarity, importance, and temporal decay) and extending results via 3-layer cognitive graph traversal (Hebbian, Temporal, Entity).

1.1. Run 1: Balanced Family Evaluation (365-Day Dataset, 50 Queries)¶

This run evaluates retrieval behavior using a broad, general-purpose family persona with routine daily activities, work projects, and community involvement across 50 benchmark queries.

Summary Metrics (Top-10 Retrieval)¶

Pairwise Statistical Comparisons¶

1.2. Run 2: Interest-Diversified Evaluation (365-Day Dataset)¶

This run evaluates a scaled-up dataset with a richer, more interconnected interest graph — the persona has intense hobby interests (backyard astronomy, local LLM hacking, smart home automation) that create deeply cross-linked memory clusters.

Summary Metrics (Top-10 Retrieval)¶

Pairwise Statistical Comparisons (Balanced Mode)¶

Cognitive Profile Forced-Override Comparison (Forced on all queries)¶

Because the persona has intense, deeply interconnected hobby interests (space, astronomy, coding local LLMs), we evaluated how different forced cognitive profile overrides perform against pure similarity:

1.3. Takeaways & Insights¶

1. Cognitive Scoring Now Matches or Exceeds Pure Similarity: After switching from additive to multiplicative fusion (similarity × (1 + β·importance_norm·decay)) and replacing the 5-level step importance function with a continuous sigmoid, cognitive scoring outperforms pure similarity on the balanced dataset (Cohen's d = +0.248) and matches it on the interest-diversified dataset (d = -0.030, p = 0.889). Importance now contributes to 42-68% of queries (up from 9%).
2. All Profiles Now Functional: SYSTEMATIZER improved from 0.056 → 0.507 nDCG (+805%), DIVERGENT from 0.000 → 0.359, and THE_EXECUTOR from 0.000 → 0.113. The continuous importance function gives every memory a unique score, enabling importance-weighted profiles to finally differentiate candidates.
3. Massive Boost from Cognitive Gating (+361% to +420% nDCG): Applying contextual constraints (synaptic tags via Bloom filters, emotional valence filtering) before scoring results in a massive improvement over raw vector databases. Cohen's d values of 0.613 and 1.265 indicate a highly significant shift in retrieval precision.
4. HYPERFOCUS and CRITICAL Profiles Excel (+69% nDCG vs Balanced): The HYPERFOCUS profile clamps time decay to zero for focus-matched memories, and CRITICAL boosts high-importance memories. Both achieve nDCG@10 = 0.793 on the interest-diversified dataset. This validates the profile-switching architecture for specialized retrieval modes.

2. Dataset & Quality Profiles¶

To thoroughly stress-test Spector, we built two complex, chronologically coherent synthetic datasets representing Mike Thompson's family life.

2.1. Dataset A: Balanced Family (365 Days)¶

• Persona Context: Mike Thompson, a 36-year-old Senior Product Manager at Vertex Health. Broad lifestyle interests (soccer coaching, family calendar, woodworking, home repairs).
• Scale: 11,367 total records chronologically spanning 365 days.
• Narrative: Features daily morning calendar briefings, family chores, extended family coordination, and evening journal logs.
• Graph Structures: 115 entity relations, 1,824 temporal chains, 5,422 Hebbian edges.

2.2. Dataset B: Interest-Diversified (365 Days)¶

• Persona Context: Mike Thompson with a richer, more interconnected interest graph and intense hobby focus areas.
• Enriched Interests: Computerized backyard stargazing, reading space science and exoplanet research papers, hacking custom local LLMs, and writing custom smart home Jarvis APIs.
• Scale: 12,879 total records chronologically spanning 365 days. Includes a dedicated hobby-focused interest history block in biographical memories.
• Graph Structures: 115 entity relations, 1,824 temporal chains, 4,576 Hebbian edges.
• Metadata calibration: Higher interest, challenge, and arousal ranges for hobby-focused events, with specific queries and judgments mapped to specialized profiles.

3. Cognitive Memory vs. Standard Semantic Search¶

Most AI agents and databases (e.g., pgvector, Milvus, Chroma) treat memory as a flat vector search problem. The table below illustrates how Spector's cognitive architecture fundamentally differs from this traditional approach:

4. Technical Differentiation Deep Dive¶

I. Synaptic Tag Gating (Bloom Filters)¶

Standard vector databases suffer from the "Semantic Noise Trap." For example, if an agent asks: "What was Greg Holloway's feedback on the PM launch?", a flat semantic search might match woodworking logs because the word "launch" or "feedback" is used in other contexts (e.g., "Sarah gave feedback on the birdhouse launch").

Spector avoids this by encoding tags into high-performance, zero-overhead Bloom filters mapped directly inside the 64-byte off-heap record headers. If a candidate record's Bloom filter does not overlap with the query's synaptic filter tags, it is pruned in Phase 2 of the pipeline—long before expensive distance calculations. This acts as a thalamic filter, keeping the agent focused on the active domain (e.g., #vertex-health).

II. Importance Fusion (ICNU Model)¶

Not all memories are created equal. Traditional systems treat "I drank coffee" and "My wife Sarah and I decided to refinance our mortgage" with equal structural weight.

Spector resolves this by fusing four parameters during memory ingestion: - Interest (\(x_I\)): The user's or agent's engagement level. - Challenge (\(x_C\)): Technical or cognitive complexity. - Novelty (\(x_N\)): Mathematically calculated at runtime by comparing the incoming embedding's L2 distance against active Working Memory slots. - Urgency (\(x_U\)): Temporal priority and time-sensitivity.

These values are combined into a final importance score:

This score directly dictates how long a memory stays in Episodic Memory before being pruned or consolidated, and acts as a multiplier in retrieval scoring.

III. Emotional Valence (Amygdala)¶

Human memory is heavily influenced by emotion; we recall traumatic failures or high-joy successes far more easily than neutral facts. Spector implements this using Valence and Arousal: - Valence (-128 to 127): Represents the emotional tone. Negative valence indicates problems, bugs, or conflicts; positive valence indicates successes, solutions, and milestones. - Arousal (0 to 127): Represents emotional intensity. High arousal (e.g., a critical server outage) triggers "flashbulb memory" mechanics, pinning the memory and preventing decay.

Furthermore, different Cognitive Profiles filter retrieval based on valence: - DEBUGGING & PARANOID_SENTINEL: Restrict searches to negative valence (filtering out positive/neutral noise to locate errors). - RECALLING: Focuses only on positive valence (recalling past success patterns).

IV. The 3-Layer Cognitive Graph¶

Standard databases cannot perform associative retrieval without keyword overlap. Spector overcomes this by traversing three interconnected graphs: 1. Hebbian Graph: Uses Spike-Timing-Dependent Plasticity (STDP) to link concepts that co-occur in the agent's context. If a user asks about "refinancing," the Hebbian graph might automatically activate associated memories about "personal finance" or "Austin mortgage rates" even if those terms are not in the query text. 2. Temporal Chain: Links chronologically adjacent memories together, enabling the agent to walk forward or backward in time (e.g., "What did we do right after the server crashed?"). 3. Entity Graph: Resolves relationship networks (e.g., matching "Sarah" to "wife", or "Vertex Health" to "Greg Holloway") to find multi-hop contextual links.

5. Appendix: Benchmark Evaluation Metrics Reference¶

To ensure standard scientific rigor, Spector evaluates memory retrieval using established information retrieval (IR) metrics calculated over the top 10 retrieved candidates (Top-10):

I. Normalized Discounted Cumulative Gain (nDCG@10)¶

• Concept: Evaluates the ranking quality of retrieved items based on graded relevance (e.g. relevance grade 3 for exact matches, 2 for strong associations, 1 for partial relevance).
• Formula: $\(\text{nDCG} = \frac{\text{DCG}}{\text{IDCG}}\)$ where Discounted Cumulative Gain (DCG) at position \(p\) is defined as: $\(\text{DCG}_p = \sum_{i=1}^p \frac{2^{rel_i} - 1}{\log_2(i + 1)}\)$ and IDCG is the Ideal DCG (the maximum possible DCG value achieved by sorting the results by their true relevance grades).
• Interpretation: A higher nDCG@10 score indicates that the system consistently places the most highly relevant memories at the top of the search results (positions 1–3) rather than burying them at the bottom.

II. Mean Reciprocal Rank (MRR@10)¶

• Concept: Measures the position of the first relevant document in the retrieved list.
• Formula: $\(\text{MRR} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i}\)$ where \(\text{rank}_i\) is the position of the first correctly retrieved relevant memory for query \(i\).
• Interpretation: If the first relevant memory is at rank 1, the score is \(1.0\); at rank 2, it is \(0.5\); at rank 10, it is \(0.1\). If no relevant items appear in the top 10, the score is \(0\). This is highly relevant for conversational agents that rely on immediate, single-item context lookup.

III. Recall at K (Recall@10)¶

• Concept: Measures the system's coverage of relevant items within the retrieved set.
• Formula: $\(\text{Recall@K} = \frac{|\text{Relevant Items in Top-K}|}{|\text{All Relevant Items in Dataset for Query}|}\)$
• Interpretation: Measures what percentage of all labeled relevant memories were successfully surfaced within the top 10 slots. A score of \(1.0\) means all relevant records were retrieved.