ADR-0028: Dual-Plane Memory Audit Architecture (Separate Regions)¶

1. Context¶

1.2 The Two Distinct Audit Needs in Spector Architecture¶

A deep architectural analysis across the Spectrayan portfolio reveals two fundamentally distinct auditing tiers within the spector-memory engine (in addition to the high-level application/REST audit log in spector-synapse #211):

2. Problem Statement¶

1.1 The Mixed 64-Byte Cache Line Pathology¶

In Spector Memory, every cognitive memory record in off-heap Panama FFM storage (spector-memory) consists of a 64-byte SynapticHeader (HeaderLayout64) followed by a quantized vector payload (INT8 scalar quantized float vector).

The 64-byte header was designed to match a single CPU cache line (64 bytes). However, as cognitive capabilities evolved (emotional gating, Zeigarnik effect, Two-Factor Bjork memory decay, auto-LTP cooldowns, adaptive cognitive profile learning, and soul-state tracking), the 64-byte header became a mixed-tenancy cache line containing two fundamentally opposing classes of data:

• Immutable Encoding Identity: header_version, flags, valence, arousal, base_importance, timestamp_ms, exact_norm, synaptic_tags (Bloom filter), centroid_id, consolidation_flags, encoding_profile, encoding_alpha/beta, soul_version, encoding_surprise. (Written once during ingestion).
• Mutable Recall Audit Telemetry: agent_recall_count, storage_strength (\(S(t)\)), spector_recall_cnt, last_auto_ltp, last_recall_profile, effective_importance. (Mutated on every query retrieval and reinforcement).

3. Decision Drivers¶

• 100% Cache Line Isolation: Immutable memory identity headers must remain untouched by frequent query retrieval mutations.
• Sub-Millisecond ACT-R Power-Law Decay: Support 8-slot ring-buffer timestamp tracking for Anderson (1993) spacing effect without heap churn.
• Zero-Allocation Mmap Stride: Per-slot recall audit telemetry must align to fixed byte strides accessed via Panama FFM memory segments.
• Append-Only Provenance Separation: Decouple high-frequency per-slot recall stats from long-term immutable ingestion and consolidation provenance logs.

4. Considered Options¶

Option 1: Monolithic 128-Byte Expanded Header¶

• Description: Expand HeaderLayout64 to 128 bytes, putting both identity and audit telemetry on adjacent cache lines.
• Advantages: Single contiguous allocation per record.
• Disadvantages: Doubling header footprint severely impairs vector scan cache residency; false sharing across threads.

Option 2: External Relational / JDBC Audit Logging¶

• Description: Direct all recall counters and timestamp updates to an external RDBMS or embedded H2 database.
• Advantages: Rich SQL querying.
• Disadvantages: Introduces multi-millisecond disk I/O and transaction serialization onto the sub-millisecond recall hot path.

Option 3: Dual-Plane Separate Region Architecture (Selected)¶

• Description: Physically split immutable encoding headers (in data slabs) from mutable recall telemetry (in dedicated fixed-stride mmap regions: RegionId.STRENGTH / RECALL_AUDIT), while recording engine-level provenance in append-only chronicle regions (RegionId.PROVENANCE_CHRONICLE, superseded by ADR-0029).
• Advantages: 100% L1/L2 cache isolation; zero-allocation point updates; unblocks ACT-R 8-slot power-law calculations.
• Disadvantages: Requires managing an additional mmap region per partition.

5. Decision Outcome¶

D1: Pure 64-Byte V2 Synaptic Header (HeaderLayout64V2)¶

The Synaptic Header is bumped to header_version = 2. All mutable runtime counters are excised from the header cache line.

The freed space is leveraged to:

1. Upgrade Synaptic Tags Bloom Filter to 128-Bit (16 Bytes): Offsets 24–39 now hold a 128-bit double-hash Bloom filter, reducing false-positive rates during pre-filtering from ~3.2% down to <0.05% across 100K concepts.
2. Establish 16-Byte Reserved Gating Block: Offsets 48–63 provide future-proof reserved capacity for holographic manifold representations, Riemannian curvature invariants, and multi-modal sensory routing without requiring further header size shifts.

Complete V2 Synaptic Header Byte Map (64 Bytes)¶

D2: Single Unified Partition Strength Region (RegionId.STRENGTH & StrengthLayout)¶

Instead of creating separate strength/audit regions per tier (AUDIT_SEMANTIC, AUDIT_EPISODIC, AUDIT_PROCEDURAL), PartitionBundle allocates a single unified RegionId.STRENGTH(4) region (originally named RegionId.AUDIT(4) in initial design; renamed to RegionId.STRENGTH to disambiguate recall strength and retention telemetry from operational execution auditing).

Architectural Design of Unified Strength Space:¶

1. Memory Type Ordinal Embedded: The StrengthLayout embeds the memoryType ordinal (2 bits in audit_flags), matching the memory type bits in SynapticHeaderConstants.FLAG_TYPE_MASK.

2. Cumulative Slot Indexing: The total capacity of the strength region equals \(C_{\text{total}} = C_{\text{semantic}} + C_{\text{episodic}} + C_{\text{procedural}}\). Addressing is computed by cumulative tier base offsets: $\(\text{strengthOffset}(\text{tier}, \text{slot}) = (\text{tierBaseSlot}(\text{tier}) + \text{slot}) \times \text{recordStride}\)$

3. Parity with Text Region: Just as RegionId.TEXT in PartitionBundle serves all tiers as a single shared text blob store, RegionId.STRENGTH serves all tiers as a single shared strength store.

flowchart TD
    subgraph bundle ["PARTITION BUNDLE FILE (partition.bundle)"]
        direction TB
        header["64B SMKM MemoryHeader"]
        subHeader["64B SPTB BundleSubHeader"]
        dirEntries["Bundle Directory Entries (SEMANTIC, EPISODIC, PROCEDURAL, TEXT, STRENGTH)"]

        region0["Region 0: SEMANTIC (64B V2 Header + INT8 Vector) ──► Pure Read-Only Pages"]
        region1["Region 1: EPISODIC (Episodic Log Structure)"]
        region2["Region 2: PROCEDURAL (64B V2 Header + INT8 Vector) ──► Pure Read-Only Pages"]
        region3["Region 3: TEXT (Shared variable length text blobs)"]

        subgraph region4 ["Region 4: STRENGTH (Single Unified Recall Strength Region) ──► Write-Heavy"]
            direction TB
            semStr["[0 .. N_sem - 1] : Semantic Strength Records"]
            epiStr["[N_sem .. N_sem+epi] : Episodic Strength Records"]
            proStr["[N_sem+epi .. Total] : Procedural Strength Records"]
            semStr ~~~ epiStr ~~~ proStr
        end

        header ~~~ subHeader ~~~ dirEntries ~~~ region0 ~~~ region1 ~~~ region2 ~~~ region3 ~~~ region4
    end

Complete AuditRecordLayout Byte Map (96 Bytes, 32-Byte Aligned)¶

D3: Full Anderson (1993) ACT-R 8-Slot Ring Buffer Engine¶

With 32 dedicated bytes in AuditRecordLayout, we expand the ACT-R ring buffer from a disabled 3-slot construct to an active 8-slot circular ring buffer:

where \(t_j = \text{nowMs} - (\text{creationMs} + \Delta t_j \times 1000)\).

Evaluation:¶

• Stored as uint32 seconds elapsed since creationMs (\(\approx 136\) years range).
• Evaluated in \(O(1)\) via precomputed decay table lookup (DecayStrategy#ageToBucket) and algebraic sigmoid normalization \(\sigma(\ln x) = \frac{x}{x + 1}\) — zero Math.pow, zero Math.log, zero Math.exp at query time.

D4: Kernel Provenance & Lineage Chronicle Region (RegionId.PROVENANCE_LOG)¶

To fulfill MF-001 (Memory Model Algebra & Conformance Rules) and Issue #171 (Memory Lineage & Provenance) at the storage layer:

1. Dedicated Append-Only Region: RuntimeBundle and PartitionBundle support RegionId.PROVENANCE_LOG(26).

2. Binary Provenance Record (ProvenanceEntry):

   • timestamp_ms (8B), target_memory_id (16B TSID), operation_code (1B: INGEST, CONSOLIDATE, RECONSOLIDATE, REINFORCE, FORGET, RETRACT, SIMULATE_COMMIT).
   • source_uri_hash (8B), parent_trace_count (2B), parent_trace_ids (variable array of parent TSIDs collapsed during semantic reflection), lineage_diff (text/binary delta).

3. Difference from MemoryWal: While MemoryWal is a short-lived, rolling recovery log compacted during checkpoints, the PROVENANCE_LOG is an immutable longitudinal audit chronicle that guarantees full explainability: "Why does the agent know this fact, which raw episodes were synthesized to form it, and when was it modified?"

3. Class Hierarchy & SPI Contracts¶

classDiagram
    class HeaderLayout {
        <<sealed interface>>
        +headerBytes() int
        +version() int
        +readTimestamp(seg, off) long
        +readSynapticTags(seg, off) long
        +readExactNorm(seg, off) float
        +readBaseImportance(seg, off) float
        +readFlags(seg, off) byte
    }

    class HeaderLayout64V1 {
        <<legacy record>>
        +version() 1
    }

    class HeaderLayout64V2 {
        <<record>>
        +version() 2
        +readSynapticTagsLo(seg, off) long
        +readSynapticTagsHi(seg, off) long
        +readConsolidationFlags(seg, off) byte
        +readEncodingProfile(seg, off) byte
        +readEncodingSurprise(seg, off) float
    }

    class AuditRecordLayout {
        <<record>>
        +recordStride() 96
        +layoutId() int
        +schemaVersion() int
        +readAgentRecallCount(seg, off) int
        +incrementAgentRecallCount(seg, off) int
        +readEffectiveImportance(seg, off) float
        +casEffectiveImportance(seg, off, fn) float
        +readStorageStrength(seg, off) float
        +casStorageStrength(seg, off, fn) float
        +recordRecall(seg, off, creationMs, nowMs) void
        +computeBaseLevelActivation(seg, off, creationMs, nowMs) float
    }

    class CognitiveRecordLayout {
        +headerLayout() HeaderLayout
        +auditLayout() AuditRecordLayout
        +vectorOffset(recordOffset) long
        +stride() int
    }

    class AuditRecordMemory {
        <<class>>
        -MemorySegment auditSegment
        -AuditRecordLayout auditLayout
        +auditOffset(type, slot) long
        +auditRecord(type, slot) AuditRecord
    }

    HeaderLayout <|-- HeaderLayout64V1 : permits
    HeaderLayout <|-- HeaderLayout64V2 : permits
    CognitiveRecordLayout o-- HeaderLayout
    CognitiveRecordLayout o-- AuditRecordLayout
    AuditRecordMemory o-- AuditRecordLayout

6. Pros and Cons of the Options¶

7. Implementation Plan¶

1. Phase 1: Define RegionId.STRENGTH / RecallAuditLayout in memory/spector-kernel.
2. Phase 2: Migrate mutable telemetry fields out of HeaderLayout64.
3. Phase 3: Implement 8-slot timestamp ring buffer for ACT-R Bjork spacing decay.
4. Phase 4: Author BundleMigrationCli to migrate V1 shards to the dual-plane layout.
5. Phase 5: Superceded §D4 provenance chronicle via ADR-0029 (LineageProvenanceRegion).

8. Code Reference & Verification¶

Positive¶

• 100% Cache Isolation: Read-mostly encoding headers remain pristine in CPU caches during intense concurrent recall and reinforcement.
• Unified Region Efficiency: Exactly 1 strength region (RegionId.STRENGTH, formerly RegionId.AUDIT) per partition bundle simplifies directory structure, allocations, and OS paging.
• Unblocks ACT-R Cognitive Modeling: Provides 8 dedicated timestamp slots for full Anderson (1993) power-law decay and spacing effect calculations.
• 60× Higher Tag Filtering Precision: 128-bit Bloom filter slashes false-positive candidate generation in large memory stores.
• Full MF-001 Provenance Conformance: Cleanly separates fast slot-indexed recall telemetry from long-term immutable provenance logs.

Negative / Trade-Offs¶

• Disk Footprint Growth: +96 bytes per allocated record in the partition bundle (~91 MB per 1M records). Mitigated by fixed-size bundle provisioning and sparse allocation.
• Migration Requirement: Existing V1 persistent shards require a one-time migration step via BundleMigrationCli.

Code Reference & Verification Gate¶

• Primary Module(s): memory/spector-kernel, memory/spector-memory, bench/spector-bench
• Key Packages: com.spectrayan.spector.kernel.layout, com.spectrayan.spector.memory.scheduler
• Classes: EncodingHeaderLayout.java, TaskRunAuditRecord.java, MindSpanStrengthAndAuditInspectionTest.java
• Verification Tests: EncodingHeaderLayoutTest.java, EncodingHeaderFieldsTest.java