📝 WAL Design — Write-Ahead Log¶

Biological Analog: The hippocampus doesn't write memories directly to the neocortex. It first records a transient "replay buffer" — a sequential log of experiences — and consolidates them during sleep. The WAL is the digital equivalent: an ordered, append-only log of every memory mutation that can be replayed to reconstruct state.

Why a WAL?¶

Cognitive memory stores mutable state (importance, valence, recall count, tags) in off-heap MemorySegment buffers. Without durability, a JVM crash loses everything. The WAL provides:

Architecture Overview¶

graph TD
    subgraph "Write Path"
        A["SpectorMemory.remember()"] --> B["MemoryWal.append()"]
        B --> C["writeLock.lock()"]
        C --> D["events.add(event)"]
        C --> E["writeEventToChannel()"]
        E --> F["CRC-32 header + payload"]
        F --> G["FileChannel.write()"]
        G --> H{"chunk ≥ 8MB?"}
        H -->|yes| I["rollChunk()"]
        H -->|no| J["fsync (optional)"]
    end

    subgraph "Read Path (Recovery)"
        K["JVM restart"] --> L["recoverFromDisk()"]
        L --> M["findChunkFiles()"]
        M --> N["readChunkFile() × N"]
        N --> O["Validate magic + CRC"]
        O --> P["Rebuild in-memory cache"]
        P --> Q["Restore sequenceCounter"]
    end

    subgraph "Replication"
        R["CloudSync.exportEvents()"] --> S["replay(afterHwm)"]
        S --> T["Ship to remote agent"]
        T --> U["importEvents() + CRDT merge"]
    end

    style A fill:#6c5ce7,color:white
    style B fill:#00b894,color:white
    style K fill:#e17055,color:white
    style R fill:#0984e3,color:white

Dual Mode Operation¶

MemoryWal operates in two modes, selected at construction time:

Event Types¶

Every memory mutation produces a WAL event with the following fields:

Binary Record Format (V2)¶

File Header¶

Each WAL chunk file begins with an 8-byte header:

Offset   Size   Field      Value
──────   ────   ─────      ─────
  0       4B    magic      0x53504543 ("SPEC" in ASCII)
  4       4B    version    2

Record Layout¶

Each event is serialized as a 40-byte fixed header followed by variable-length segments, aligned to 8-byte boundaries:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         recMagic (2B)         |  version (1B) |   flags (1B)  |  ← Offset 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  typeOrd (1B) |          idLen (2B)           | reserved (1B) |  ← Offset 4
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                      sequence (8B)                            +  ← Offset 8
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                timestamp — epoch millis (8B)                  +  ← Offset 16
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    payloadLen (4B)                             |  ← Offset 24
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    payloadCRC (4B)                             |  ← Offset 28
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     reserved (4B)                             |  ← Offset 32
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     headerCRC (4B)                            |  ← Offset 36
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  memoryId (idLen bytes, UTF-8)                |  ← Offset 40
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          payload (payloadLen bytes, optionally compressed)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 padding (0–7 bytes to 8-byte align)           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Field Reference¶

Total record size: 40 + idLen + payloadLen + padding

Integrity: Dual CRC-32¶

Every record has two independent CRC-32 checksums:

graph LR
    H["Header bytes 0-35"] -->|CRC-32| HC["Header CRC (offset 36)"]
    P["Payload bytes"] -->|CRC-32| PC["Payload CRC (offset 28)"]

    HC -->|verified on read| V1["✅ Header intact"]
    PC -->|verified on read| V2["✅ Payload intact"]

    style HC fill:#00b894,color:white
    style PC fill:#00b894,color:white

This split design detects:

• Torn headers: header CRC fails → truncate at record start
• Corrupt payloads: payload CRC fails → quarantine chunk file
• Partial writes: record magic missing → truncate at boundary

Chunked File Layout¶

WAL data is spread across multiple chunk files in a directory:

.spector/memory/wal/
├── wal-000000.bin    ← oldest chunk (may be truncated after snapshot)
├── wal-000001.bin
├── wal-000002.bin
├── wal-000003.bin    ← active chunk (currently being written)
└── .quarantine/      ← corrupted chunks moved here
    └── wal-000001.bin

Chunk Rolling¶

When the active chunk exceeds maxChunkBytes (default 8 MB), the WAL:

1. Flushes all data and metadata to disk
2. Closes the file
3. Increments the chunk index
4. Opens a new chunk file with a fresh file header

Compaction & Garbage Collection¶

As memories decay or undergo sleep-consolidation, older WAL chunks become redundant. The WAL enforces snapshot-driven truncation — chunks are only deleted after a snapshot proves their events have been fully materialized to disk.

flowchart TD
    A["Active Writing Chunk"] -->|"size ≥ maxChunkBytes (8MB)"| B["rollChunk()"]
    B --> C["Immutable Closed Chunk"]
    D["Background Consolidation Daemon"] -->|"runs memory consolidation"| E["Generate Disk Snapshot"]
    E -->|"write metadata"| F["Persist Snapshot High-Water Mark"]
    F -->|"trigger compaction"| G{"truncateBefore(snapshotHwm)"}
    G -->|"chunk maxSeq ≤ snapshotHwm"| H["Safe to Delete"]
    G -->|"chunk maxSeq > snapshotHwm"| I["Must Retain"]
    G -->|"chunk == activeChunkPath"| J["Never Touched"]
    H --> K["Files.delete(chunk)"]
    H --> L["events.removeIf(seq ≤ hwm)"]

    style A fill:#6c5ce7,color:white
    style H fill:#00b894,color:white
    style J fill:#e17055,color:white

How it works:

1. Snapshot trigger: The consolidation daemon (hippocampus) periodically snapshots the full in-memory state to disk (mmap partition files)
2. HWM declaration: The snapshot records the highest WAL sequence number that has been fully materialized
3. Chunk disposal: truncateBefore(snapshotHwm) sweeps all closed chunks — any chunk where the maximum sequence ≤ HWM is safely deleted
4. Active chunk protection: The currently active chunk is never deleted, even if all its events are below the HWM
5. In-memory cache pruning: Events with sequence ≤ HWM are also removed from the in-memory cache to prevent bloating

Example: After a snapshot at sequence 5042:

• ✅ Deletes wal-000000.bin (maxSeq=3200) and wal-000001.bin (maxSeq=4980)
• ⏳ Retains wal-000002.bin (maxSeq=5100 — has events after HWM)
• 🔒 Retains wal-000003.bin (active chunk — never touched)

Crash Recovery¶

On startup, MemoryWal automatically recovers from disk:

sequenceDiagram
    participant JVM as ☕ JVM Restart
    participant WAL as 📝 MemoryWal
    participant FS as 💾 Filesystem

    JVM->>WAL: new MemoryWal(walDir)
    WAL->>FS: findChunkFiles() — sorted by name
    loop Each chunk file
        WAL->>FS: Open FileChannel (READ+WRITE)
        WAL->>WAL: Validate file header (magic + version)
        loop Each record
            WAL->>WAL: Read 40B header
            WAL->>WAL: Verify record magic (0x5741)
            WAL->>WAL: Verify header CRC-32
            WAL->>WAL: Read variable segments
            WAL->>WAL: Verify payload CRC-32
            alt Torn write detected
                WAL->>FS: truncate(startPos) — repair in place
                WAL->>WAL: Stop reading this chunk
            else Mid-log corruption
                WAL->>FS: Move to .quarantine/
                WAL->>WAL: Throw WalCorruptionException
            end
        end
    end
    WAL->>WAL: Restore sequenceCounter to max(seq)
    WAL->>WAL: Open next chunk for writing

Corruption Recovery Strategy¶

Because distributed nodes can experience power cuts, OS crashes, or disk hardware decay, the recovery process must handle corruption gracefully and never allow silent data loss.

Classification of Corruptions¶

graph TD
    A["WAL Boot Scan"] --> B{"Verify Record CRC?"}
    B -->|"All Valid"| C["✅ Replay Completed"]
    B -->|"CRC Mismatch / Truncated"| D{"Corruption at file tail?"}
    D -->|"Yes — Torn Write"| E["Auto-Repair: truncate(startPos)"]
    D -->|"No — Mid-Log Bit Rot"| F["Fatal: Quarantine Protocol"]
    E --> G["Resume writing after last valid record"]
    F --> H["Halt boot + move file to .quarantine/"]
    H --> I["Throw WalCorruptionException"]
    I --> J["Cold Bootstrap from healthy peer"]

    style C fill:#00b894,color:white
    style E fill:#fdcb6e,color:black
    style F fill:#d63031,color:white

A. Torn Writes (End-of-File Corruption)¶

Action: The WAL truncates the file to the last valid record boundary (startPos) and flushes to disk. Writing resumes from the repaired position.

B. Mid-Log Corruption (Bit Rot)¶

Action: The chunk file is moved to the .quarantine/ directory to preserve forensic evidence. A WalCorruptionException halts startup. In cluster mode, the node initiates a Cold Bootstrap from a healthy peer.

Summary Matrix¶

Compression¶

Payload compression is opt-in and uses DEFLATE:

When compression is enabled:

1. Payloads larger than the threshold are DEFLATE-compressed before writing
2. The flags byte (offset 3) has bit 0 set to 1
3. On read, the flag is checked and the payload is decompressed with Inflater
4. CRC-32 is computed on the compressed bytes (what's on disk)

Distributed Sync — CloudSync¶

CloudSync provides pull-based replication between agents using the WAL as the replication log:

graph LR
    subgraph "Agent A"
        WA["MemoryWal A"] --> CSA["CloudSync A"]
    end

    subgraph "Agent B"
        WB["MemoryWal B"] --> CSB["CloudSync B"]
    end

    CSA -->|"exportEvents(remoteHwm)"| EVENTS["WAL Events"]
    EVENTS -->|"importEvents()"| CSB

    CSB -->|"CRDT merge"| WB

    style CSA fill:#0984e3,color:white
    style CSB fill:#0984e3,color:white
    style EVENTS fill:#fdcb6e,color:black

Replication Protocol¶

1. Agent B sends its highWaterMark to Agent A
2. Agent A calls wal.replay(remoteHwm) → returns only new events
3. Events are shipped to Agent B (in-process V2, HTTP/gRPC V3)
4. Agent B replays each event into its local memory store
5. Conflicts are resolved via CRDT merge (see below)

Cold Bootstrap¶

When a new agent joins (or corruption triggers a full resync):

The new agent requests a full state snapshot from a healthy leader via GET /api/v2/memory/snapshot. The leader serves its entire off-heap state as a zip archive. The new agent unpacks it, restoring all mmap partition files and WAL chunks.

CRDT Merge Strategy¶

When two agents modify the same memory concurrently, CrdtMergeStrategy resolves conflicts deterministically:

Convergence guarantee: All merge operations are commutative, associative, and idempotent — any order of merges from any agents produces the same final state.

The merge is applied only if the remote state would actually change the local state, avoiding unnecessary writes.

Thread Safety¶

Configuration¶

WAL behavior is controlled via spector.yml:

spector:
  memory:
    persistence-mode: DISK          # DISK | IN_MEMORY
    persistence-path: .spector/memory

Storage Adapter SPI¶

For cloud-based WAL replication, the StorageAdapter SPI provides a pluggable backend:

The StorageAdapter SPI provides a pluggable backend for cloud-based WAL replication. Implementations must support:

Planned implementations:

Next Steps¶

• Off-Heap Panama Design — how mmap partitions store cognitive records
• Hippocampus — Sleep Consolidation — the consolidation daemon that triggers snapshot + truncation
• Architecture — system overview
• Synapse — Tags & Scoring — the synaptic header that WAL events serialize