ADR-0067: Cell-Based High Availability and Namespace-Sticky Sharding (Initial Draft)¶

1. Context¶

Note: This initial design summary draft has been formally superseded by ADR-0034, which provides the complete 1,370-line specification covering consensus, fencing, lease renewal, asynchronous WAL replication, and Kubernetes operator deployment.

Spector stores cognitive memory with physical per-namespace isolation: each user/agent gets its own on-disk tree of mmap'd binary stores (SIMD hot path). That design:

• Eliminates the truncation trap of shared HNSW + user_id filters
• Makes provable tenant isolation and cheap hard-delete (rm -rf + DEK shred) possible
• Forces a different HA model than “shared DB + replicas”

An initial enterprise draft proposed multi-region cells, L1/L2/L3 (hot mmap / warm NVMe / cold object storage), and checkpoint-driven snapshot replication. Review found the draft correct on isolation and cells, but unsafe on write ownership: multiple R/W leaders behind round-robin / least-conn load balancing with local RWO NVMe would send writes for the same namespace to different nodes and diverge trees.

The draft also described pre-bundle on-disk files (semantic.mem, episodic.mem, separate .graph files). Current target layout is ADR-0004 V4 bundles (one master mmap segment with region slices).

Goals¶

1. High performance — keep local NVMe mmap + SIMD; avoid shared remote FS for hot path
2. Data isolation — physical namespace directories; no cross-tenant index mixing
3. Replication & HA — survive node loss without silent split-brain
4. Honest durability — state async snapshot RPO explicitly (not sync commit)

Non-goals¶

• Synchronous multi-master writes to one namespace
• Shared NFS/EFS as primary hot store
• Replacing NWB/electrophysiology lakes or claiming medical HA classes

2. Problem Statement¶

Operating Spector Memory at scale across multiple data center regions requires high availability without suffering from distributed locking latencies across write paths:

1. Cross-Region Latency Hazards: Running cross-region distributed transactions on every memory write destroys sub-millisecond ingestion throughput.
2. Blast Radius Containment: A failure or data corruption event in one geographic region must not take down global memory services.
3. Partition Ownership Ambiguity: Without clear single-writer tenancy, concurrent writes to the same off-heap memory-mapped bundle trigger fatal data corruption.

3. Decision Drivers¶

• Cell-Based Isolation: Deploy independent regional cells containing isolated nodes and storage volumes.
• Sticky Namespace Routing: Pin all writes for a given tenant to a single primary cell owner.
• Asynchronous Replication: Replicate engram logs asynchronously to secondary standby cells.
• Deterministic Fencing: Prevent split-brain writes using generation leases.

4. Considered Options¶

Alternatives Evaluated in Initial Draft¶

5. Decision Outcome¶

Architectural Decisions (Initial Topology)¶

1. Deploy as regional cells¶

Each cell is an independently deployable unit (leaders + followers + same-region cold bucket).

• Blast radius limited to that cell’s orgs
• Data sovereignty by pinning orgs to a region (e.g. GDPR → EU cell)
• Scale by growing a busy cell or adding cells — not by one global shared mmap pool

Global control plane holds only org → cell routing (and optional capacity signals). Cognitive payloads do not cross cells except via explicit, consented migration.

graph TB
    GW["API Gateway<br/>org → cell"]
    REG["Routing registry<br/>org→cell, ns→primary"]

    subgraph Cell_US["Cell us-east-1"]
        LB1["Sticky LB<br/>namespace → primary"]
        P0["Primary A<br/>shards α"]
        P1["Primary B<br/>shards β"]
        F0["Follower(s)<br/>read-only"]
        S3u["S3 us-east-1 cold"]
    end

    subgraph Cell_EU["Cell eu-central-1"]
        LB2["Sticky LB"]
        P2["Primary"]
        F1["Follower"]
        S3e["S3 eu-central-1 cold"]
    end

    GW --> LB1
    GW --> LB2
    REG -.-> GW
    LB1 --> P0
    LB1 --> P1
    LB1 -.-> F0
    P0 -->|"bundle snapshots"| F0
    P1 -->|"bundle snapshots"| F0
    P0 --> S3u
    P2 --> S3e

2. Namespace-sticky write ownership (mandatory)¶

Exactly one primary writer per namespace at a time.

Rejected: multi-leader R/W + round-robin with local disks (split-brain).

Accepted alternatives (ops trade-off):

• Single primary per cell + N read replicas (simpler; less write scale)
• Per-shard follower sets (better isolation; more pods)

3. On-disk unit of replication = V4 bundles¶

Supersede legacy multi-file snapshot lists. Snapshot packaging ships:

• PartitionBundle(s) under partitions/NNN_epoch/ (cognitive region slabs)
• RuntimeBundle (graphs, indexes, coactivation, BM25, checkpoint region, …)
• Namespace metadata (namespace.json, layout version, WAL/checkpoint high-water if present)
• Integrity: bundle directory checksum (e.g. xxHash64) + optional outer CRC

Directory discovery uses 2-level hash sharding (StorageLayout / equivalent) + DataLayoutVersion + atomic LayoutMigrator — not only flat data/namespaces/<tenant>/<ns>.

Illustrative layout (names may evolve; bundle files are the contract):

data/namespaces/
  a3/f7/<tenant>/
    tenant.json
    <namespace>/
      namespace.json
      layout.version
      partitions/
        000_<epoch>/
          partition.bundle          # PartitionBundle master mmap
      runtime/
        runtime.bundle              # RuntimeBundle master mmap
      global/                       # checkpoint meta / trackers as required

NamespaceSnapshotSync packages live bundle files (and required sidecars), not a hardcoded V3 semantic.mem + text.dat + *.graph checklist.

4. Hot / warm / cold (JIT memory pager)¶

Physical isolation implies not every namespace can stay mmap’d.

Evict L1→L2 on pressure/idle; archive L2→L3 after idle policy. Cold rehydrate must not block MCP/API without a deadline — return warming / queue, then promote.

5. Replication = checkpoint-driven bundle snapshots¶

Prefer snapshot shipping over multiplexed per-event WAL fan-out across thousands of namespaces.

sequenceDiagram
    participant CK as CheckpointDaemon
    participant RC as ReplicationCoordinator
    participant SS as NamespaceSnapshotSync
    participant F as Follower

    CK->>RC: CheckpointCompleted(namespace, hwm)
    RC->>RC: mark dirty
    RC->>SS: syncNamespace(ns)
    SS->>SS: package PartitionBundle + RuntimeBundle + meta
    SS->>F: ship compressed snapshot (mTLS)
    F->>F: verify checksum, fence epoch, unpack, remmap
    F-->>RC: ACK

Durability class (honest): async snapshots ⇒ RPO ≈ snapshot interval / min-changes window (e.g. tens of seconds to minutes), not zero RPO. Product/docs must not imply sync commit HA unless a future sync path ships.

Follower lag: if lag exceeds threshold → FULLRESYNC of that namespace with epoch fencing so a late snapshot cannot clobber a newer primary open.

Deprecated for new work: multiplexed WalReplicationLeader / frame demux as the primary HA path (may remain for legacy; docs must not present it as current).

Leader election: Kubernetes Lease and/or static role for lab; election must align with namespace (or shard) ownership, not “all nodes are writers.”

6. DR & sovereignty¶

• Cold tier and CRR stay inside the sovereignty boundary (EU primary cold ↔ EU standby). Do not use US→EU CRR as the GDPR story.
• Org→cell mapping is sticky; DR failover updates routing to the standby cell in the same legal region unless the customer opts into multi-region.
• Illustrative targets (tunable): RPO driven by cold/snapshot lag; RTO by cell scale-up + JIT rehydrate.

7. Analytics stays CQRS/CDC¶

Org-level analytics must not scan every namespace bundle. Stream metadata only (counts, types, tokens, tags) via CDC; never raw text/vectors/graph edges to the OLAP path by default.

8. Target vs Implemented¶

This ADR is the architecture contract. Implementations may lag.

Docs and sales materials must label Target vs Implemented explicitly — no blanket “all HA components shipped.”

6. Pros and Cons of the Options¶

Consequences & Trade-offs¶

Positive¶

• Preserves mmap/SIMD performance and isolation guarantees under horizontal scale
• Eliminates split-brain from naïve multi-leader LB
• Aligns replication with V4 bundle reality (fewer FDs, coherent CRC, simpler ship unit)
• Clear sovereignty and durability narrative

Negative / trade-offs¶

• Sticky routing + rebalancing is operationally harder than round-robin
• Async RPO may be insufficient for some enterprise RFPs (call out; offer higher-frequency snapshots or single-node sync disk as premium later)
• Follower remmap after snapshot has cost; read-your-writes should prefer primary
• Bundle growth/remap (ADR-0004) must coordinate with snapshot fencing

Follow-ups¶

1. Spec NamespaceOwnership API (hash ring vs lease registry) + gateway sticky key
2. Update NamespaceSnapshotSync contract to V4 bundle manifest
3. Rewrite enterprise HA runbooks; remove WAL-primary language from port matrices
4. Property tests: single-writer invariant; snapshot epoch monotonicity; cold warm non-blocking deadline
5. Load test: shard rebalance while serving remember/recall

7. Implementation Plan¶

• Evolution to Comprehensive Specification: The initial design documented here served as the working foundation for the complete clustering architecture codified in ADR-0034. Refer to ADR-0034 for complete consensus protocols, lease algorithms, and failover sequence diagrams.

8. Code Reference & Verification¶

• Cluster Module Implementation: cluster/spector-cluster/src/main/java/com/spectrayan/spector/cluster/
• Superseded By: docs/adr/0034-cell-ha-namespace-ownership.md