🎛️ Engine & Algorithmic Tuning¶

Deep tuning guide for Spector's programmatic Java engine, HNSW graph parameters, quantization profiles, and hybrid search weights. Learn how to optimize for recall, latency, throughput, or memory footprint using the SpectorConfig builder API.

🎯 Programmatic Core Parameters¶

When initializing Spector via the Java API, SpectorConfig provides the foundational builder for vector memory:

Choosing a similarity function:

• COSINE — Normalized embeddings (most models)

• DOT_PRODUCT — Unnormalized embeddings where magnitude matters

• EUCLIDEAN — Spatial/geometric data

🗜️ Quantization Parameters¶

🎛️ Quantization Profiles¶

Oversampling Tuning¶

The oversamplingFactor controls how many extra candidates are retrieved before rescoring with exact distances:

• 1 — No rescore (fastest, quantized scores returned directly)

• 3 — Good balance for INT4 (retrieves 3×K candidates, rescores to top-K)

• 5 — Recommended for INT2 (compensates for aggressive quantization)

• 10+ — Diminishing returns; use only if recall is still insufficient

// INT4 with custom oversampling
var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(50_000_000)
    .withQuantization(QuantizationType.SCALAR_INT4)
    .withRescore(5);  // Higher oversampling = better recall, slightly slower

🌐 HNSW Index Parameters¶

🎛️ Tuning Profiles¶

📝 BM25 Parameters¶

🧬 Hybrid Search (RRF)¶

• k = 60 — Original paper recommendation, works well generally

• Lower k (10–30) — Emphasizes top-ranked results more strongly

• Higher k (100+) — Flattens rank importance

🎮 GPU Configuration¶

🚀 ColBERT v2 Reranker Configuration¶

🖥️ Server Configuration¶

# Start Spector Synapse server on port 7070 with custom API key
SPECTOR_API_KEY=my-secret-key mvn -Psynapse -pl synapse/spector-synapse spring-boot:run

🌐 Cluster Configuration¶

🧠 Memory Configuration¶

Operating Mode¶

Memory Tier Parameters¶

Strength & Recall Telemetry Parameters¶

🗂️ Namespace Storage Layout & Routing¶

• spector.data-dir: The node-level data directory holding db/synapse.mv.db (catalog) and identity/ (character-prefix sharded soul & salience bundles).
• spector.memory.persistence-path (or persistence-path): The rememberer persistence root (defaults to ${spector.data-dir}/cognitive in Synapse, or .spector/memory in embedded kernel). All data plane rememberers (NamespacePathResolver) resolve relative to this directory.

Retrieval Stack Parameters¶

Partitioned Semantic Storage¶

When using DISK persistence mode, semantic memories are stored in rolling partition files:

.spector/memory/semantic/
  semantic-000.mem     ← partition 0 (oldest, immutable)
  semantic-001.mem     ← partition 1 (immutable)
  semantic-002.mem     ← partition 2 (active, accepts writes)

Tuning nodesPerPartition:

• Smaller partitions (1K–5K) → faster compaction, more parallel search threads, more files
• Larger partitions (10K–50K) → fewer files, slightly lower overhead per partition
• Default (10K) → good balance for most workloads

Cluster Replication for Partitions¶

When enabled, immutable semantic partitions are shipped as snapshots to replica nodes. Only the active (mutable) partition requires WAL-based delta replication.

📥 Ingestion & Chunking Configuration¶

🎯 Configuration Examples¶

🎯 High-Recall Setup¶

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(500_000)
    .withQuantization(QuantizationType.SCALAR_INT8)
    .withM(32)
    .withEfConstruction(400)
    .withEfSearch(200);

🗜️ Balanced Compression (INT4)¶

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(50_000_000)
    .withQuantization(QuantizationType.SCALAR_INT4)
    .withRescore(3);  // default for INT4

💾 Maximum Compression (INT2)¶

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(200_000_000)
    .withQuantization(QuantizationType.SCALAR_INT2)
    .withRescore(5);  // default for INT2

⚡ Low-Latency Setup¶

var config = SpectorConfig.DEFAULT
    .withDimensions(128)
    .withCapacity(100_000)
    .withM(12)
    .withEfConstruction(100)
    .withEfSearch(30);

🎮 GPU-Accelerated Batch Processing¶

var config = SpectorConfig.DEFAULT
    .withDimensions(768)
    .withCapacity(1_000_000)
    .withGpu(true)
    .withGpuMemoryBudget(2048);  // 2 GB

🤖 RAG Pipeline¶

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withMaxTokens(1024)
    .withOverlapTokens(100)
    .withEmbeddingBatchSize(64);

🌐 Cell High Availability & Ownership Ring (spector.cell.*)¶

Spector Synapse supports multi-node Cell High Availability via a Ketama consistent hash ring.

• Dynamic Reload: Membership and ring configuration can be loaded at startup or reloaded dynamically without restart via control store updates.
• Ownership vs. Placement: Namespace ownership (which server process is the authoritative single writer) is decided purely in-memory by the Ketama ring prior to namespace open. Placement (which filesystem directory holds the partition bundle) is determined by storage tenant paths (StoragePaths) and remains completely invariant to cluster routing changes (Invariant J6).
• Fail-Closed: Non-owner nodes refuse both writes and recall with HTTP 421 Misdirected Request (NamespaceNotOwnedException) identifying the authoritative owner node and ring epoch. Non-owner nodes never invoke runtime.attach or map files into memory.

⚡ Cell Routing Cache & Gateway Resilience (spector.routing.*)¶

Spector Synapse accelerates routing resolution via a three-tier waterfall (L1 Caffeine \(\to\) L2 Redis \(\to\) L3 Ketama Hash Ring fallback), provides pub/sub invalidation, and implements gateway forwarding with bounded retry.

• What breaks when Redis is down: Latency increases slightly as lookups fall back to local pure-computation Ketama ring evaluation. Failover overrides are temporarily masked until the control store republishes.
• What does NOT break: Single-writer correctness and write availability never break. The cell functions continuously as a single-writer system even if Redis crashes entirely (Invariant K1). Zero dual-writers, zero data loss. Operators should not page on transient Redis hiccups.
• Production Topology (Req R9.3, R9.4): Managed cloud offerings (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis) are recommended for production; containerized Redis is for local testing. Redis instances must be provisioned per cell, never shared across multi-region cells.

🔄 Cell Snapshot Replication & Replica Recall (spector.replication.*)¶

Spector Synapse enables high-throughput asynchronous namespace replication from authoritative cell owners to clean replica nodes over a dedicated mutual TLS 1.3 transport, bounded-staleness replica recall, and sealed-partition deduplication.

• Default OFF & Opt-In Reads (Req R10.3): Replica reads default OFF. A client that does not explicitly pass X-Spector-Allow-Replica: true will never receive stale reads, even if replica infrastructure is healthy.
• Three Simultaneous Conditions (Req R10.1, N7): A replica serves recall if and only if: (1) namespace is locally mapped, (2) \((\text{now} - \text{snapshotTime}) \le \text{maxReplicaLagSeconds}\), and (3) incoming request permits replica reads. If staleness bound is violated, the replica refuses rather than serves stale data (ReplicaStalenessExceededException).
• Unconditional Write Refusal (Req R10.6, Invariant N2): A replica never writes. Any write operation dispatched to a replica is rejected immediately with HTTP 405 (ReplicaWriteRefusedException). The X-Spector-Allow-Replica header widens read consistency, never write routing.
• HWM-Advance-Last Crash Safety (Req R5.2, N5): The replica apply engine extracts bundles to temporary staging, fsyncs, cryptographically verifies magic 0x534D4B4D, layout 0x42554E44, and SHA-256 digests against the manifest, atomically renames files into place, replays WAL records, and advances the High-Water Mark last. A crash during transfer or apply leaves existing data intact and corrupt bytes unpublishable.

🛡️ Cell Failover, Fencing & Coordination (spector.failover.*, spector.coordinator.*, spector.control-store.*)¶

Spector Synapse provides lease coordinator election, monotonic write fence tokens, automated monitored failover, dynamic ring reloads, and maintenance draining across multi-node cells.

• Monotonic Fence Tokens: Every write request carries the current namespace fence token (X-Spector-Fence). Writing to a partitioned, stale, or superseded owner node is rejected with HTTP 409 Conflict (FENCED). Validation is in-memory, allocation-free, and I/O-free.
• Precedence Over Hash: Active override leases in the control store strictly supersede consistent hash ring resolution. Hash ring resolution resumes only after safe lease expiration or explicit release.
• Verify Before Promote: Survivor candidate data is cryptographically and structurally validated before promotion. If validation fails, promotion is refused to prevent defect propagation into data loss.
• In-Flight Request Expectations: In-flight requests holding region leases on a failed node die with the process. Clients receive connection drops or refusals and retry transparently against the newly promoted survivor.
• Volatile Memory Handling: In-memory working state is lost during ungraceful node crash. Durable committed records are preserved with zero data loss (RPO = 0). Clients detect failover transitions via the incremented namespace epoch.

☸️ Kubernetes Cell Topology & Resource Isolation (spector.cell.*, spector.pager.*)¶

Spector cells separate cluster responsibilities into three distinct roles: owners (authoritative single-writers managing live namespaces and local NVMe storage), replicas (read/warm followers maintaining pre-warmed namespace subsets), and gateways (stateless API routers).

• Filesystem Selection: Local storage directories must be formatted with XFS or ext4. Off-heap memory-mapped (mmap) page cache flushing and durability guarantees are strictly validated on these filesystems. Non-standard filesystems trigger startup warnings.
• Map Count Arithmetic: Required memory mappings are calculated using layout arithmetic: \(\approx (\text{hotCap} \times 2) + \text{WAL segments} + \text{overhead}\). Each open namespace utilizes ~2 maps (the runtime bundle and the active partition bundle).
• Kernel Tuning Parameters:
  • vm.max_map_count: Set to at least 262144 (or 1048576 for dense multi-tenant deployments). An under-provisioned host fails readiness probes at startup.
  • vm.swappiness: Set to 1 to prevent memory-mapped working sets from paging to swap disk.
  • ulimit -n: Set to at least 262144 to prevent descriptor exhaustion under high connection and bundle volume.
  • memlock: Configured as unlimited for memory-locking preambles.
• Memory Parity Invariant: StatefulSet owner configurations must enforce requests.memory == limits.memory == 16Gi (with 2 GiB heap). This prevents Linux cgroups v2 from evicting clean page-cache pages prematurely under transient memory pressure.
• Inter-Pod Anti-Affinity: Owner pods require strict host anti-affinity (topologyKey: kubernetes.io/hostname) to prevent multiple owner replicas from co-locating on the same physical compute node.

🛡️ Cell Disaster Recovery & Compliance (spector.dr.*)¶

Disaster recovery provides cross-cell resilience and snapshot exports to S3-compatible object stores for mutable active namespaces. Remote uploads pace background bandwidth to avoid competing with foreground client traffic, enforce geographic residency boundaries, and preserve verifiable bundle integrity.

• Atomic Manifest Publication: Partition bundles and payload files are transferred to object storage first. The manifest.json descriptor is uploaded strictly last to guarantee atomic visibility. Restorations ignore partial or unmanifested epochs.
• Strict Verification Before Serving: Restoring nodes validate bundle preamble magic (0x534D4B4D), layout identifiers (0x42554E44), and SHA-256 payload checksums. Corrupted or mismatched snapshots are strictly rejected and refuse to serve.
• Explicit Standby Promotion: Standby cell activation requires human decision-making with mandatory --force acknowledgment and audit trail logging. Automatic failover across cell boundaries is forbidden to eliminate split-brain write corruption.
• Compliance Erasure & Legal Hold: Tenant deletion executes physical recursive wiping across local NVMe storage and prefix purging across cloud object storage. Namespaces tagged with legal hold strictly refuse deletion.
• Data Residency Enforcement: Org-to-cell residency bindings are enforced during both export and restore. Any transfer violating regional jurisdiction triggers hard refusal.

📊 Memory Analytics (spector.memory.analytics.*, spector.memory.stats.*)¶

Spector Synapse supports namespace-isolated memory analytics snapshots, periodic telemetry persistence, and stats caching (ADR-0083).

# ── Memory Analytics (ADR-0083) ────────────────────────
spector:
  memory:
    analytics:
      history:
        enabled: true          # set false to disable H2 snapshot writes
        interval: 10000        # ms between snapshot flushes
      instance-id: local       # pod identity for composite PK
    stats:
      cache-ttl: 5s            # namespace stats cache TTL

🔗 See Also¶

• Disaster Recovery Runbook — Step-by-step failover, rehydration, and verification playbook

• Performance Tuning — Benchmarks and optimization strategies

• Architecture Overview — How configuration affects system behavior

• Distributed Mode — Cluster-specific configuration

• GPU Acceleration — GPU setup requirements