metal-kompanion/docs/memory-architecture.md

Memory Architecture Roadmap (2025-10-15)

Current Snapshot

• PgDal remains an in-memory stub; libpq/pqxx are not linked and there is no HAVE_PG compile guard.
• contract_memory now builds via tests/contract_memory.cpp as a dedicated CTest executable (stub DAL still backing it).
• MCP handlers directly invoke the stub DAL and perform ad-hoc JSON parsing.
• No resources/ descriptors exist for episodic memory APIs or the semantic sync workflow.

1. CTest Target: contract_memory

1. Keep contract_memory.cpp focused on exercising PgDal write/read surfaces; expand as DAL features land.
2. Ensure the executable runs without Postgres by defaulting to stub://memory when PG_DSN is absent.
3. Layer follow-up assertions once pqxx-backed code exists (e.g., detect HAVE_PG and connect to real DB in CI).

2. DAL Prepared Statements (HAVE_PG)

Dependencies

• System packages: libpq headers and libpqxx (>= 7.8).
• CMake: find_package(PostgreSQL REQUIRED) and find_package(pqxx REQUIRED); link PostgreSQL::PostgreSQL and pqxx::pqxx.

Implementation Steps

1. Extend src/dal/CMakeLists.txt to define HAVE_PG and link the libraries when detection succeeds.
2. Enhance PgDal to manage both modes:
   • stub:// DSN → in-memory containers (status quo).
   • Other DSNs → establish pqxx::connection, store it via std::unique_ptr, and call conn->prepare(...) during connect.
3. Prepare statements for: ensure_namespace, upsert_item, insert_chunk, insert_embedding, search_text, search_vector, hybrid_search.
4. Implement transactional helpers using pqxx::work, with RAII to guarantee commit()/abort() pairing.
5. Translate pqxx exceptions into std::runtime_error with context so MCP handlers can emit useful error JSON.
6. Document required environment variables (PG_DSN, optional PGSSLMODE) plus migration expectations in docs/dal-skeleton.md.

3. MCP resources/* & Episodic→Semantic Sync

Directory Layout

• Create resources/memory/kom.memory.v1/ for tool descriptors and schema fragments:
  • episodic.json – raw conversation timeline.
  • semantic.json – chunked embeddings metadata.
  • jobs/semantic_sync.json – background job contract.

Design Highlights

1. Episodic resource fields: namespace, thread_id, speaker, content, sensitivity, tags, created_at.
2. Semantic resource references episodic items (episodic_id, chunk_id, model, dim, vector_ref).
3. DAL sync job flow:
   • Locate episodic rows with embedding_status='pending' (and sensitivity!='secret').
   • Batch call embedder(s); write memory_chunks + embeddings.
   • Mark episodic rows as embedding_status='done', capture audit entries (e.g., ledger append).
4. Expose a placeholder MCP tool kom.memory.v1.sync_semantic that enqueues or executes the job.
5. Note TTL and privacy requirements; skip items with expires_at in the past or flagged secret.

Ξlope Alignment Notes (2025-10-15)

• Episodic resources capture resonance links and identity hints so the Librarian layer (see elope/doc/architecture_memory.md) can strengthen cross-agent patterns without raw content sharing.
• Semantic resources surface identity_vector and semantic_weight, enabling supersemantic indexing once crystallization occurs.
• jobs/semantic_sync maintains cursor_event_id and skips sensitivity=secret, mirroring the elope crystallization guidance in /tmp/mem-elope.txt.

4. hybrid_search_v1 with pgvector

SQL Components

1. Update migrations (sql/pg/001_init.sql) to include:
   • tsvector generated column or expression for lexical search.
   • GIN index on the lexical field (either to_tsvector or pg_trgm).
   • Per-model ivfflat index on embeddings.vector.
2. Prepared statements:
   • Text: SELECT id, ts_rank_cd(...) AS score FROM memory_items ... WHERE namespace_id=$1 AND text_query=$2 LIMIT $3.
   • Vector: SELECT item_id, 1 - (vector <=> $2::vector) AS score FROM embeddings ... WHERE namespace_id=$1 ORDER BY vector <-> $2 LIMIT $3.
3. Merge results in C++ with Reciprocal Rank Fusion or weighted sum, ensuring deterministic ordering on ties.

Handler Integration

1. Extend PgDal::hybridSearch to dispatch to the prepared statements when HAVE_PG is defined; reuse in-memory fallback otherwise.
2. Return richer matches (id, score, optional chunk text) to satisfy MCP response schema.
3. Update HandlersMemory::search_memory to surface the new scores and annotate whether lexical/vector contributed (optional metadata).
4. Add a contract test scenario once pqxx-backed execution is available (requires live Postgres fixture later).

5. Secret Handling, Snapshots, and CLI Hooks

• Secret propagation: episodic sensitivity + embeddable flags gate embedding generation. DAL queries will add predicates (metadata->>'sensitivity' != 'secret') before hybrid search.
• Snapshots: episodic entries with content_type = snapshot reference durable artifacts; sync summarises them into semantic text while retaining snapshot_ref for CLI inspection.
• Hybrid policy: pgSearchVector will filter by caller capability (namespace scope, secret clearance) before ranking; contract tests must assert omission of secret-tagged items.
• CLI sketch: plan for a Qt QCoreApplication tool (kom_mctl) exposing commands to list namespaces, tail episodic streams, trigger sync_semantic, and inspect resonance graphs—all wired through the new prepared statements.
• Observability: CLI should read the jobs/semantic_sync state block to display cursors, pending counts, and last error logs; dry-run mode estimates embeddings without committing.

Next-Step Checklist

• Detect pqxx via CMake and plumb HAVE_PG.
• Normalize contract_memory CTest target and remove stale library target.
• Author resources/memory/ descriptors and sync job outline.
• Extend DAL header to carry prepared-statement aware APIs (may introduce new structs).
• Update docs/mcp-memory-api.md to mention episodic sync + hybrid search fields.
• Create follow-up acf subtasks when concrete implementation begins (pgvector migration, scheduler hook, runtime wiring).