metal-kompanion/docs/memory-architecture.md

Memory Architecture Roadmap (2025-10-15)

Current Snapshot

• PgDal now prefers Qt6/QSql (QPSQL) with an in-memory fallback for stub:// DSNs; schema migrations live in db/init/.
• kompanion --init guides DSN detection (psql socket probing), applies migrations, and persists config via ~/.config/kompanionrc.
• MCP handlers still parse JSON manually but leverage the shared DAL; resource descriptors under resources/memory/kom.memory.v1/ capture episodic/semantic contracts.
• Contract tests (contract_memory, contract_mcp_tools, mcp_memory_exchange) validate the Qt-backed DAL and MCP handlers.

1. CTest Target: contract_memory

1. Keep contract_memory.cpp focused on exercising PgDal write/read surfaces; expand as DAL features land.
2. Ensure the executable runs without Postgres by defaulting to stub://memory when PG_DSN is absent.
3. Layer follow-up assertions once the QSql path is exercised end-to-end (CI can target the packaged test database).

2. DAL (Qt6/QSql) Evolution

Dependencies

• Qt6 (Core, Sql) with the QPSQL driver available at runtime.
• KDE Frameworks ConfigCore for persisting DSNs in kompanionrc.

Implementation Steps

1. Parse libpq-style DSNs with QUrl, open QSqlDatabase connections when the DSN is not stub://, and maintain the existing in-memory fallback for tests.
2. Use QSqlQuery INSERT ... RETURNING statements for namespaces, items, chunks, and embeddings; emit vector literals ([0.1,0.2]) when targeting pgvector columns.
3. Surface detailed QSqlError messages (throwing std::runtime_error) so MCP handlers and the CLI can report actionable failures.
4. Share configuration between CLI and MCP runners via KConfig (Database/PgDsn), seeded through the new kompanion --init wizard.

3. MCP resources/* & Episodic→Semantic Sync

Directory Layout

• Create resources/memory/kom.memory.v1/ for tool descriptors and schema fragments:
  • episodic.json – raw conversation timeline.
  • semantic.json – chunked embeddings metadata.
  • jobs/semantic_sync.json – background job contract.

Design Highlights

1. Episodic resource fields: namespace, thread_id, speaker, content, sensitivity, tags, created_at.
2. Semantic resource references episodic items (episodic_id, chunk_id, model, dim, vector_ref).
3. DAL sync job flow:
   • Locate episodic rows with embedding_status='pending' (and sensitivity!='secret').
   • Batch call embedder(s); write memory_chunks + embeddings.
   • Mark episodic rows as embedding_status='done', capture audit entries (e.g., ledger append).
4. Expose a placeholder MCP tool kom.memory.v1.sync_semantic that enqueues or executes the job.
5. Note TTL and privacy requirements; skip items with expires_at in the past or flagged secret.

Ξlope Alignment Notes (2025-10-15)

• Episodic resources capture resonance links and identity hints so the Librarian layer (see elope/doc/architecture_memory.md) can strengthen cross-agent patterns without raw content sharing.
• Semantic resources surface identity_vector and semantic_weight, enabling supersemantic indexing once crystallization occurs.
• jobs/semantic_sync maintains cursor_event_id and skips sensitivity=secret, mirroring the elope crystallization guidance in /tmp/mem-elope.txt.

4. hybrid_search_v1 with pgvector

SQL Components

1. Update migrations (sql/pg/001_init.sql) to include:
   • tsvector generated column or expression for lexical search.
   • GIN index on the lexical field (either to_tsvector or pg_trgm).
   • Per-model ivfflat index on embeddings.vector.
2. Prepared statements:
   • Text: SELECT id, ts_rank_cd(...) AS score FROM memory_items ... WHERE namespace_id=$1 AND text_query=$2 LIMIT $3.
   • Vector: SELECT item_id, 1 - (vector <=> $2::vector) AS score FROM embeddings ... WHERE namespace_id=$1 ORDER BY vector <-> $2 LIMIT $3.
3. Merge results in C++ with Reciprocal Rank Fusion or weighted sum, ensuring deterministic ordering on ties.

Handler Integration

1. Ensure PgDal::hybridSearch delegates to SQL-based lexical/vector search when a database connection is active, reusing the in-memory fallback only for stub://.
2. Return richer matches (id, score, optional chunk text) to satisfy MCP response schema.
3. Update HandlersMemory::search_memory to surface the new scores and annotate whether lexical/vector contributed (optional metadata).
4. Exercise hybrid queries in contract tests against the packaged test database (db/scripts/create-test-db.sh).

5. Secret Handling, Snapshots, and CLI Hooks

• Secret propagation: episodic sensitivity + embeddable flags gate embedding generation. DAL queries will add predicates (metadata->>'sensitivity' != 'secret') before hybrid search.
• Snapshots: episodic entries with content_type = snapshot reference durable artifacts; sync summarises them into semantic text while retaining snapshot_ref for CLI inspection.
• Hybrid policy: pgSearchVector will filter by caller capability (namespace scope, secret clearance) before ranking; contract tests must assert omission of secret-tagged items.
• CLI sketch: plan for a Qt QCoreApplication tool (kom_mctl) exposing commands to list namespaces, tail episodic streams, trigger sync_semantic, and inspect resonance graphs—all wired through the new prepared statements.
• Observability: CLI should read the jobs/semantic_sync state block to display cursors, pending counts, and last error logs; dry-run mode estimates embeddings without committing.
• Activation parity: Long term, mirror the KDE akonadiclient/akonadi-console pattern—Kompanion CLI doubles as an MCP surface today and later as a DBus-activated helper so tools can be socket-triggered into the memory service.
• KConfig defaults: kom_mcp and kompanion load Database/PgDsn from ~/.config/kompanionrc (see docs/configuration.md) when PG_DSN is unset, keeping deployments kioskable.
• CLI UX: kompanion --init guides first-run setup (auto-detects databases, applies schemas); -I/--interactive keeps a JSON REPL open, and -V/--verbose echoes request/response streams for future HTTP transport parity.

Next-Step Checklist

• Promote Qt6/QSql backend (QPSQL) as default DAL; retain stub:// fallback for tests.
• Normalize contract_memory CTest target and remove stale library target.
• Author resources/memory/ descriptors and sync job outline.
• Extend DAL header to expose richer query structs (filters, pagination, secret handling).
• Update docs/mcp-memory-api.md to mention episodic sync + hybrid search fields.
• Create follow-up acf subtasks when concrete implementation begins (pgvector migration, scheduler hook, runtime wiring).