70 lines
4.6 KiB
Markdown
70 lines
4.6 KiB
Markdown
# Memory Architecture Roadmap (2025-10-15)
|
||
|
||
## Current Snapshot
|
||
- `PgDal` remains an in-memory stub; `libpq`/`pqxx` are not linked and there is no `HAVE_PG` compile guard.
|
||
- `contract_memory` now builds via `tests/contract_memory.cpp` as a dedicated CTest executable (stub DAL still backing it).
|
||
- MCP handlers directly invoke the stub DAL and perform ad-hoc JSON parsing.
|
||
- No `resources/` descriptors exist for episodic memory APIs or the semantic sync workflow.
|
||
|
||
## 1. CTest Target: `contract_memory`
|
||
1. Keep `contract_memory.cpp` focused on exercising `PgDal` write/read surfaces; expand as DAL features land.
|
||
2. Ensure the executable runs without Postgres by defaulting to `stub://memory` when `PG_DSN` is absent.
|
||
3. Layer follow-up assertions once pqxx-backed code exists (e.g., detect `HAVE_PG` and connect to real DB in CI).
|
||
|
||
## 2. DAL Prepared Statements (`HAVE_PG`)
|
||
**Dependencies**
|
||
- System packages: `libpq` headers and `libpqxx` (>= 7.8).
|
||
- CMake: `find_package(PostgreSQL REQUIRED)` and `find_package(pqxx REQUIRED)`; link `PostgreSQL::PostgreSQL` and `pqxx::pqxx`.
|
||
|
||
**Implementation Steps**
|
||
1. Extend `src/dal/CMakeLists.txt` to define `HAVE_PG` and link the libraries when detection succeeds.
|
||
2. Enhance `PgDal` to manage both modes:
|
||
- `stub://` DSN → in-memory containers (status quo).
|
||
- Other DSNs → establish `pqxx::connection`, store it via `std::unique_ptr`, and call `conn->prepare(...)` during `connect`.
|
||
3. Prepare statements for: `ensure_namespace`, `upsert_item`, `insert_chunk`, `insert_embedding`, `search_text`, `search_vector`, `hybrid_search`.
|
||
4. Implement transactional helpers using `pqxx::work`, with RAII to guarantee `commit()`/`abort()` pairing.
|
||
5. Translate `pqxx` exceptions into `std::runtime_error` with context so MCP handlers can emit useful error JSON.
|
||
6. Document required environment variables (`PG_DSN`, optional `PGSSLMODE`) plus migration expectations in `docs/dal-skeleton.md`.
|
||
|
||
## 3. MCP `resources/*` & Episodic→Semantic Sync
|
||
**Directory Layout**
|
||
- Create `resources/memory/kom.memory.v1/` for tool descriptors and schema fragments:
|
||
- `episodic.json` – raw conversation timeline.
|
||
- `semantic.json` – chunked embeddings metadata.
|
||
- `jobs/semantic_sync.json` – background job contract.
|
||
|
||
**Design Highlights**
|
||
1. Episodic resource fields: `namespace`, `thread_id`, `speaker`, `content`, `sensitivity`, `tags`, `created_at`.
|
||
2. Semantic resource references episodic items (`episodic_id`, `chunk_id`, `model`, `dim`, `vector_ref`).
|
||
3. DAL sync job flow:
|
||
- Locate episodic rows with `embedding_status='pending'` (and `sensitivity!='secret'`).
|
||
- Batch call embedder(s); write `memory_chunks` + `embeddings`.
|
||
- Mark episodic rows as `embedding_status='done'`, capture audit entries (e.g., ledger append).
|
||
4. Expose a placeholder MCP tool `kom.memory.v1.sync_semantic` that enqueues or executes the job.
|
||
5. Note TTL and privacy requirements; skip items with `expires_at` in the past or flagged secret.
|
||
|
||
## 4. `hybrid_search_v1` with `pgvector`
|
||
**SQL Components**
|
||
1. Update migrations (`sql/pg/001_init.sql`) to include:
|
||
- `tsvector` generated column or expression for lexical search.
|
||
- `GIN` index on the lexical field (either `to_tsvector` or `pg_trgm`).
|
||
- Per-model `ivfflat` index on `embeddings.vector`.
|
||
2. Prepared statements:
|
||
- Text: `SELECT id, ts_rank_cd(...) AS score FROM memory_items ... WHERE namespace_id=$1 AND text_query=$2 LIMIT $3`.
|
||
- Vector: `SELECT item_id, 1 - (vector <=> $2::vector) AS score FROM embeddings ... WHERE namespace_id=$1 ORDER BY vector <-> $2 LIMIT $3`.
|
||
3. Merge results in C++ with Reciprocal Rank Fusion or weighted sum, ensuring deterministic ordering on ties.
|
||
|
||
**Handler Integration**
|
||
1. Extend `PgDal::hybridSearch` to dispatch to the prepared statements when `HAVE_PG` is defined; reuse in-memory fallback otherwise.
|
||
2. Return richer matches (id, score, optional chunk text) to satisfy MCP response schema.
|
||
3. Update `HandlersMemory::search_memory` to surface the new scores and annotate whether lexical/vector contributed (optional metadata).
|
||
4. Add a contract test scenario once pqxx-backed execution is available (requires live Postgres fixture later).
|
||
|
||
## Next-Step Checklist
|
||
- [ ] Detect pqxx via CMake and plumb `HAVE_PG`.
|
||
- [x] Normalize contract_memory CTest target and remove stale library target.
|
||
- [ ] Author `resources/memory/` descriptors and sync job outline.
|
||
- [ ] Extend DAL header to carry prepared-statement aware APIs (may introduce new structs).
|
||
- [x] Update `docs/mcp-memory-api.md` to mention episodic sync + hybrid search fields.
|
||
- [ ] Create follow-up acf subtasks when concrete implementation begins (pgvector migration, scheduler hook, runtime wiring).
|