feat(hippocampal): v2 memory system — glossary, namespaces, rollback, auto-inject, daemon

COMMIT_MSG.txt

@@ -0,0 +1,32 @@
+feat(hippocampal): v2 memory system — glossary, namespaces, rollback, auto-inject, daemon
+
+Complete v2 overhaul of the hippocampal memory system:
+
+Storage layer (crates/memory/):
+- Schema migration system (schema_version table) for safe upgrades
+- Namespaces table for workspace/project-level isolation
+- Glossary table + fact_glossary/entity_glossary for keyword tagging
+- fact_versions table for rollback support (every update saves history)
+- New store methods: namespace CRUD, glossary CRUD, fact versioning,
+  rollback, get_memory_stats, namespace-scoped queries
+- 5 new unit tests (namespace, glossary, versioning, rollback, stats)
+
+Agent tools (crates/tui/src/tools/):
+- memorize: new glossary_tags[] and namespace input parameters
+- recall: new namespace and glossary_tag filtering, glossary tag display
+- consolidate: NEW tool — stats/rollback/prune/merge actions
+
+System integration (crates/tui/src/core/engine.rs):
+- memory_context_block(): injects top-8 important facts + usage guidance
+  into the system prompt on every refresh (cross-session awareness)
+- memory_daemon_loop(): background tokio task, prunes low-importance
+  facts every 6 hours, logs memory statistics
+
+Prompt guidance (crates/tui/src/prompts/):
+- hippocampal_guidance.md: tells the model to auto-call memorize when
+  it discovers architecture decisions, user preferences, etc.
+
+Design doc: MEMORY_DESIGN.md covers architecture decisions, comparison
+with Mem0 / Nocturne Memory / Awareness-Local, and future roadmap.
+
+Refs: #2933, #3234

Cargo.toml

@@ -8,6 +8,7 @@ members = [
     "crates/execpolicy",
     "crates/hooks",
     "crates/mcp",
+    "crates/memory",
     "crates/protocol",
     "crates/release",
     "crates/secrets",

MEMORY_DESIGN.md

@@ -0,0 +1,160 @@
+# Hippocampal Memory System — Design Document
+
+> **PR**: #2933 (v2 update)
+> **Discussion**: #3234
+> **Author**: @cy2311
+
+## 1. Motivation
+
+CodeWhale's 1M-token context provides ample short-term workspace, but no mechanism for
+**cross-session recall**. Every `/compact` or new session starts blank — architecture
+decisions, user preferences, and project conventions learned in previous sessions are lost.
+
+The Hippocampal Memory System provides a persistent, structured, SQLite-backed memory store
+that survives compaction and spans sessions.
+
+## 2. Design Principles
+
+| Principle | Explanation |
+|---|---|
+| **Local-first** | Memory lives in a local SQLite file. No cloud dependency. |
+| **Explicit + Automatic** | Agent can explicitly `memorize`/`recall`, AND the system auto-injects context. |
+| **Structured, not flat** | Entity-relation graph + FTS5 full-text search. |
+| **Tiered importance** | Facts scored 0.0–1.0. High-importance facts survive pruning. |
+| **Opt-in** | Disabled by default, enabled via `memory_db_path` config. |
+
+## 3. Architecture
+
+```
+Layer 3: System Integration
+  ├─ Prompt Injection: top-8 facts into `<memory_context>` on every refresh
+  └─ Memory Daemon: tokio background task, 6h prune + stats
+
+Layer 2: Agent Tools
+  ├─ memorize(content, importance, entity, glossary_tags[], namespace)
+  ├─ recall(query, namespace, glossary_tag, include_graph, limit)
+  └─ consolidate(stats|rollback|prune|merge)
+
+Layer 1: Storage (crates/memory/)
+  ├─ entities + relations (entity graph)
+  ├─ facts + facts_fts (FTS5 full-text search)
+  ├─ glossary + fact_glossary + entity_glossary (keyword system)
+  ├─ namespaces (workspace isolation)
+  ├─ fact_versions (rollback support)
+  └─ SQLite (bundled via rusqlite, zero extra deps)
+```
+
+## 4. Storage Design
+
+### 4.1 Why SQLite + FTS5 (not Vector DB)
+
+| Approach | Pros | Cons |
+|---|---|---|
+| **SQLite + FTS5** (selected) | Zero deps (bundled via rusqlite), ACID, FTS5 fast keyword search, <10MB | No semantic search |
+| **Vector DB** | Semantic search | Heavy deps, service needed, overkill for structured facts |
+| **Flat JSON** | Simple | No indexing, no concurrent access |
+
+**Decision**: For an AI coding agent, memory retrieval is primarily keyword-driven
+("what was the decision about database schema?"). FTS5 provides instant prefix-match search.
+Vector search can be added later as an optional layer on top.
+
+### 4.2 Schema Maps
+
+**v1 (original - preserved)**:
+- `entities(id, kind, name, description, created_at, updated_at)`
+- `relations(id, source_id, target_id, kind, strength, created_at, session_id)`
+- `facts(id, entity_id, content, source, importance, created_at, session_id)`
+- `facts_fts` — FTS5 virtual table
+
+**v2 additions**:
+- `namespaces(id, name, description, created_at, updated_at)` — workspace isolation
+- `glossary(id, term, definition, category, namespace_id)` — keyword/tag system
+- `fact_glossary(fact_id, glossary_id)` + `entity_glossary(entity_id, glossary_id)` — M:N links
+- `fact_versions(id, fact_id, content, source, importance, version, session_id)` — rollback
+
+### 4.3 Migration System
+
+Schema changes via `schema_version` table. Each migration is a numbered function
+that runs exactly once (safe for existing databases).
+
+### 4.4 Namespace Isolation
+
+Workspaces get namespaces (`workspace:/path/to/project`). Facts/entities can be scoped,
+enabling multi-project isolation from a single DB file.
+
+### 4.5 Fact Versioning & Rollback
+
+Every `update_fact` saves the previous version. The `consolidate rollback` action
+restores any version. Version numbers are monotonic; rollback preserves the full audit trail.
+
+## 5. Tool Design
+
+### memorize — Explicit Storage
+
+Input: `{ content, importance (0.0-1.0), entity_kind?, entity_name?, glossary_tags?, namespace? }`
+- Creates/updates entity, creates fact (version 1), links glossary tags
+- Auto-approved (low risk)
+
+### recall — Structured Retrieval
+
+Input: `{ query, limit, namespace?, glossary_tag?, include_graph? }`
+- FTS5 full-text search ordered by importance DESC
+- Optional namespace + glossary tag filtering
+- Returns facts with linked entities, relations, and tags
+- Fallback: top important facts when query returns empty
+
+### consolidate — Maintenance
+
+Input: `{ action (stats|rollback|prune|merge), fact_id?, target_version?, importance_threshold?, older_than_days? }`
+- stats: memory usage report
+- prune: delete low-importance facts (active forgetting)
+- rollback: restore fact to previous version
+- merge: deduplicate identical facts
+
+## 6. System Integration
+
+### Prompt Injection
+
+On every `refresh_system_prompt()`, the engine queries the top 8 facts and injects:
+```
+<memory_context>
+1. [imp=0.9] Service X uses PostgreSQL...
+   tags: [database, postgresql]
+...
+Automatically call memorize when you discover architecture decisions...
+</memory_context>
+```
+
+### Auto-Memorize Guidance
+
+System prompt tells the model to auto-call `memorize` for architecture decisions,
+user preferences, project conventions, etc. Zero extra API calls.
+
+### Background Daemon
+
+`tokio::spawn` task every 6 hours: prune facts (importance < 0.3, age > 30 days) + log stats.
+
+## 7. Comparison with Alternatives
+
+| Feature | Codewhale v2 | Mem0 | Nocturne Memory | Awareness-Local |
+|---|---|---|---|---|
+| Storage | SQLite+FTS5 | SQLite+Vector | SQLite+FTS5 | SQLite |
+| Entity Graph | ✅ | ❌ | ✅ | ❌ |
+| Importance Scoring | ✅ 0.0-1.0 | ❌ | ❌ | ❌ |
+| Active Forgetting | ✅ prune+daemon | ❌ | ❌ | ❌ |
+| Fact Versioning | ✅ versions+rollback | ❌ | ❌ | ❌ |
+| Glossary/Tags | ✅ glossary table | ❌ | ✅ keywords | ❌ |
+| Namespace Isolation | ✅ | ❌ | ✅ v2.0 | ❌ |
+| Migration System | ✅ versioned | ❌ | ✅ 13 migrations | ❌ |
+| Background Daemon | ✅ 6h interval | ❌ | ❌ | ✅ daemon.mjs |
+| Prompt Injection | ✅ memory_context | API only | MCP auto | MCP auto |
+
+## 8. Future Roadmap
+
+**Short-term**: Vector search layer, MCP Server mode (OpenClaw compatible),
+config integration (`[memory.hippocampal]` in config.toml), `/memory` CLI.
+
+**Medium-term**: Semantic merge (LLM dedup), access-frequency importance scoring,
+memory decay curves, export/import, web UI.
+
+**Long-term**: Multi-user shared namespaces, adaptive pruning, temporal reasoning.

PR_DESCRIPTION.md

@@ -0,0 +1,63 @@
+## Hippocampal Memory v2 — Glossary, Namespaces, Rollback, Auto-Inject, Background Daemon
+
+This PR upgrades the hippocampal memory system from v1 (basic entity graph + FTS5) to a full-featured cross-session memory layer.
+
+### What's New
+
+**Storage Layer** (`crates/memory/`)
+- Schema migration system with `schema_version` table for safe upgrades
+- `namespaces` table — workspace/project-level isolation
+- `glossary` table — keyword/tag system with many-to-many links to facts and entities
+- `fact_versions` table — full version history, enabling rollback to any previous version
+- 21 new store methods including namespace CRUD, glossary CRUD, fact versioning, rollback, and memory statistics
+
+**Agent Tools** (`crates/tui/src/tools/`)
+- **`memorize`** — new `glossary_tags[]` and `namespace` parameters for structured tagging
+- **`recall`** — new `namespace` and `glossary_tag` filtering, glossary tag display in results
+- **`consolidate`** — **new tool** with four actions:
+  - `stats` — memory usage report
+  - `rollback` — restore a fact to a previous version
+  - `prune` — delete low-importance facts (active forgetting)
+  - `merge` — deduplicate identical facts
+
+**System Integration** (`crates/tui/src/core/engine.rs`)
+- Auto-injects top-8 important facts into the system prompt as `<memory_context>` block on every refresh — the model sees cross-session knowledge without an explicit `recall` call
+- Background memory daemon via `tokio::spawn`: prunes low-importance facts (importance < 0.3, age > 30 days) every 6 hours, logs memory statistics
+- System prompt guidance telling the model to auto-call `memorize` when it discovers architecture decisions, user preferences, etc.
+
+### Design Document
+
+See `MEMORY_DESIGN.md` for:
+- Architecture overview (3-layer: storage → tools → integration)
+- Why SQLite + FTS5 (not vector DB) — comparison table
+- Schema design rationale
+- Comparison with Mem0 (58k ⭐), Nocturne Memory, Awareness-Local
+- Future roadmap
+
+### Implementation Status
+
+- `cargo check -p codewhale-memory` ✅
+- `cargo check -p codewhale-tui` ✅
+- 9 unit tests in `crates/memory/src/store.rs` (4 original + 5 new)
+- Unit test execution blocked by disk space on dev machine
+
+### Changed Files
+
+```
+ M Cargo.lock
+ M crates/memory/Cargo.toml                    (+tracing dep)
+ M crates/memory/src/lib.rs                    (new exports)
+ M crates/memory/src/schema.rs                 (migration system v1→v2)
+ M crates/memory/src/store.rs                  (21 new methods, 5 tests)
+ M crates/tui/src/core/engine.rs               (prompt injection + daemon)
+ M crates/tui/src/core/engine/tool_setup.rs    (consolidate registration)
+ M crates/tui/src/tools/memorize.rs            (+glossary_tags, +namespace)
+ M crates/tui/src/tools/mod.rs                 (+pub mod consolidate)
+ M crates/tui/src/tools/recall.rs              (+namespace/glossary filtering)
+ M crates/tui/src/tools/registry.rs            (+with_consolidate_tool)
+?? MEMORY_DESIGN.md                            (design document)
+?? crates/tui/src/prompts/hippocampal_guidance.md
+?? crates/tui/src/tools/consolidate.rs         (new tool)
+```
+
+Refs: #2933, Discussion #3234

crates/memory/Cargo.toml

@@ -0,0 +1,20 @@
+[package]
+name = "codewhale-memory"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+description = "Hippocampal memory system — structured entity graph with cross-session recall"
+
+[dependencies]
+anyhow.workspace = true
+chrono.workspace = true
+rusqlite.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+sha2.workspace = true
+tracing.workspace = true
+uuid.workspace = true
+
+[dev-dependencies]
+tempfile = "3"

crates/memory/src/lib.rs

@@ -0,0 +1,24 @@
+//! # Codewhale Hippocampal Memory System
+//!
+//! A structured, SQLite-backed memory store that enables the agent to remember
+//! facts, entities, and relationships across sessions — the foundation for
+//! true infinite-context and cross-session recall.
+//!
+//! ## Core Concepts
+//!
+//! - **Entities**: Files, PRs, issues, concepts, people, decisions — anything
+//!   the model might need to reference later.
+//! - **Relations**: Directed edges connecting entities (e.g. `dispatch.rs` is
+//!   `part_of` `PR #2933`).
+//! - **Facts**: Standalone factual statements, optionally bound to an entity.
+//!   Stored with an importance score (0.0–1.0) for active forgetting.
+//! - **Namespaces**: Workspace/project-level isolation for multi-repo setups.
+//! - **Glossary**: Keyword/tags for cross-referencing facts and entities.
+//! - **Fact Versions**: Version history enabling rollback to previous states.
+
+pub mod schema;
+pub mod store;
+
+pub use store::{
+    Entity, Fact, FactVersion, GlossaryTerm, MemoryStats, MemoryStore, Namespace, Relation,
+};