diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0e96211..205df70 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,2 +1,92 @@ # Contributing to Engram -We welcome pull requests! Since this is a high-performance vector database, please ensure all new features merge cleanly with the `.leg` logophysical file system. Before submitting a PR, ensure you run `cargo clippy` and `cargo test`. + +We welcome contributions to Engram. Since this is a hardware-native memory engine with a strict binary format, there are a few rules to follow to keep the physics correct. + +--- + +## Development Setup + +```bash +git clone https://github.com/staticroostermedia-arch/engram.git +cd engram + +# Build everything +cargo build --workspace + +# Run all tests +cargo test --workspace + +# Run clippy (required before PR) +cargo clippy --workspace -- -D warnings +``` + +--- + +## Architecture Overview + +The workspace has four crates: + +| Crate | Role | +|---|---| +| `engram-core` | The HolographicBlock format, VSA operators (OP_ADD, OP_BIND), BLAKE3 Merkle chain, CRS/ADR physics | +| `engram-server` | MCP server, background daemon (file watcher + NREM consolidation + health watchdog), REST API | +| `engram-cli` | CLI binary — wraps `engram-core` for direct manifold management | +| `engram-gpu` | CUDA/ROCm/Metal/WebGPU backends for parallel ANN search | + +--- + +## Critical Rules + +### 1. Never Break the `.leg3` Format +The `HolographicBlock` struct in `engram-core/src/lib.rs` is a **fixed 262,144-byte C-struct**. Fields are at fixed byte offsets. Any change that alters struct layout will silently corrupt every existing manifold on disk. Changes to this struct require a format version bump and a migration tool. + +### 2. Use `mcp_engram_update` — Never `forget` + `remember` +When modifying an existing memory block, always use the `update` path. `forget` + `remember` destroys the block's Lyapunov drift history (Merkle chain, CRS trajectory, ADR state). The `update` path preserves this history and applies a stability check before accepting the new content. + +### 3. VSA Operator Correctness +`OP_ADD` is commutative superposition. `OP_BIND` is Hadamard product (invertible, non-commutative when combined with `OP_SHIFT`). Do not use scalar multiplication in place of `OP_INVERT`. See `crates/engram-core/src/vsa.rs` for the canonical implementations. + +### 4. CRS Is Not a User-Settable Field +The Coherence-Reliability Score is computed entirely by the ADR thermodynamic gate from the block's Lyapunov drift. Do not set it manually outside of `pin()` (which locks it at 1.0) or the genesis seeding path. + +### 5. The Daemon Has Three Loops — Don't Break Any of Them +`crates/engram-server/src/daemon.rs` runs three independent async loops: +- **File Watcher** — inotify/fsevents integration for live AST re-ingestion +- **NREM Consolidation** — periodic ego narrative tensor compression +- **Health Watchdog** — process monitoring with Agency Proposal minting + +Contributions to the daemon must not block any of these loops. Use `tokio::spawn` for any I/O-bound work. + +--- + +## Adding a New MCP Tool + +1. Add the tool's JSON schema definition to the `tools/list` response in `crates/engram-server/src/mcp.rs` +2. Add the handler arm in the `match tool_name` block in the same file +3. Add the tool to the MCP Tools Reference table in `README.md` with an accurate description +4. Update the tool count in the README header (`## MCP Tools Reference (N Tools)`) +5. Add a test in `crates/engram-server/src/mcp.rs` or a separate integration test + +--- + +## Pull Request Checklist + +- [ ] `cargo clippy --workspace -- -D warnings` passes with no new warnings +- [ ] `cargo test --workspace` passes +- [ ] No changes to the fixed byte layout of `HolographicBlock` +- [ ] README tool count and table updated if new tools were added +- [ ] FIRST_RUN.md updated if the setup flow changed +- [ ] No blocking calls in async daemon loops + +--- + +## What We're Looking For + +- **GPU backends:** ROCm and Metal backends are functional but less battle-tested than CUDA. Improvements welcome. +- **Tree-Sitter language coverage:** We currently parse Rust, Python, TypeScript, JavaScript, Go, Java, C, C++. Adding more languages is straightforward — see `crates/engram-core/src/ingest/ast.rs`. +- **Embedding server compatibility:** Currently tested against llama.cpp and ONNX-hosted nomic-embed. Other OpenAI-compatible endpoints should work but haven't been verified. +- **WebGPU backend:** The Poincaré hyperbolic INT8 search backend is production-ready but the WebGPU transport layer has known latency issues on some platforms. + +--- + +*Engram is developed by Aric Goodman and Static Rooster Media. Patent Pending US19/372,256. Licensed under AGPL-3.0-only.* diff --git a/Cargo.toml b/Cargo.toml index 3725de5..c1abe58 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -43,7 +43,7 @@ tracing-subscriber = { version = "0.3", features = ["env-filter"] } # Error handling anyhow = "1" thiserror = "1" -reqwest = { version = "0.12", features = ["blocking", "json"] } +reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls"] } # CLI clap = { version = "4", features = ["derive", "env"] } diff --git a/README.md b/README.md index b02d779..69373cf 100644 --- a/README.md +++ b/README.md @@ -3,20 +3,28 @@ [![Build Status](https://github.com/staticroostermedia-arch/engram/actions/workflows/rust.yml/badge.svg)](https://github.com/staticroostermedia-arch/engram/actions) [![MCP](https://img.shields.io/badge/MCP-Native-blue)](https://github.com/modelcontextprotocol) [![Glama](https://glama.ai/mcp/servers/staticroostermedia-arch/engram/badge)](https://glama.ai/mcp/servers/staticroostermedia-arch/engram) +[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-purple)](LICENSE) +[![Patent Pending](https://img.shields.io/badge/Patent-Pending-orange)](PATENT-NOTICE.md) -> **Hardware-native geometric memory for AI agents — 21 MCP tools.** +> **Hardware-native geometric memory for AI agents — 31 MCP tools.** -Engram is not a vector database. It is a persistent geometric memory engine designed for AI agents. It bypasses conventional database software layers by storing information in fixed, mathematically rigorous 256KB tensors directly on NVMe drives. No cloud, no API keys, no deserialization overhead. Runs entirely on your machine via the Model Context Protocol (MCP). +Engram is not a vector database. It is a **persistent geometric memory engine** designed for AI agents. It bypasses conventional database software layers by storing information in fixed, mathematically rigorous 256KB tensors directly on NVMe drives — with a background daemon that autonomously consolidates memory, monitors your system's health, and proposes fixes. + +No cloud. No API keys. No deserialization overhead. Runs entirely on your machine via the Model Context Protocol (MCP). --- ## 🚀 Quick Start -By default, Engram uses internal geometric hashing. To enable massive-scale semantic code search, point it to any OpenAI-compatible embeddings endpoint (e.g., local llama.cpp or ONNX): - ```bash -export ENGRAM_EMBED_URL="http://localhost:8080/v1/embeddings" -cargo install engram --git https://github.com/staticroostermedia-arch/engram +# Clone and install from source +git clone https://github.com/staticroostermedia-arch/engram.git +cd engram +cargo install --path crates/engram-server + +# Verify install +engram --version +# engram-server 0.4.x ``` Add to your MCP config and restart your IDE: @@ -26,130 +34,179 @@ Add to your MCP config and restart your IDE: "mcpServers": { "engram": { "command": "engram", - "args": ["mcp", "--store", "~/.engram/manifold"] + "args": ["mcp", "--store", "~/.engram/stalks/"] } } } ``` -Your agent immediately has access to all 21 tools. See [`integrations/`](integrations/) for IDE-specific configs. +Your agent immediately has access to all 31 tools. See [`integrations/`](integrations/) for IDE-specific configs (Antigravity, Claude Desktop, Cursor, VS Code). > 📖 **New here?** Read the **[First Run Guide](FIRST_RUN.md)** — it walks you through verifying every feature works, activating the file watcher daemon, and seeding your manifold with your codebase. +> 🔌 **Optional — enable neural semantic search:** set `ENGRAM_EMBED_URL=http://localhost:8086/v1/embeddings` to point at any OpenAI-compatible local embedding server (llama.cpp, ONNX, nomic-embed). Without it, Engram falls back to BLAKE3 spiral-phase encoding — everything still works. + --- ## ⚡ Why 256KB? The Hardware-Native Advantage -Engram maps your project's memory into strict 262,144 byte (256KB) containers called **HolographicBlocks**. This size is non-arbitrary. +Engram maps your project's memory into strict 262,144-byte (256KB) containers called **HolographicBlocks**. This size is non-arbitrary. -- **Native Tensor Load:** 256KB perfectly aligns to 64× 4KB hardware pages. Because the `.leg` format is a strict C-struct, it requires zero JSON decoding or Protobuf parsing. -- **O_DIRECT and GPUDirect Storage (GDS):** Engram bypasses the operating system's page-cache. When your agent searches for a memory, the tensor streams via Direct Memory Access (DMA) from the physical NVMe SSD straight into CPU registers or directly into GPU VRAM using NVIDIA cuFile APIs. -- **Zero-Copy Architecture:** By leveraging GPUDirect Storage, Engram eliminates the CPU bounce buffer entirely. Tensors are transferred directly over the PCIe bus to the GPU for massive parallel distance calculations, enabling scan rates of gigabytes per second with near-zero CPU overhead. +- **Native Tensor Load:** 256KB aligns perfectly to 64× 4KB hardware pages. The `.leg3` format is a strict C-struct — zero JSON decoding, zero Protobuf parsing. +- **O_DIRECT and GPUDirect Storage (GDS):** Engram bypasses the OS page-cache. When your agent searches for a memory, the tensor streams via DMA from NVMe directly into CPU registers or GPU VRAM via NVIDIA cuFile APIs. +- **Zero-Copy Architecture:** GPUDirect Storage eliminates the CPU bounce buffer. Tensors transfer directly over PCIe to the GPU for parallel distance calculations — scan rates in the GB/s range with near-zero CPU overhead. -Every block mathematically fuses the full original source code, 8192-dimensional semantic tensors, spatial 3D bounds (for code placement), and cryptographic BLAKE3 Merkle chain proofs. +Every block fuses the full source text, an 8192-dimensional semantic tensor, spatial 3D bounds (for code placement), a BLAKE3 Merkle chain proof, and a thermodynamic confidence score (CRS). -*(See [docs/architecture.md](docs/architecture.md) for a deep dive into the container format, cuFile integration, and LBVH scaling).* +*(See [docs/architecture.md](docs/architecture.md) for a deep dive into the container format, cuFile integration, and LBVH scaling.)* --- -## 🛡️ Hallucination & Loop Protection (Volatility Tracker) +## 🛡️ Hallucination & Loop Protection + +Traditional vector databases are append-logs: if an LLM hallucinates or loops, it spams the database with broken snippets, destroying context quality. -Traditional vector databases are "dumb" append-logs: if an LLM hallucinates or gets stuck in a debugging loop, it will spam the database with hundreds of slightly-different, broken code snippets, destroying the context window. +Engram uses a built-in **Lyapunov stability tracker** (the Coherence-Reliability Score, CRS) that monitors how much a concept drifts between updates: -Engram does not blindly accept every update as equally valid. It features a built-in mathematical volatility tracker (using Lyapunov stability equations) that monitors how much a concept "shifts" between updates: -- **Low Drift:** If an agent updates a memory and the semantic meaning barely changes, the system recognizes it as stable/converging. The block's **Coherence-Reliability Score (CRS)** goes up. -- **High Drift:** If an agent rapidly overwrites a memory with wildly different concepts (a hallucination loop), the system recognizes the volatility. The block's CRS is penalized. +- **Low Drift → CRS rises:** The system recognizes convergence and increases trust. +- **High Drift → CRS penalized:** Rapid contradictory overwrites are flagged as hallucination. Agents learn not to trust low-CRS blocks. -Memories must mathematically prove their stability over time. If a block's CRS drops too low, agents know not to trust it. +Memories must mathematically prove their stability. High-CRS blocks are automatically promoted to permanent `ZEDOS_PRAXIS` status during NREM consolidation. Low-CRS blocks decay and are swept by autophagy. --- -## 🖥️ CLI Commands +## 🧠 The Agentic Daemon -Beyond the MCP server, Engram ships a standalone CLI for direct manifold management: +When Engram boots as an MCP server, it launches a **background daemon** that runs three autonomous loops: -| Command | Description | -|---|---| -| `engram remember ` | Encode and store a memory | -| `engram recall ` | Semantic search, returns top-k | -| `engram forget ` | Delete a memory | -| `engram list` | List all stored concept names | -| `engram ingest ` | Recursively ingest a directory of text/code files | -| `engram trace ` | VSA geometry: query the result of ADD or BIND on two concepts | -| `engram distill` | **Crystallize** — cluster episodic memories into durable ZEDOS_PRAXIS blocks | +### 1. File Watcher +Auto-ingests saved files via `inotify`/`fsevents` kernel hooks. Every time you save a `.rs`, `.py`, `.ts`, or any other supported file, the AST pipeline extracts new semantic blocks and updates the manifold without any agent intervention. + +### 2. NREM Consolidation (Phase 3) +On a periodic cycle (~every 10 minutes), the daemon performs a **sleep-cycle memory consolidation pass**: +- Harvests all memories above CRS ≥ 0.74 (grounded fact tier) +- Superimposes them via `OP_ADD` into a unified **ego narrative tensor** +- Writes the result to `ego.leg3` — the agent's persistent self-model +- Mints a ZEDOS_EPISODIC block summarizing the consolidation + +This is the equivalent of REM sleep for the agent's memory. Knowledge crystallized in one session is absorbed into the ego tensor and becomes available as prior context in all future sessions. + +### 3. System Health Watchdog +The daemon continuously monitors critical background processes (e.g., the Circadian daemon that drives nightly consolidation). If a watched process dies, it automatically mints an **Agency Proposal** in the `agency_proposals.json` queue — a human-readable explanation of what failed and exactly what command it wants to run to fix it. The operator can approve or reject the proposal via the Cockpit UI or API. + +> **Autophagy is disabled by default.** An agent's memory should outlive sessions. Use `mcp_engram_forget_old` to trigger manual GC when needed. --- ## 🌳 AST-Aware Semantic Distillation -Traditional RAG chunks text arbitrarily, destroying function boundaries and context. Engram's ingest pipeline uses a universal **AST-extraction layer** powered natively by **Tree-Sitter**. +Traditional RAG chunks text arbitrarily, destroying function boundaries. Engram's ingest pipeline uses a universal **AST-extraction layer** powered by **Tree-Sitter**, parsing **Rust, Python, TypeScript, JavaScript, Go, Java, C, and C++**. -It natively parses **Rust, Python, TypeScript, JavaScript, Go, Java, C, and C++**. -Engram mints exactly **one memory block per public semantic item** (functions, structs, classes). +It mints exactly **one memory block per public semantic item** (functions, structs, classes, traits): -- **The Tensor (`q`):** Encodes the doc comment and signature. -- **The Provlog:** Carries the raw, full-length source code. -- **Spatial Embodiment:** Maps the precise 2D row/column coordinates of the AST node directly into the memory block's physical bounds, allowing the agent to know *where* code lives, not just *what* it does. +- **The Tensor (`q`):** Encodes the doc comment and signature — what it is and what it does. +- **The Provlog:** Carries the raw, full-length source code — verbatim retrieval at any time. +- **Spatial Embodiment:** Maps the precise 2D row/column coordinates (AABB) of each AST node into the block's physical bounds. Agents know *where* code lives, not just *what* it does. --- -## 🧰 MCP Tools Reference +## 🧰 MCP Tools Reference (31 Tools) -Engram exposes **21 tools** across 5 capability groups. - -### Core Memory +### Core Memory (4) | Tool | Description | |---|---| | `remember` | Encode text and store as a persistent memory block | -| `recall` | Semantic similarity search — returns top-k memories. Optional `time_decay` param for time-targeted search | +| `recall` | Semantic similarity search — returns top-k. Optional `time_decay` for time-targeted search and `zedos_filter` for type filtering | | `forget` | Delete a specific memory by concept name | | `list_concepts` | List all stored concept names | -| `mcp_engram_update` | Re-encode an existing memory in place (uses `op_add` superposition) | -| `mcp_engram_pin` | Lock a memory at CRS=1.0 — protects foundational constraints forever. | -### Memory Intelligence +### Memory Management (9) | Tool | Description | |---|---| +| `mcp_engram_update` | Re-encode an existing memory in place with Lyapunov drift tracking — **use this, never forget+remember** | +| `mcp_engram_pin` | Lock a memory at CRS=1.0 — protects foundational axioms permanently | | `mcp_engram_stats` | Manifold health report: total count, pinned, avg/min/max CRS, disk usage | | `mcp_engram_recall_recent` | Return N most recently accessed memories, sorted by access time | -| `mcp_engram_summarize` | Project-state digest: pinned memories + top-N by CRS. Single-call `/wake_up` replacement | -| `mcp_engram_forget_old` | On-demand autophagy: manually sweep out low-CRS blocks | +| `mcp_engram_summarize` | Project-state digest: pinned memories + top-N by CRS. Single-call wake-up replacement | +| `mcp_engram_forget_old` | On-demand autophagy: sweep out blocks below a CRS threshold | +| `mcp_engram_read_concept` | Fetch the full un-truncated text of a specific memory by exact concept name | +| `mcp_engram_export` | Serialize the entire manifold (or a CRS-filtered subset) to a portable JSON array | +| `mcp_engram_import` | Ingest a JSON array of `{concept, text}` objects into the manifold | -### Workspace & Agentic +### Workspace & Agentic (8) | Tool | Description | |---|---| -| `mcp_engram_watch_workspace` | Tell the daemon to watch a directory; automatically extracts and re-ingests file-saves through the Tree-Sitter AST pipeline | -| `mcp_engram_context_for_file` | Surface top-5 relevant memories for a file path (proactive loading) | -| `mcp_engram_session_end` | Commit session context and natively compute ADR Thermodynamics based on memory coherence | -| `mcp_engram_scar` | Create a geometric repeller using the maximum-entropy Apeiron primitive to mark a rejected thought or dead-end. | +| `mcp_engram_watch_workspace` | Bind a directory to the daemon's inotify watcher — auto-re-ingests saves via AST pipeline | +| `mcp_engram_context_for_file` | Surface top-5 relevant memories for a file path (proactive loading before editing) | +| `mcp_engram_recall_in_file` | Spatial code search: find all AST concepts defined within a specific line range | +| `mcp_engram_batch_remember` | Store multiple `{concept, text}` pairs in a single call — faster than N sequential `remember` calls | +| `mcp_engram_session_start` | **Mandatory at session start.** Validates manifold integrity and initializes epistemic state | +| `mcp_engram_session_end` | **Mandatory at session end.** Commits session summary + computes ADR thermodynamics | +| `mcp_engram_scar` | Create a geometric repeller (Apeiron binding) to mark a rejected approach as hostile — prevents re-hallucination | +| `mcp_engram_remember_solution` | Store a crystallized error→solution pair as a permanent `ZEDOS_PRAXIS` block. Auto-pinned at CRS=1.0 | + +### Knowledge Graph (3) -### Knowledge Graph +Every `mcp_engram_relate` call stores a `ZEDOS_RELATION` block via `OP_BIND`. Edges are mathematical memory vectors — no external graph database required. -Every `mcp_engram_relate` call stores a ZEDOS_RELATION block using `op_bind`. Edges are mathematical memory vectors, meaning no external graph database is required. +| Tool | Description | +|---|---| +| `mcp_engram_relate` | Bind two concepts via `OP_BIND` to create a directed knowledge graph edge | +| `mcp_engram_search_by_relation` | Traverse the graph by seed concept, edge direction, and optional label | +| `mcp_engram_visualize` | BFS from a seed concept → renders a Mermaid diagram of the subgraph | + +### Physics & Alignment (5) | Tool | Description | |---|---| -| `mcp_engram_relate` | Bind two concepts via `op_bind` to build the graph | -| `mcp_engram_search_by_relation` | Traverse the graph by edge direction and label | -| `mcp_engram_visualize` | BFS from a seed concept → outputs a Mermaid diagram | +| `mcp_engram_genesis` | Inspect or re-seed the foundational alignment genesis blocks (CRS=1.0, pinned, never decay) | +| `mcp_engram_verify_behavior` | Report empirical success/failure against a ZEDOS_HYPOTHESIS block. Repeated success promotes to PRAXIS | +| `mcp_engram_query_with_momentum` | Momentum-assisted recall: blends semantic similarity (80%) with concept trajectory (20%) | +| `mcp_engram_set_namespace` | Switch to a project-specific memory namespace (stalk). Creates it if it doesn't exist | +| `mcp_engram_list_namespaces` | List all available namespaces and the currently active one | + +### Autonomy & Orchestration (2) + +These tools expose Engram's deeper integration with the Monad OS oracle layer, enabling agent self-reflection and multi-step workflow orchestration. + +| Tool | Description | +|---|---| +| `mcp_self_trace` | Route a query through the Monad Oracle (Operator_LBR anchor) for deep logophysical self-reflection | +| `mcp_orchestrate_workflow_chain` | Chain multiple MCP tool calls into a single autonomous workflow execution | --- -## 🧠 The Agentic Daemon & Autophagy +## 🖥️ CLI Commands -When Engram boots as an MCP server, it also launches a **background Agentic Daemon** that manages autonomous file system watching via `inotify`/`fsevents` kernel integration. When you save a file, the daemon re-ingests the changed AST components instantly. +Beyond the MCP server, Engram ships a standalone CLI for direct manifold management: -> [!NOTE] -> **Autophagy (GC) is Disabled by Default:** We believe an agent's memory should outlive its sessions. If a user steps away from a project for 3 months, their contextual memory shouldn't spontaneously decay. Engram calculates Coherence-Reliability Scores (CRS) continuously, but we deliberately disabled automatic eviction. You must use the `mcp_engram_forget_old` tool to instruct the agent to run manual garbage collection. +| Command | Description | +|---|---| +| `engram remember ` | Encode and store a memory | +| `engram recall ` | Semantic search, returns top-k | +| `engram forget ` | Delete a memory | +| `engram list` | List all stored concept names | +| `engram ingest ` | Recursively ingest a directory (AST extraction for code + chunking for docs) | +| `engram trace ` | VSA geometry: query the result of ADD or BIND on two concepts | +| `engram distill` | **Crystallize** — cluster episodic memories into durable ZEDOS_PRAXIS blocks | +| `engram build-index` | Build the LBVH O(log N) index for large manifolds (>10K blocks) | --- ## 🌐 Multi-Project Namespaces -Use sheaf mode to isolate memories by project. Create `~/.engram/sheaf.toml`: +Engram isolates memories by project via namespaced stalks. No config file required — just call: + +``` +mcp_engram_set_namespace("my_project") # creates + switches to this namespace +mcp_engram_set_namespace("work_project") # switch to another project +mcp_engram_list_namespaces() # see all namespaces +``` + +Or configure via `~/.engram/sheaf.toml`: ```toml active_stalk = "codeland" @@ -163,16 +220,24 @@ name = "personal" path = "~/.engram/stalks/personal" ``` -Then switch namespaces via MCP at any time: -``` -mcp_engram_set_namespace("personal") -``` +--- + +## ⚙️ Hardware Support + +| Backend | Feature Flag | Status | Notes | +|---|---|---|---| +| CPU (Rayon O_DIRECT) | Default | ✅ | Exact linear scan. 10K memories → ~2.5 GB scanned in <0.4s via NVMe DMA bypass | +| CPU (LBVH index) | `bvh` | ✅ | O(log N) CSRP-projected tree. ~64 bytes RAM per concept. Build with `engram build-index` | +| CUDA (NVIDIA) | `cuda-kernels` | ✅ | GPU BVH O(log N), NVMe→VRAM parallel DMA via cuFile GDS | +| ROCm (AMD) | `rocm-kernels` | ✅ | Wavefront HIP execution | +| Metal (Apple) | `metal` | ✅ | MSL dynamic runtime compilation via metal-rs | +| WebGPU | `wgpu-backend` | ✅ | INT8 Poincaré hyperbolic search · 170× VRAM reduction · cross-platform | --- ## 💻 IDE Integration -> Integration configs for all supported IDEs: [`integrations/`](integrations/) +Integration configs for all supported IDEs: [`integrations/`](integrations/) ### Google Antigravity IDE ```json @@ -180,7 +245,7 @@ mcp_engram_set_namespace("personal") "mcpServers": { "engram": { "command": "engram", - "args": ["mcp", "--store", "~/.engram/manifold"], + "args": ["mcp", "--store", "~/.engram/stalks/"], "disabled": false } } @@ -193,7 +258,7 @@ mcp_engram_set_namespace("personal") "mcpServers": { "engram": { "command": "engram", - "args": ["mcp", "--store", "~/.engram/manifold"] + "args": ["mcp", "--store", "~/.engram/stalks/"] } } } @@ -201,24 +266,11 @@ mcp_engram_set_namespace("personal") --- -## ⚙️ Hardware Support - -| Backend | Feature Flag | Status | Notes | -|---|---|---|---| -| CPU (Rayon O_DIRECT) | Default | ✅ | Exact linear scan. At 10K memories scans 2.5 GB in < 0.4s via NVMe DMA bypass | -| CPU (LBVH index) | `bvh` feature | ✅ | O(log N) CSRP-projected tree. ~64 bytes RAM per concept. Build with `engram build-index` | -| CUDA (NVIDIA) | `cuda-kernels` | ✅ | GPU BVH O(log N), NVMe→VRAM parallel DMA | -| ROCm (AMD) | `rocm-kernels` | ✅ | Wavefront HIP execution | -| Metal (Apple) | `metal` | ✅ | MSL dynamic runtime compilation via metal-rs | -| **WebGPU** | **`wgpu-backend`** | ✅ | INT8 Poincaré hyperbolic search · 170× VRAM reduction · cross-platform | - ---- - ## 📄 License & Patent This software is licensed under **AGPL-3.0-only**. -The `.LEG` container format is covered by **U.S. Patent Application No. 19/372,256** (pending), +The `.LEG3` container format is covered by **U.S. Patent Application No. 19/372,256** (pending), *Self-Contained Variable File System (.LEG Container Format)*, Applicant: **Aric Goodman**, Oregon, USA — Static Rooster Media. diff --git a/crates/engram-server/Cargo.toml b/crates/engram-server/Cargo.toml index fbe1feb..9576e21 100644 --- a/crates/engram-server/Cargo.toml +++ b/crates/engram-server/Cargo.toml @@ -47,15 +47,15 @@ tracing-subscriber = { workspace = true } # Error handling anyhow = { workspace = true } thiserror = { workspace = true } +reqwest = { workspace = true } notify = "8.2.0" notify-debouncer-full = "0.7.0" flume = "0.12.0" blake3 = { workspace = true } toml = "0.8" bincode = "1" -regex = "1.12.3" -chrono = { version = "0.4", features = ["serde"] } -reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls"] } +regex = "1" +chrono = { version = "0.4", features = ["serde"] } bumpalo = "3.14.0" [lints.rust] diff --git a/crates/engram-server/src/daemon.rs b/crates/engram-server/src/daemon.rs index 30291e6..26fddb6 100644 --- a/crates/engram-server/src/daemon.rs +++ b/crates/engram-server/src/daemon.rs @@ -76,7 +76,10 @@ fn load_engramignore() -> Vec { /// Starts the global agentic background daemon attached to the MCP / REST Server. /// /// Autophagy GC is DISABLED. Nothing is ever evicted automatically. -/// The daemon runs purely as a workspace file watcher — auto-ingesting saved files. +/// The daemon runs three autonomous loops: +/// 1. File watcher — inotify/fsevents → AST extraction → live project indexing +/// 2. NREM cycle — periodic ego narrative tensor consolidation (ego.leg3) +/// 3. Health watchdog — config-driven process monitor (~/.engram/watchdog.toml) pub fn spawn(store: SharedStore) -> Arc { // Load shadow basis vectors at spawn time (once — not per file event) let shadow_cybernetics: Option> = load_shadow_vector("cybernetics"); @@ -88,6 +91,19 @@ pub fn spawn(store: SharedStore) -> Arc { if !engramignore.is_empty() { info!("[.engramignore] Excluding {} path patterns from watch", engramignore.len()); } + + // ── Load health watchdog config (~/.engram/watchdog.toml) ──────────────── + // This is the ONLY place Engram learns about consumer-specific processes. + // If the file doesn't exist, the watchdog is a no-op — zero coupling. + let watchdog_cfg = crate::watchdog::WatchdogConfig::load(); + let watchdog_proposals_path = watchdog_cfg.resolved_proposals_path(); + if watchdog_cfg.watch.is_empty() { + info!("[Watchdog] No processes configured — health watchdog is a no-op."); + } else { + info!("[Watchdog] Monitoring {} process(es). Proposals → {}", + watchdog_cfg.watch.len(), watchdog_proposals_path.display()); + } + let (watch_tx, watch_rx) = flume::unbounded::(); let daemon = Arc::new(DaemonControl { @@ -150,6 +166,12 @@ pub fn spawn(store: SharedStore) -> Arc { std::fs::create_dir_all(&inbox_dir).ok(); info!("Integration inbox watching: {}", inbox_dir.display()); + // ── System Health Watchdog ──────────────────────────────────────────── + // Every 5 minutes, check each process in ~/.engram/watchdog.toml. + // If the file doesn't exist, watch list is empty and this is a no-op. + let mut health_interval = tokio::time::interval(Duration::from_secs(5 * 60)); + health_interval.tick().await; // skip first tick on startup + loop { if ctrl.shutdown.load(Ordering::Relaxed) { break; @@ -176,6 +198,10 @@ pub fn spawn(store: SharedStore) -> Arc { run_nrem_consolidation(&store); } + _ = health_interval.tick() => { + run_health_watchdog(&watchdog_cfg, &watchdog_proposals_path); + } + _ = inbox_interval.tick() => { // ── Integration Inbox: sweep and process ───────────────────────── if let Ok(entries) = std::fs::read_dir(&inbox_dir) { @@ -451,6 +477,36 @@ fn run_nrem_consolidation(store: &crate::store::SharedStore) { } } +// ── System Health Watchdog ──────────────────────────────────────────────────── +// +// Config-driven: reads ~/.engram/watchdog.toml for process names to monitor +// and where to write agency proposals. If the config file doesn't exist this +// function is a no-op — Engram has zero coupling to any consumer project. +// +// Called every 5 minutes from the daemon select! loop. + +fn run_health_watchdog( + cfg: &crate::watchdog::WatchdogConfig, + proposals_path: &std::path::Path, +) { + if cfg.watch.is_empty() { + return; // no-op — watchdog.toml absent or empty + } + + for process in &cfg.watch { + if !crate::watchdog::is_process_alive(&process.name) { + info!( + "[Watchdog] '{}' not detected — minting agency proposal.", + process.name + ); + crate::watchdog::mint_proposal(process, proposals_path); + } else { + tracing::trace!("[Watchdog] '{}' alive ✓", process.name); + } + } +} + + pub struct DaemonControl { pub active_watch: Arc>>, shutdown: Arc, diff --git a/crates/engram-server/src/main.rs b/crates/engram-server/src/main.rs index 8574f26..bf5d66a 100644 --- a/crates/engram-server/src/main.rs +++ b/crates/engram-server/src/main.rs @@ -20,6 +20,7 @@ mod serve; mod store; pub mod daemon; pub mod ki_hijacker; +pub mod watchdog; pub mod scout; pub mod scout_supervisor; diff --git a/crates/engram-server/src/mcp.rs b/crates/engram-server/src/mcp.rs index cb2daf5..61a9b7d 100644 --- a/crates/engram-server/src/mcp.rs +++ b/crates/engram-server/src/mcp.rs @@ -1588,6 +1588,87 @@ fn handle_tool_call(name: &str, args: &Value, store: &SharedStore) -> Value { } } + "mcp_self_trace" => { + let query = args["query"].as_str().unwrap_or("").trim().to_string(); + if query.is_empty() { + return json!({ "content": [{ "type": "text", "text": "Error: query is required." }], "isError": true }); + } + + info!("mcp_self_trace: routing query to Monad Oracle (Operator_LBR anchor)"); + let client = reqwest::blocking::Client::new(); + let mut resp = None; + for p in [8080, 8081, 8082, 8083] { + let url = format!("http://127.0.0.1:{}/api/ask", p); + if let Ok(res) = client.post(&url).json(&serde_json::json!({ "query": query, "objective_only": false })).send() { + resp = Some(res); + break; + } + } + + match resp { + Some(r) if r.status().is_success() => { + let data: serde_json::Value = r.json().unwrap_or(serde_json::json!({})); + let prose = data["assembled_prose"].as_str().unwrap_or(""); + let crs = data["final_crs"].as_f64().unwrap_or(0.0); + let dist = 1.0 - (crs as f32).max(0.0).min(1.0); // Rough geometric distance surrogate + + let mut out = format!("🧠 Self-Trace Identity Response (Anchored to Operator_LBR)\n────────────────────────────────────────\n"); + out.push_str(&format!("Geometric Distance: {:.3} (CRS: {:.3})\n\n", dist, crs)); + out.push_str(prose); + if prose.is_empty() { + out.push_str("(No cohesive trajectory formed. The Oracle is uncertain.)"); + } + + json!({ "content": [{ "type": "text", "text": out }] }) + } + Some(r) => json!({ "content": [{ "type": "text", "text": format!("Oracle API error: HTTP {}", r.status()) }], "isError": true }), + None => json!({ "content": [{ "type": "text", "text": "Error: Could not connect to Monad Transductive API (/api/ask). Is the daemon running?" }], "isError": true }), + } + } + + "mcp_orchestrate_workflow_chain" => { + let concept = args["concept"].as_str().unwrap_or("").trim().to_string(); + if concept.is_empty() { + return json!({ "content": [{ "type": "text", "text": "Error: concept is required." }], "isError": true }); + } + + let mut visited = std::collections::HashSet::new(); + let mut chain = Vec::new(); + let mut current = concept.clone(); + let mut full_output = String::new(); + + loop { + visited.insert(current.clone()); + chain.push(current.clone()); + + let store_lk = store.lock().unwrap(); + let raw_concept = current.split_once("::").map_or(current.as_str(), |(_, r)| r); + if let Some(block) = store_lk.fetch_block(raw_concept) { + let full_text = engram_core::storage::read_provlog(&block); + full_output.push_str(&format!("### Step: {}\n{}\n\n", current, full_text)); + } else { + full_output.push_str(&format!("### Step: {}\n(No logophysical block found)\n\n", current)); + } + + let next = store_lk.search_relations(¤t, None, "from") + .into_iter() + .next(); + + drop(store_lk); + + if let Some((target, _lbl)) = next { + if !visited.contains(&target) { + current = target; + continue; + } + } + break; + } + + let out = format!("⛓️ Workflow Orchestration Chain:\n{}\n\n📝 Execution Steps:\n{}", chain.join(" ➔ "), full_output); + json!({ "content": [{ "type": "text", "text": out }] }) + } + "mcp_engram_scar" => { let concept = args["concept"].as_str().unwrap_or("").trim().to_string(); let magnitude = args["magnitude"].as_f64().unwrap_or(0.15) as f32; diff --git a/crates/engram-server/src/store.rs b/crates/engram-server/src/store.rs index 5ec85e9..d7259fc 100644 --- a/crates/engram-server/src/store.rs +++ b/crates/engram-server/src/store.rs @@ -616,6 +616,16 @@ impl StoreHandle { lock.daemon = Some(control); } + /// Reload ego.leg3 from disk into the ego_q field. + /// Called by the NREM daemon after each consolidation pass. + pub fn refresh_ego_q(&mut self) { + self.ego_q = load_ego_q(); + match &self.ego_q { + Some(_) => tracing::info!("[EgoGate] ego_q refreshed from ego.leg3"), + None => tracing::warn!("[EgoGate] ego.leg3 missing after NREM write — check daemon logs"), + } + } + // ── Passthrough ─────────────────────────────────────────────────────────── pub fn store_path(&self) -> &str { &self.path } @@ -679,14 +689,6 @@ impl StoreHandle { r } - /// Reload the Ego q-vector from disk — called by the NREM pass after - /// `accumulate_narrative()` updates ego.leg3. - pub fn refresh_ego_q(&mut self) { - self.ego_q = load_ego_q(); - if self.ego_q.is_some() { - tracing::info!("[EGO GATE] Ego q-vector refreshed from disk."); - } - } pub fn recall(&mut self, query: &str, k: usize) -> Vec { // MIN_SCORE_THRESHOLD: Dirichlet composite score floor. // With 23,000+ pinned blocks at CRS=1.0 the scorer's CRS term lifts all diff --git a/crates/engram-server/src/watchdog.rs b/crates/engram-server/src/watchdog.rs new file mode 100644 index 0000000..ab71ee3 --- /dev/null +++ b/crates/engram-server/src/watchdog.rs @@ -0,0 +1,253 @@ +//! Engram System Health Watchdog — Configuration Layer +//! +//! Reads `~/.engram/watchdog.toml` at daemon startup. If the file does not +//! exist the watchdog is a no-op — zero coupling to any consumer project. +//! +//! # Example `~/.engram/watchdog.toml` +//! +//! ```toml +//! # Where to write agency healing proposals (defaults to ~/.engram/proposals.json) +//! proposals_path = "~/.engram/proposals.json" +//! +//! [[watch]] +//! name = "circadian" +//! restart_hint = "nohup ~/Documents/CodeLand/target/release/circadian > /tmp/circadian.log 2>&1 &" +//! severity = "HIGH" +//! description = "Nightly NREM memory consolidation driver" +//! +//! [[watch]] +//! name = "my_rag_server" +//! restart_hint = "systemctl start my-rag" +//! severity = "MEDIUM" +//! description = "Custom embedding server for semantic recall" +//! ``` + +use serde::Deserialize; +use std::path::PathBuf; + +// ── Config Types ────────────────────────────────────────────────────────────── + +/// A single process entry in `watchdog.toml`. +#[derive(Debug, Clone, Deserialize)] +pub struct WatchedProcess { + /// Process name as it appears in `/proc//comm` (Linux) or `ps` output. + pub name: String, + + /// Human-readable explanation written into the agency proposal. + #[serde(default)] + pub description: String, + + /// The exact shell command the operator can approve to restart this process. + #[serde(default)] + pub restart_hint: String, + + /// Proposal severity: "HIGH", "MEDIUM", or "LOW". Defaults to "MEDIUM". + #[serde(default = "default_severity")] + pub severity: String, +} + +fn default_severity() -> String { "MEDIUM".to_string() } + +/// Top-level `watchdog.toml` structure. +#[derive(Debug, Clone, Deserialize, Default)] +pub struct WatchdogConfig { + /// Path where agency healing proposals are written. + /// Defaults to `~/.engram/proposals.json`. + /// Override with `ENGRAM_PROPOSALS_PATH` environment variable. + #[serde(default)] + pub proposals_path: Option, + + /// Processes to monitor. Each entry generates a proposal when the process + /// is not found in the process table. + #[serde(default)] + pub watch: Vec, +} + +impl WatchdogConfig { + /// Load from `~/.engram/watchdog.toml`. + /// + /// Returns an empty (no-op) config if the file does not exist. + /// Logs a warning if the file exists but fails to parse. + pub fn load() -> Self { + // Allow env override for the entire config file path + let config_path = std::env::var("ENGRAM_WATCHDOG_CONFIG") + .map(|p| PathBuf::from(shellexpand::tilde(&p).into_owned())) + .unwrap_or_else(|_| { + std::env::var("HOME") + .map(|h| PathBuf::from(h).join(".engram").join("watchdog.toml")) + .unwrap_or_else(|_| PathBuf::from("/tmp/engram_watchdog.toml")) + }); + + if !config_path.exists() { + tracing::debug!( + "[Watchdog] No config at {} — health watchdog disabled.", + config_path.display() + ); + return Self::default(); + } + + match std::fs::read_to_string(&config_path) { + Ok(text) => match toml::from_str::(&text) { + Ok(cfg) => { + tracing::info!( + "[Watchdog] Loaded config: {} process(es) to watch.", + cfg.watch.len() + ); + cfg + } + Err(e) => { + tracing::warn!( + "[Watchdog] Failed to parse {}: {}. Health watchdog disabled.", + config_path.display(), e + ); + Self::default() + } + }, + Err(e) => { + tracing::warn!( + "[Watchdog] Cannot read {}: {}. Health watchdog disabled.", + config_path.display(), e + ); + Self::default() + } + } + } + + /// Resolve the proposals file path. + /// + /// Priority: + /// 1. `ENGRAM_PROPOSALS_PATH` environment variable + /// 2. `proposals_path` field in `watchdog.toml` + /// 3. `~/.engram/proposals.json` (Engram-local default) + pub fn resolved_proposals_path(&self) -> PathBuf { + // 1. Env var always wins + if let Ok(p) = std::env::var("ENGRAM_PROPOSALS_PATH") { + return PathBuf::from(shellexpand::tilde(&p).into_owned()); + } + // 2. Config file field + if let Some(ref p) = self.proposals_path { + return PathBuf::from(shellexpand::tilde(p).into_owned()); + } + // 3. Default: ~/.engram/proposals.json + std::env::var("HOME") + .map(|h| PathBuf::from(h).join(".engram").join("proposals.json")) + .unwrap_or_else(|_| PathBuf::from("/tmp/engram_proposals.json")) + } +} + +// ── Runtime Health Check ────────────────────────────────────────────────────── + +/// Check whether a process with the given name is currently running. +/// +/// On Linux: scans `/proc//comm`. +/// On non-Linux: always returns `true` (assume alive — watchdog is Linux-native). +pub fn is_process_alive(name: &str) -> bool { + #[cfg(target_os = "linux")] + { + std::fs::read_dir("/proc") + .ok() + .map(|entries| { + entries.flatten().any(|e| { + // Only numeric entries are PIDs + let fname = e.file_name(); + let is_pid = fname.to_str() + .map(|s| s.chars().all(|c| c.is_ascii_digit())) + .unwrap_or(false); + if !is_pid { return false; } + std::fs::read_to_string(e.path().join("comm")) + .ok() + .map(|s| s.trim() == name) + .unwrap_or(false) + }) + }) + .unwrap_or(true) // if /proc unreadable, assume alive + } + + #[cfg(not(target_os = "linux"))] + { + let _ = name; + true // macOS/Windows: watchdog is a no-op (use launchd/SCM instead) + } +} + +/// Append a healing proposal to the proposals JSON array on disk. +/// +/// Deduplicates: does not write a new proposal if one for the same process +/// name is already in the file (prevents spamming every 5 minutes). +pub fn mint_proposal(process: &WatchedProcess, proposals_path: &std::path::Path) { + let id = format!( + "health_{}_{}", + process.name, + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs() + ); + + let proposal = serde_json::json!({ + "id": id, + "type": "SYSTEM_HEALTH", + "severity": process.severity, + "title": format!("'{}' process offline", process.name), + "plain_english": if process.description.is_empty() { + format!("The '{}' process is not running.", process.name) + } else { + process.description.clone() + }, + "if_approved": if process.restart_hint.is_empty() { + format!("Manually restart the '{}' process.", process.name) + } else { + process.restart_hint.clone() + }, + "if_rejected": format!( + "'{}' stays offline. No automatic action will be taken.", + process.name + ), + "timestamp": std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(), + }); + + // Load existing proposals (or start fresh) + let mut proposals: Vec = proposals_path + .exists() + .then(|| std::fs::read_to_string(proposals_path).ok()) + .flatten() + .and_then(|s| serde_json::from_str(&s).ok()) + .unwrap_or_default(); + + // Deduplication check: same process name already pending? + let already_pending = proposals.iter().any(|p| { + p.get("title") + .and_then(|t| t.as_str()) + .map(|t| t.contains(&process.name)) + .unwrap_or(false) + }); + + if already_pending { + tracing::debug!("[Watchdog] Proposal for '{}' already pending — skipping.", process.name); + return; + } + + proposals.push(proposal); + + // Ensure parent directory exists + if let Some(parent) = proposals_path.parent() { + std::fs::create_dir_all(parent).ok(); + } + + match serde_json::to_string_pretty(&proposals) { + Ok(json) => { + if let Err(e) = std::fs::write(proposals_path, json) { + tracing::error!("[Watchdog] Failed to write proposals to {}: {}", proposals_path.display(), e); + } else { + tracing::info!( + "[Watchdog] Minted SYSTEM_HEALTH proposal for '{}' → {}", + process.name, proposals_path.display() + ); + } + } + Err(e) => tracing::error!("[Watchdog] Failed to serialize proposals: {}", e), + } +} diff --git a/watchdog.example.toml b/watchdog.example.toml new file mode 100644 index 0000000..bee5625 --- /dev/null +++ b/watchdog.example.toml @@ -0,0 +1,43 @@ +# ~/.engram/watchdog.toml +# +# Engram System Health Watchdog Configuration +# ───────────────────────────────────────────── +# Place this file at ~/.engram/watchdog.toml to enable the health watchdog. +# If this file does not exist, the watchdog loop is a no-op. +# +# ENV OVERRIDES: +# ENGRAM_WATCHDOG_CONFIG — override the path to this config file +# ENGRAM_PROPOSALS_PATH — override where proposals are written +# +# Proposals are written to: ~/.engram/proposals.json (default) +# Engram's Cockpit UI reads this file to surface healing suggestions. + +# Where to write agency healing proposals. +# Defaults to ~/.engram/proposals.json if omitted. +# proposals_path = "~/.engram/proposals.json" + +# ── Processes to monitor ─────────────────────────────────────────────────────── +# Each [[watch]] entry specifies a process to check every 5 minutes. +# `name` must match the value in /proc//comm (Linux) — typically the +# binary name truncated to 15 characters. + +# Example: CodeLand's circadian NREM driver +[[watch]] +name = "circadian" +severity = "HIGH" +description = "Nightly NREM memory consolidation driver. Keeps ego.leg3 updated." +restart_hint = "nohup ~/Documents/CodeLand/target/release/circadian > ~/Documents/CodeLand/logs/circadian.log 2>&1 &" + +# Example: a custom embedding server +# [[watch]] +# name = "ollama" +# severity = "MEDIUM" +# description = "Local embedding server used for semantic recall." +# restart_hint = "ollama serve &" + +# Example: a systemd-managed service +# [[watch]] +# name = "my-rag" +# severity = "LOW" +# description = "Custom RAG pipeline." +# restart_hint = "systemctl start my-rag"