V-3.0.0

CHANGELOG.md

@@ -4,6 +4,91 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [3.0.0] - 2026-04-01 — Event-Sourced Cognition Substrate
+
+Dhee v3 is a ground-up architectural overhaul that transforms the memory layer into an immutable-event, versioned cognition substrate. Raw memory is now immutable truth; derived cognition (beliefs, policies, insights, heuristics) is provisional, rebuildable, and auditable.
+
+### New Architecture
+
+- **Immutable raw events** — `remember()` writes to `raw_memory_events`. Corrections create new events with `supersedes_event_id`, never mutate originals.
+- **Type-specific derived stores** — Beliefs, Policies, Anchors, Insights, Heuristics each have their own table with type-appropriate schemas, indexes, lifecycle rules, and invalidation behavior.
+- **Derived lineage** — Every derived object traces to its source raw events via `derived_lineage` table with contribution weights.
+- **Candidate promotion pipeline** — Consolidation no longer writes through `memory.add()`. Distillation produces candidates; promotion validates, dedupes, and transactionally promotes them to typed stores.
+
+### Three-Tier Invalidation
+
+- **Hard invalidation** — Source deleted: derived objects tombstoned, excluded from retrieval.
+- **Soft invalidation** — Source corrected: derived objects marked stale, repair job enqueued.
+- **Partial invalidation** — One of N sources changed with contribution weight < 30%: confidence penalty, not full re-derive. Weight >= 30% escalates to soft invalidation.
+
+### 5-Stage RRF Fusion Retrieval
+
+1. Per-index retrieval (raw, distilled, episodic — parallel, zero LLM)
+2. Min-max score normalization within each index
+3. Weighted Reciprocal Rank Fusion (k=60, distilled=1.0, episodic=0.7, raw=0.5)
+4. Post-fusion adjustments (recency boost, confidence normalization, staleness penalty, invalidation exclusion, contradiction penalty)
+5. Dedup + final ranking — zero LLM calls on the hot path
+
+### Conflict Handling
+
+- **Cognitive conflicts table** — Contradictions are explicit rows, not silent resolution.
+- **Auto-resolution** — When confidence gap >= 0.5 (one side >= 0.8, other <= 0.3), auto-resolve in favor of high-confidence side.
+
+### Job Registry
+
+- Replaces phantom `agi_loop.py` with real, observable maintenance jobs.
+- SQLite lease manager prevents concurrent execution of same job.
+- Jobs are named, idempotent, leasable, retryable, and independently testable.
+
+### Anchor Resolution
+
+- Per-field anchor candidates with individual confidence scores.
+- Re-anchoring: corrections re-resolve without touching raw events.
+
+### Materialized Read Model
+
+- `retrieval_view` materialized table for fast cold-path queries.
+- Delta overlay for hot-path freshness.
+
+### Migration Bridge
+
+- Dual-write: v2 path + v3 raw events in parallel (`DHEE_V3_WRITE=1`, default on).
+- Backfill: idempotent migration of v2 memories into v3 raw events via content-hash dedup.
+- Feature flag `DHEE_V3_READ` (default off) for gradual cutover.
+
+### Observability
+
+- `v3_health()` reports: raw event counts, derived invalidation counts per type, open conflicts, active leases, candidate stats, job health, retrieval view freshness, lineage coverage.
+
+### Consolidation Safety
+
+- Breaks the feedback loop: `_promote_to_passive()` uses `infer=False`, tags `source="consolidated"`.
+- `_should_promote()` rejects already-consolidated content.
+
+### Other Changes
+
+- `UniversalEngram.to_dict(sparse=True)` — omits None, empty strings, empty lists, empty dicts.
+- `agi_loop.py` cleaned: removed all phantom `engram_*` package imports.
+- 913 tests passing.
+
+### New Files
+
+- `dhee/core/storage.py` — Schema DDL for all v3 tables
+- `dhee/core/events.py` — RawEventStore
+- `dhee/core/derived_store.py` — BeliefStore, PolicyStore, AnchorStore, InsightStore, HeuristicStore, DerivedLineageStore, CognitionStore
+- `dhee/core/anchor_resolver.py` — AnchorCandidateStore, AnchorResolver
+- `dhee/core/invalidation.py` — Three-tier InvalidationEngine
+- `dhee/core/conflicts.py` — ConflictStore with auto-resolution
+- `dhee/core/read_model.py` — Materialized ReadModel
+- `dhee/core/fusion_v3.py` — 5-stage RRF fusion pipeline
+- `dhee/core/v3_health.py` — Observability metrics
+- `dhee/core/v3_migration.py` — Dual-write bridge + backfill
+- `dhee/core/lease_manager.py` — SQLite lease manager
+- `dhee/core/jobs.py` — JobRegistry + concrete jobs
+- `dhee/core/promotion.py` — PromotionEngine
+
+---
+
 ## [2.2.0b1] - 2026-03-31 — Architectural Cleanup
 
 Beta release focused on internal discipline rather than new features.

dhee/__init__.py

@@ -31,7 +31,7 @@
 # Default: CoreMemory (lightest, zero-config)
 Memory = CoreMemory
 
-__version__ = "2.2.0b1"
+__version__ = "3.0.0"
 __all__ = [
     # Memory classes
     "CoreMemory",

dhee/core/agi_loop.py

@@ -1,18 +1,27 @@
-"""AGI Loop — the full cognitive cycle.
+"""Dhee v3 — Cognitive Maintenance Cycle.
 
-Orchestrates all memory subsystems in a single cycle:
-Perceive → Attend → Encode → Store → Consolidate → Retrieve →
-Evaluate → Learn → Plan → Act → Loop
+Replaces the phantom AGI loop with honest, real maintenance operations.
 
-This module provides the run_agi_cycle function called by the
-heartbeat behavior, plus system health reporting.
+v2.2 had 8 steps, 6 of which imported non-existent engram_* packages.
+v3 runs only what actually exists:
+    1. Consolidation (active → passive, via safe consolidation engine)
+    2. Decay (forgetting curves)
+
+Planned but not yet implemented (will be added as real Job classes):
+    - Anchor candidate resolution
+    - Distillation promotion
+    - Conflict scanning
+    - Stale intention cleanup
+
+The old API surface (run_agi_cycle, get_system_health) is preserved
+for backward compatibility with existing callers.
 """
 
 from __future__ import annotations
 
 import logging
 from datetime import datetime, timezone
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, Optional
 
 logger = logging.getLogger(__name__)
 
@@ -22,22 +31,20 @@ def run_agi_cycle(
     user_id: str = "default",
     context: Optional[str] = None,
 ) -> Dict[str, Any]:
-    """Run one iteration of the AGI cognitive cycle.
-
-    Each step is optional — missing subsystems are gracefully skipped.
+    """Run one maintenance cycle. Only executes real subsystems.
 
     Args:
-        memory: Engram Memory instance
+        memory: Dhee Memory instance
         user_id: User identifier for scoped operations
-        context: Optional current context for reconsolidation
+        context: Optional current context (reserved for future use)
 
     Returns:
-        Dict with status of each subsystem step
+        Dict with status of each step
     """
     now = datetime.now(timezone.utc).isoformat()
     results: Dict[str, Any] = {"timestamp": now, "user_id": user_id}
 
-    # 1. Consolidate — run distillation (episodic → semantic)
+    # Step 1: Consolidation — run distillation (episodic → semantic)
     try:
         if hasattr(memory, "_kernel") and memory._kernel:
             consolidation = memory._kernel.sleep_cycle(user_id=user_id)
@@ -47,96 +54,19 @@ def run_agi_cycle(
     except Exception as e:
         results["consolidation"] = {"status": "error", "error": str(e)}
 
-    # 2. Decay — apply forgetting
+    # Step 2: Decay — apply forgetting curves
     try:
         decay_result = memory.apply_decay(scope={"user_id": user_id})
         results["decay"] = {"status": "ok", "result": decay_result}
     except Exception as e:
         results["decay"] = {"status": "error", "error": str(e)}
 
-    # 3. Reconsolidation — auto-apply high-confidence proposals
-    try:
-        from engram_reconsolidation import Reconsolidation
-        rc = Reconsolidation(memory, user_id=user_id)
-        pending = rc.list_pending_proposals(limit=5)
-        auto_applied = 0
-        for p in pending:
-            if p.get("confidence", 0) >= rc.config.min_confidence_for_auto_apply:
-                rc.apply_update(p["id"])
-                auto_applied += 1
-        results["reconsolidation"] = {
-            "status": "ok", "pending": len(pending), "auto_applied": auto_applied
-        }
-    except ImportError:
-        results["reconsolidation"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["reconsolidation"] = {"status": "error", "error": str(e)}
-
-    # 4. Procedural — scan for extractable procedures
-    try:
-        from engram_procedural import Procedural
-        proc = Procedural(memory, user_id=user_id)
-        procedures = proc.list_procedures(status="active", limit=5)
-        results["procedural"] = {
-            "status": "ok", "active_procedures": len(procedures)
-        }
-    except ImportError:
-        results["procedural"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["procedural"] = {"status": "error", "error": str(e)}
-
-    # 5. Metamemory — calibration check
-    try:
-        from engram_metamemory import Metamemory
-        mm = Metamemory(memory, user_id=user_id)
-        gaps = mm.list_knowledge_gaps(limit=5)
-        results["metamemory"] = {
-            "status": "ok", "open_gaps": len(gaps)
-        }
-    except ImportError:
-        results["metamemory"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["metamemory"] = {"status": "error", "error": str(e)}
-
-    # 6. Prospective — check intention triggers
-    try:
-        from engram_prospective import Prospective
-        pm = Prospective(memory, user_id=user_id)
-        triggered = pm.check_triggers()
-        results["prospective"] = {
-            "status": "ok", "triggered": len(triggered)
-        }
-    except ImportError:
-        results["prospective"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["prospective"] = {"status": "error", "error": str(e)}
-
-    # 7. Working memory — decay stale items
-    try:
-        from engram_working import WorkingMemory
-        wm = WorkingMemory(memory, user_id=user_id)
-        items = wm.list()
-        results["working_memory"] = {
-            "status": "ok", "active_items": len(items)
-        }
-    except ImportError:
-        results["working_memory"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["working_memory"] = {"status": "error", "error": str(e)}
-
-    # 8. Failure — check for extractable anti-patterns
-    try:
-        from engram_failure import FailureLearning
-        fl = FailureLearning(memory, user_id=user_id)
-        stats = fl.get_failure_stats()
-        results["failure_learning"] = {"status": "ok", **stats}
-    except ImportError:
-        results["failure_learning"] = {"status": "skipped", "reason": "not installed"}
-    except Exception as e:
-        results["failure_learning"] = {"status": "error", "error": str(e)}
-
-    # Compute overall status
-    statuses = [v.get("status", "unknown") for v in results.values() if isinstance(v, dict)]
+    # Compute summary
+    statuses = [
+        v.get("status", "unknown")
+        for v in results.values()
+        if isinstance(v, dict) and "status" in v
+    ]
     ok_count = statuses.count("ok")
     error_count = statuses.count("error")
     skipped_count = statuses.count("skipped")
@@ -152,53 +82,56 @@ def run_agi_cycle(
 
 
 def get_system_health(memory: Any, user_id: str = "default") -> Dict[str, Any]:
-    """Report health status across all cognitive subsystems.
+    """Report health status across real cognitive subsystems.
 
-    Returns a dict with each subsystem's availability and basic stats.
+    Only reports subsystems that actually exist — no phantom package checks.
     """
     now = datetime.now(timezone.utc).isoformat()
     systems: Dict[str, Dict] = {}
 
-    # Core systems (always available)
+    # Core memory
     try:
         stats = memory.get_stats(user_id=user_id)
         systems["core_memory"] = {"available": True, "stats": stats}
     except Exception as e:
         systems["core_memory"] = {"available": False, "error": str(e)}
 
-    # Knowledge Graph
+    # Knowledge graph
     systems["knowledge_graph"] = {
-        "available": hasattr(memory, "knowledge_graph") and memory.knowledge_graph is not None,
+        "available": (
+            hasattr(memory, "knowledge_graph")
+            and memory.knowledge_graph is not None
+        ),
     }
     if systems["knowledge_graph"]["available"]:
         try:
             systems["knowledge_graph"]["stats"] = memory.knowledge_graph.stats()
         except Exception:
             pass
 
-    # Power packages
-    _optional_packages = [
-        ("engram_router", "router"),
-        ("engram_identity", "identity"),
-        ("engram_heartbeat", "heartbeat"),
-        ("engram_policy", "policy"),
-        ("engram_skills", "skills"),
-        ("engram_spawn", "spawn"),
-        ("engram_resilience", "resilience"),
-        ("engram_metamemory", "metamemory"),
-        ("engram_prospective", "prospective"),
-        ("engram_procedural", "procedural"),
-        ("engram_reconsolidation", "reconsolidation"),
-        ("engram_failure", "failure_learning"),
-        ("engram_working", "working_memory"),
-    ]
-
-    for pkg_name, system_name in _optional_packages:
+    # Cognition kernel
+    has_kernel = hasattr(memory, "_kernel") and memory._kernel is not None
+    systems["cognition_kernel"] = {"available": has_kernel}
+    if has_kernel:
         try:
-            __import__(pkg_name)
-            systems[system_name] = {"available": True}
-        except ImportError:
-            systems[system_name] = {"available": False}
+            systems["cognition_kernel"]["stats"] = memory._kernel.cognition_health(
+                user_id=user_id
+            )
+        except Exception:
+            pass
+
+    # Active memory / consolidation
+    systems["consolidation"] = {
+        "available": (
+            hasattr(memory, "_consolidation_engine")
+            and memory._consolidation_engine is not None
+        ),
+    }
+
+    # v3 stores (if wired)
+    systems["v3_event_store"] = {
+        "available": hasattr(memory, "_event_store") and memory._event_store is not None,
+    }
 
     available = sum(1 for s in systems.values() if s.get("available"))
     total = len(systems)