Evaluate extracting reusable packages from internal/

[blackwell-systems]

Context

knowing's internal/ contains several components with clean boundaries that could serve broader use cases as independent Go packages. This issue tracks evaluation of what to extract, when, and under what stability guarantees.

Candidates

Strong candidates (clean boundaries, general-purpose)

Package	Current location	What it does	Potential consumers
Hierarchical Merkle tree	internal/snapshot/hierarchical.go + merkle.go	Semantic-boundary Merkle trees with subgraph roots, typed diffs, context pack roots	Any content-addressed system: config management, infrastructure graphs, dependency trackers, audit systems
Subgraph cache	internal/cache/subgraph.go	TTL-bounded cache keyed by Merkle roots with selective package-scoped invalidation	Anything using content-addressed caching
GCF/TOON wire formats	internal/wire/gcf.go + toon.go	Token-optimized wire formats for LLM context delivery	Any MCP server, any tool sending structured data to agents
Community detection	internal/community/	Pluggable Algorithm interface with Louvain + label propagation	Graph analysis, visualization, social network analysis
Hash identity	internal/types/types.go (Hash, NewHash, domain prefixes, Verify)	Content-addressed identity with type-safe domain prefixes	Any Go project using SHA-256 content addressing
Equivalence classes	internal/context/equivalence.go + universal_seeds.go	Vocabulary bridging between natural language and code symbol names	Code search tools, developer-facing retrieval systems

Moderate candidates (need interface decoupling)

Package	Blocker
Tree-sitter extractor	Depends on types package; needs a minimal interface
LSP enrichment	Depends on types + store; the pattern is reusable but needs abstraction
RWR + HITS algorithms	Graph algorithms are general but wrapped in knowing-specific scoring

Not extractable (too knowing-specific)

• internal/store/sqlite.go (schema is knowing-specific)
• internal/mcp/ (tool definitions are product-specific)
• internal/daemon/ (watcher + lifecycle is product-specific)
• cmd/knowing/ (CLI is the product)

Stability concerns

The Merkle tree API is NOT stable yet:

• Hash domain prefixes shipped 2026-05-18 (broke all existing hashes)
• File-level roots are planned (Phase 4, would add a tree level)
• The flat tree was just dropped (hierarchical root is now canonical)

Do not extract until:

1. Hash format is stable for at least one release cycle
2. File-level roots ship or are explicitly deferred
3. Subgraph cache has been validated by daily use
4. At least one external consumer validates the API

Suggested extraction order

1. GCF/TOON first (helps MCP ecosystem, creates network effect, format is stable)
2. Community detection second (generic algorithms, no competitive advantage from keeping private)
3. Equivalence classes third (the concept is useful broadly, knowing's classes are tuned for knowing)
4. Hierarchical Merkle tree last (the differentiator, extract only after API is stable)

Decision

This is a tracking issue. No extraction should happen until the conditions above are met. The purpose is to maintain awareness of what's extractable so internal code stays clean at the boundaries.

[blackwell-systems]

Update (v0.3.0, 2026-05-19)

New candidate: Graph notes table

graph_notes (migration 012) is a general-purpose metadata-beside-identity pattern: (object_hash, key, value) with upsert semantics, never affects Merkle computation. Any content-addressed system that needs annotations, quality scores, or cached derived state alongside immutable identity would use this. Add to strong candidates.

Community detection: IncrementalAlgorithm interface shipped

DetectIncremental(g, previous, changedNodes) now exists on both Louvain and LabelPropagation. Benchmarked: 6.9x (Louvain) and 38.4x (LP) speedup when 1 package changes. This adds API surface that needs to stabilize before extraction. The Algorithm interface is unchanged; IncrementalAlgorithm extends it.

Stability clock started

v0.3.0 shipped with hash domain prefixes (node\0, edge\0, snapshot\0, merkle\0). The "one release cycle" stability criterion for hash format starts now. If v0.4.0 ships without hash changes, the Merkle tree extraction gate is met.

Flat tree dropped

The flat Merkle tree was removed in this release. Hierarchical root is canonical. This simplifies the extraction surface for the Merkle tree package.

Updated extraction order

1. GCF/TOON (stable, ready after one more consumer validates)
2. Community detection (now includes IncrementalAlgorithm; wait for P1 wiring)
3. Graph notes (new candidate, pattern is generic)
4. Equivalence classes (unchanged)
5. Hierarchical Merkle tree (wait for hash stability + file-level roots decision)