Extract sidecar schema to a shared JSON Schema (single source for plugin + agent conformer)

[bobthearsonist]

Problem

The sidecar contract is enforced in two hand-maintained implementations that can drift:

• Plugin (renderer): plugins/obsidian-plugin/src/schema.ts — parseSidecar() is a hand-written zod schema (node id kebab regex, node classes regex ^(system|task|decision) (completed|active|context|blocked)$, edge classes enum strong-edge/weak-edge, required bounded position x[-200,5000]/y[-200,3000], strict top-level keys, _extractedBy/_schemaVersion/_pinned semantics, edge-endpoint referential check).
• Agent conformer (Engine A): regenerate.py in the ai-private visual-notes skill — re-implements those same rules by hand in Python (NODE_ID_RE, NODE_CLASS_RE, EDGE_CLASSES, ALLOWED_TOP, position clamps, max nodes/edges). Its own docstring says it "mirrors plugins/obsidian-plugin/src/schema.ts."

Two hand-mirrored implementations of one contract drift. Recent near-misses all trace to this: edge classes must be strong-edge/weak-edge (not "edge"), strict top-level keys, the node classes regex, and position bounds.

Current mechanism

A JSON Schema already exists at shared/schema.json (draft-07, @visual-notes/shared) and is a complete mirror of the contract — but it is not the enforcement source of truth. Today it is only consumed by extractor.ts via toAnthropicToolSchema(sharedSidecarSchema) to shape the LLM tool input (what the model is asked to produce). The actual validation on both sides is still hand-mirrored:

• the plugin validates with hand-written zod in schema.ts (not derived from shared/schema.json);
• the Python conformer validates/coerces with hard-coded constants (no reference to shared/schema.json at all — it's in a separate repo).

So we have the schema file but neither validator is driven by it. The drift surface is unchanged.

Proposal

Promote shared/schema.json from "LLM tool-input shape" to the single enforcement source of truth for both consumers:

• TS plugin: validate against shared/schema.json via ajv (or generate the zod schema from it via json-schema-to-zod), preserving parseSidecar()'s current behavior — including the referential edge-endpoint check (superRefine) and .strict() top-level semantics.
• Python conformer: drive coerce/validate from the same file via the jsonschema package instead of the hand-coded NODE_ID_RE / NODE_CLASS_RE / EDGE_CLASSES / bounds constants. (Requires the conformer to read the schema from the visual-notes repo — vendor or pin it.)

The schema already has $id and rich descriptions, so it can carry the prose currently duplicated in the extraction prompt and the conformer comments.

Benefits

• Single source of truth — no drift between plugin, conformer, and LLM tool-input.
• Both engines provably conform to the same artifact.
• _extractedBy / _schemaVersion / _pinned semantics centralized in one place.
• The LLM tool-input shape and the runtime validator can no longer disagree (they're the same file).

Acceptance

• shared/schema.json is the only definition of the contract (zod in schema.ts is generated from it or replaced by ajv validation against it).
• Plugin validates sidecars against shared/schema.json (preserving edge-endpoint referential check + strict top-level keys).
• Python conformer validates/coerces against the same shared/schema.json (no hand-coded rule constants).
• A shared fixture set of valid/invalid sidecars that both sides agree on (same accept/reject verdicts).

Notes

• The conformer lives in a separate repo (ai-private visual-notes skill). Centralization requires a way to share the schema across repos (vendored copy with a version pin, published @visual-notes/shared, or a sync step) — worth deciding as part of this.

[bobthearsonist]

Update (2026-06-03): findings + interim decision

The schema file exists but isn't the enforcement source. shared/schema.json (draft-07) is a faithful structural mirror — root additionalProperties: false (strict), node-class pattern, edge strong-edge/weak-edge enums, position minimum/maximum, and the _extractedBy/_pinned/_schemaVersion fields — but only extractor.ts consumes it (toAnthropicToolSchema, i.e. shaping the LLM tool input). Neither validator is driven by it.

The contract is now hand-mirrored in three places (the drift surface grew):

1. Plugin zod parseSidecar — plugins/obsidian-plugin/src/schema.ts.
2. Agent conformer (ai-private regenerate.py) — NODE_ID_RE / NODE_CLASS_RE / EDGE_CLASSES / ALLOWED_TOP / position clamps.
3. The conformer's provenance stamps _extractedBy (visual-notes-agent@1.0.0) / _schemaVersion (0.1.0, mirrors the plugin's SETTINGS_VERSION) — added in ai-private plugin v1.3.0.

Constraints found while scoping the fix:

• Python jsonschema is not available to the hook's python3 (no venv guarantee) → the conformer can't validate via jsonschema; it would read rule-values from schema.json via stdlib json.
• The plugin ships only zod (no ajv / json-schema-to-zod) → either is a new dependency + build step, and touches the locked renderer.test.ts.
• The edge-endpoint referential check (superRefine) and the if/then conditionals are not expressible in pure JSON Schema → those stay as code regardless.

Interim decision (maintainer, 2026-06-03):

• Now: duplicate schema.json into each repo — canonical shared/schema.json stays here; a pinned vendored copy goes in ai-private for the conformer. Accept the two-copy stopgap (diffing two identical-format JSON files beats reconciling hand-translated zod ↔ Python constants).
• Later (the real fix): consolidate by moving the agent-side / conformer source into this repo, where it arguably belongs — then the schema is single-sourced within one repo and the cross-repo split disappears. Single-sourcing the validators (ajv/json-schema-to-zod on TS, stdlib-json rule extraction on Python) is straightforward once co-located.

Keeping this issue open to track the consolidation + single-sourcing.