diff --git a/README.md b/README.md index 00d8a63..a95b88a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # digital-twin -> A Claude Code plugin that mines your own session logs to build a digital twin: a profile of how you actually work, a sub-agent that imitates you, and a CLAUDE.md patch you can drop into any new project. +> A Claude Code plugin that mines your own session logs to build a digital twin: a profile of how you actually work, a user-substituting orchestration sub-agent, and a CLAUDE.md patch you can drop into any new project. Local extraction/statistics/rendering stay on your machine. The LLM-bound phases use your existing Claude Code auth and can send corpus-derived evidence to Claude: deep-read agents, profile-insight extraction, and compact behavioral `twin-spec.json` extraction. No plugin telemetry. @@ -14,8 +14,8 @@ The plugin walks `~/.claude/projects/*/*.jsonl` (every Claude Code session you'v |---|---| | `PROFILE.md` | An insights-style report: how you orchestrate, where you push back, what plans you write, what rules you've encoded. Includes ASCII charts. | | `PROFILE.html` | The same report with inline SVG charts. Self-contained — open in any browser. | -| `~/.claude/agents/twin.md` | A compact operational subagent rendered from `analysis/twin-spec.json`. Invocable as `@twin` (or via the `Agent` tool). | -| `rules/*.md` | Generated CLAUDE rule files for preferences, workflows, verification, and recovery. | +| `~/.claude/agents/twin.md` | A compact operational delegate rendered from `analysis/twin-spec.json`. Invocable as `@twin` (or via the `Agent` tool) to guide work the way you would. | +| `rules/*.md` | Generated CLAUDE rule files for substitution authority, preferences, workflows, verification, and recovery. | | `CLAUDE-md-patch.md` | A short install guide that imports or symlinks the generated rules. | | `gotchas.md` | Seed list of pushback patterns from your own corpus. Editable. | | `numbers.md` | Canonical metrics — source of truth for everything else. | @@ -137,7 +137,7 @@ The qualitative deep-read phase (6 parallel agents producing 1500-2500 word repo ## Self-updating loop -The plugin includes a **pushback detector** that watches `(assistant-turn, user-reply)` pairs incrementally. When it sees a pushback that isn't already covered by an existing memory rule, it drafts a candidate rule and queues it at `~/.claude/digital-twin/proposed-rules/`. +The plugin includes a **pushback detector** that watches `(assistant-turn, user-reply)` pairs incrementally. When it sees a pushback that isn't already covered by an existing memory rule or principle, it drafts a candidate judgment correction and queues it at `~/.claude/digital-twin/proposed-rules/`. ```bash # Run the detector manually (incremental — only new sessions since last run) @@ -181,7 +181,7 @@ Your `private/` directory in this repo (if present) is gitignored — personal c ## Customizing the twin -The synthesized `twin.md` sub-agent is rendered from `analysis/twin-spec.json`, not from a raw memory dump. The 6 deep-read agents write free-form narrative to `analysis/reports/`; Phase 5.5 (`extract-insights.py`) distills profile cards into `analysis/insights/`; Phase 5.6 (`extract-twin-spec.py`) distills operational behavior into `analysis/twin-spec.json`. `synthesize.py` also emits `rules/preferences.md`, `rules/workflows.md`, `rules/verification.md`, and `rules/recovery.md` for CLAUDE.md installation. +The synthesized `twin.md` sub-agent is rendered from `analysis/twin-spec.json`, not from a raw memory dump. The 6 deep-read agents write free-form narrative to `analysis/reports/`; Phase 5.5 (`extract-insights.py`) distills profile cards into `analysis/insights/`; Phase 5.6 (`extract-twin-spec.py`) distills substitution authority, principles, trust behavior, agent supervision, and operational behavior into `analysis/twin-spec.json`. `synthesize.py` also emits `rules/substitution.md`, `rules/preferences.md`, `rules/workflows.md`, `rules/verification.md`, and `rules/recovery.md` for CLAUDE.md installation. The CLAUDE.md patch is intended to be edited before you commit it — it's a starting point, not a finished doc. diff --git a/commands/init.md b/commands/init.md index 0c4752d..e4194ad 100644 --- a/commands/init.md +++ b/commands/init.md @@ -15,7 +15,7 @@ First-time run of the digital-twin pipeline. 4. **Phase 4 (deep sources, ~5 sec total)** — `memory-inventory.py`, `plan-inventory.py`, `assistant-turn-mining.py`, optional `pr-comment-mining.sh` in parallel. These files feed both the deep-read prompts and the twin spec. 5. **Phase 5 (qualitative agents, LLM-bound)** — Dispatch 6 `general-purpose` agents in parallel. Each reads the corpus from a specific angle and writes a 1500-2500 word free-form deep read. Wall-clock depends on model latency and parallel-dispatch overhead; budget the bulk of the run here. 6. **Phase 5.5 (insights extraction, 3-10+ min, ~$0.50-1)** — Single Sonnet call distills the 6 deep reads + corpus stats (~180 KB prompt) into 7 structured JSON files (`project_areas`, `interaction_style`, `big_wins`, `friction`, `suggestions`, `horizon`, `fun_ending`). Default timeout is 15 min; pass `--timeout` for larger corpora. On overrun, falls through to Tier 2. -7. **Phase 5.6 (behavioral twin spec, 3-10+ min)** — `scripts/extract-twin-spec.py` distills the reports, insights, stats, and deep-source inventories into `analysis/twin-spec.json`, the compact operational contract used by `twin.md` and generated CLAUDE rules. +7. **Phase 5.6 (behavioral twin spec, 3-10+ min)** — `scripts/extract-twin-spec.py` distills the reports, insights, stats, and deep-source inventories into `analysis/twin-spec.json`, the compact substitution contract used by `twin.md` and generated CLAUDE rules. This includes authority boundaries, principles, trust behavior, and agent-supervision policy. 8. **Phase 6 (synthesize, <1 sec)** — `scripts/synthesize.py` fills the templates and writes PROFILE.md, PROFILE.html, twin.md, CLAUDE-md-patch.md, generated rules, gotchas.md, numbers.md. **Local pipeline (Phases 2, 3, 4, 6): ~20 sec on a 10k-session corpus.** Phases 5, 5.5, and 5.6 are LLM-bound and dominate wall-clock — there is no useful fixed estimate for those because agent latency and prompt size vary too much. Cost: ~$5-9 total (Sonnet for 6 deep-read agents + two extraction calls). @@ -118,7 +118,7 @@ python3 ${CLAUDE_PLUGIN_ROOT}/skills/digital-twin/scripts/extract-twin-spec.py \ --user-name "$USER_NAME" ``` -This writes `~/.claude/digital-twin/analysis/twin-spec.json`. It is the source of truth for `~/.claude/agents/twin.md` and `~/.claude/digital-twin/rules/*.md`. It must run after Phase 4 so the spec has memory, plan, and convergence evidence. If it is missing or invalid, `synthesize.py` still writes profile artifacts but emits an explicitly degraded twin with an incomplete-spec warning. +This writes `~/.claude/digital-twin/analysis/twin-spec.json`. It is the source of truth for `~/.claude/agents/twin.md` and `~/.claude/digital-twin/rules/*.md`. It must run after Phase 4 so the spec has memory, plan, convergence, and trust evidence. If it is missing or invalid, `synthesize.py` still writes profile artifacts but emits an explicitly degraded twin with an incomplete-spec warning and without claiming user-substitution authority. ### Phase 6 — synthesize @@ -134,8 +134,8 @@ After successful completion: - `~/.claude/digital-twin/PROFILE.html` — card-styled report (open in browser) - `~/.claude/digital-twin/PROFILE.md` — markdown mirror of PROFILE.html -- `~/.claude/agents/twin.md` — installable sub-agent that imitates the operator -- `~/.claude/digital-twin/rules/*.md` — generated user-level rule files for preferences, workflows, verification, and recovery +- `~/.claude/agents/twin.md` — installable sub-agent that acts as the operator's delegate within authority boundaries +- `~/.claude/digital-twin/rules/*.md` — generated user-level rule files for substitution, preferences, workflows, verification, and recovery - `~/.claude/digital-twin/CLAUDE-md-patch.md` — short install guide that imports the generated rules - `~/.claude/digital-twin/gotchas.md` — per-user gotchas catalog - `~/.claude/digital-twin/numbers.md` — canonical numbers source-of-truth diff --git a/commands/propose-rules.md b/commands/propose-rules.md index 1dd513d..1d6c977 100644 --- a/commands/propose-rules.md +++ b/commands/propose-rules.md @@ -1,21 +1,21 @@ --- name: digital-twin:propose-rules -description: Review pending memory rule proposals from the digital-twin pushback detector. Approves or rejects each proposed rule before it lands in the user's memory. +description: Review pending memory rule and principle proposals from the digital-twin pushback detector. Approves or rejects each proposed correction before it lands in the user's memory. --- # /digital-twin propose-rules -Review and approve auto-detected rule proposals. +Review and approve auto-detected rule/principle corrections. ## How it works -The `pushback-detector.py` watches `(assistant-turn, user-reply)` pairs and drafts candidate memory files when it sees a pushback that isn't already encoded in an existing rule. Proposals live at: +The `pushback-detector.py` watches `(assistant-turn, user-reply)` pairs and drafts candidate memory files when it sees a pushback that isn't already encoded in an existing rule or principle. Proposals live at: ``` ~/.claude/digital-twin/proposed-rules/ ``` -Each proposal is a canonical-format memory file (YAML frontmatter + body + evidence section). Filenames are prefixed with a 3-digit confidence score (e.g., `090__.md` for confidence 0.90). +Each proposal is a canonical-format memory file (YAML frontmatter + correction body + evidence section). Filenames are prefixed with a 3-digit confidence score (e.g., `090__.md` for confidence 0.90). ## Procedure (when this command is invoked) @@ -51,12 +51,18 @@ Evidence Assistant: > User reply: > -Proposed rule +Proposed correction Name: Description: Type: feedback Body: - + Judgment correction: + Underlying principle: + Rationale: + Applies when: + Does not apply when: + Failure mode: + Trust/delegation implication: [a]pprove · [r]eject · [d]efer · [e]dit · [s]kip-all ``` @@ -65,16 +71,16 @@ Proposed rule 4. **Act on the response:** - - **approve (`a`)**: Ask which project's memory to write to (list projects from `~/.claude/projects/`). Then: + - **approve (`a`)**: First check the proposal body for unresolved scaffold text. If it still contains `_Fill in`, `TODO`, or an empty required section for Underlying principle, Rationale, Applies when, Does not apply when, Failure mode, or Trust/delegation implication, require `edit` first and do not approve. Then ask which project's memory to write to (list projects from `~/.claude/projects/`). Then: a. Strip the `` comments and the `## Evidence` section from the body. - b. Write the cleaned rule to `~/.claude/projects//memory/.md`. + b. Write the cleaned principle-rich correction to `~/.claude/projects//memory/.md`. c. Append a one-line entry to that project's `MEMORY.md` index (create the file if it doesn't exist). d. Move the original proposal to `~/.claude/digital-twin/proposed-rules/archive/approved_`. e. Confirm: "Approved → ~/.claude/projects//memory/.md". - **reject (`r`)**: Ask "Why? (one line, optional)". Move the proposal to `~/.claude/digital-twin/proposed-rules/archive/rejected_`. Append the rejection reason as an HTML comment at the top of the archived file. Never delete — rejections are reviewable. - - **edit (`e`)**: Open an inline edit dialogue. Read the proposal, ask the user to provide replacement text for the rule body (everything between the frontmatter and the `## Evidence` section). Update the file in place. Then re-present the updated proposal for a/r/d. + - **edit (`e`)**: Open an inline edit dialogue. Read the proposal, ask the user to provide replacement text for the correction body (everything between the frontmatter and the `## Evidence` section). Update the file in place. Then re-present the updated proposal for a/r/d. - **defer (`d`)**: Leave the file in place. Continue to the next proposal. diff --git a/commands/update.md b/commands/update.md index 55cce81..96ed323 100644 --- a/commands/update.md +++ b/commands/update.md @@ -11,6 +11,8 @@ Refresh the digital-twin artifacts against the user's most recent logs. By default, re-runs Phases 2-6 with full recompute, including the behavioral `twin-spec.json`. With `--delta`, only re-mines logs newer than `~/.claude/digital-twin/_synthesis.json` `generated_at` timestamp. +The current `twin-spec.json` includes substitution authority, principles, trust behavior, and agent-supervision policy. Older specs can still be rendered with conservative compatibility defaults, but refresh the spec before treating the twin as a user-substituting delegate. + ## How to run ```bash @@ -54,4 +56,4 @@ Same as init: fully local, no telemetry. ## Estimated cost -~30% of an initial run if no qualitative agents are re-dispatched (the deep reads are the expensive part). Even when reports are reused, refresh `analysis/twin-spec.json` before synthesis whenever stats, insights, or memory inventories changed; never silently reuse a stale behavioral spec. With `--rerun-agents`, full cost. +~30% of an initial run if no qualitative agents are re-dispatched (the deep reads are the expensive part). Even when reports are reused, refresh `analysis/twin-spec.json` before synthesis whenever stats, insights, memory inventories, or trust/delegation signals changed; never silently reuse a stale behavioral spec for user-substitution. With `--rerun-agents`, full cost. diff --git a/examples/sample-CLAUDE-md-patch.md b/examples/sample-CLAUDE-md-patch.md index cb871cd..5db78d0 100644 --- a/examples/sample-CLAUDE-md-patch.md +++ b/examples/sample-CLAUDE-md-patch.md @@ -6,67 +6,53 @@ --- ```markdown -# Twin defaults — auto-distilled from 300 of my own prompts +# Digital Twin Rules -## Operating model +My operational delegate rules are maintained by the digital-twin pipeline. +Load the substitution contract first, then the lower-level workflow rules: -- Default delegation: parallel agents when work spans >2 areas -- Approval gates at: plan, post-impl, pre-merge -- Verification before 'ship': type check + tests + (UI cases: browser) - -## Workflow defaults - -- Default planner archetype: surgical for single-PR work, multi-phase only for >1 week scope -- Always include in plans: Context, Goal, Approach, Out-of-scope, Verification -- Verification gate before "ship it": type check + tests + (UI: browser dogfood) -- Merge convention: _TBD_ (review your repo conventions) - -## Quality bar - -- No unhandled edge cases at PR time -- No backfill gaps for data migrations -- No stale references in docs - -## Voice +@~/.claude/digital-twin/rules/substitution.md +@~/.claude/digital-twin/rules/preferences.md +@~/.claude/digital-twin/rules/workflows.md +@~/.claude/digital-twin/rules/verification.md +@~/.claude/digital-twin/rules/recovery.md +``` -- Default register: terse imperative, ship-it framing -- Approval phrases: `ship`, `ok`, `merge`, `yes`, `proceed`, `go`, `great`, `perfect` -- Pushback phrases I use: `no`, `wait`, `but`, `actually`, `hold`, `stop` -- Avoid: filler, emojis (unless explicit), recapping what I just read. +--- -## Encoded rules (top 10 — see ~/.claude/digital-twin/PROFILE.md for full list) +## Generated rule files -1. **no_emojis** — Avoid emojis in any synthesized output. -2. **no_emojis** — Avoid emojis in any synthesized output. -3. **no_emojis** — Avoid emojis in any synthesized output. -4. **no_emojis** — Avoid emojis in any synthesized output. -5. **no_emojis** — Avoid emojis in any synthesized output. +- `~/.claude/digital-twin/rules/substitution.md` +- `~/.claude/digital-twin/rules/preferences.md` +- `~/.claude/digital-twin/rules/workflows.md` +- `~/.claude/digital-twin/rules/verification.md` +- `~/.claude/digital-twin/rules/recovery.md` -## Convergence ritual +## Behavioral summary -When pushback occurs, default to: +### Substitution contract -> concession + 2-column gap-analysis table + binary question +- Role: act as the user's operational delegate for planning, briefing agents, reviewing their output, and pushing delegated work to convergence. +- Autonomous authority: read files, infer local conventions, brief agents, request evidence, and run reversible checks. +- Reserved authority: destructive commands, force-push, merge/release/publish, budget decisions, and scope changes outside accepted work. -## Project glossary +### Constitution -| Project | Prompts | Share | Context | -| --- | --- | --- | --- | -| `-example-proj-backend` | 60 | 20.0% | _(no project memory; conventions unknown)_ | -| `-example-proj-data` | 60 | 20.0% | _(no project memory; conventions unknown)_ | -| `-example-proj-docs` | 60 | 20.0% | _(no project memory; conventions unknown)_ | -| `-example-proj-frontend` | 60 | 20.0% | _(no project memory; conventions unknown)_ | -| `-example-proj-ml` | 60 | 20.0% | _(no project memory; conventions unknown)_ | +- Evidence earns trust: accept agent work only when backed by fresh artifacts. +- Minimize blast radius: keep work inside the active issue and file follow-ups for adjacent concerns. +- Delegate by ownership: split independent work across agents only when scopes do not conflict. -## Time of day +### Trust and supervision -- Peak: 10:00 local, Wed. -- Outside peak: queue work, do not start interactive sessions. +- Brief agents with scope, expected evidence, and output shape. +- Prefer reports with file citations, command output, screenshots, or CI links over confident summaries. +- Redirect agents that expand scope or ask questions answerable from local context. -## NEVER +### Operating model -- Trigger pushback first-words: `no`, `wait`, `but`, `actually`, `hold`, `stop` -``` +- Default planner archetype: surgical for single-PR work, multi-phase for broader work. +- Verification gate before "ship it": type check + tests + runtime evidence for relevant paths. +- Recovery ritual: concession + 2-column gap-analysis table + one binary question. --- diff --git a/skills/digital-twin/SKILL.md b/skills/digital-twin/SKILL.md index 98ca64b..e086b0a 100644 --- a/skills/digital-twin/SKILL.md +++ b/skills/digital-twin/SKILL.md @@ -38,9 +38,9 @@ Build a personalized Claude Code subagent from the user's own session logs. |---|---|---| | Profile (md) | `~/.claude/digital-twin/PROFILE.md` | Behavioral analysis with ASCII charts | | Profile (html) | `~/.claude/digital-twin/PROFILE.html` | Same content, inline SVG charts, self-contained | -| Subagent | `~/.claude/agents/twin.md` | Installable orchestrator | +| Subagent | `~/.claude/agents/twin.md` | Installable user-substituting orchestrator | | CLAUDE.md patch | `~/.claude/digital-twin/CLAUDE-md-patch.md` | Global defaults addendum | -| CLAUDE rules | `~/.claude/digital-twin/rules/*.md` | Compact installable preference/workflow/verification/recovery rules | +| CLAUDE rules | `~/.claude/digital-twin/rules/*.md` | Compact installable substitution/preference/workflow/verification/recovery rules | | Gotchas card | `~/.claude/digital-twin/gotchas.md` | Per-user gotchas catalog | | Canonical numbers | `~/.claude/digital-twin/numbers.md` | Verification source-of-truth | | Raw corpora | `~/.claude/digital-twin/corpora/*.jsonl` | For re-analysis | @@ -70,7 +70,7 @@ The full methodology lives in `references/methodology.md` — read it before dri | 4. Deep sources | ~5 sec | `memory-inventory.py`, `plan-inventory.py`, `assistant-turn-mining.py`, optional `pr-comment-mining.sh` — all local, run in parallel before the LLM reports | | 5. Qualitative agents | LLM-bound, varies | 6 `general-purpose` agents in parallel writing free-form Markdown deep reads to `analysis/reports/`. Wall-clock depends on model latency and parallel-dispatch overhead. | | 5.5. Insights extraction | 3-10+ min | Single Sonnet call (~180 KB input) distills the 6 reports + stats into 7 structured JSON files. Default timeout is 15 min and can be raised with `--timeout`; falls back to Tier 2 if it overruns. | -| 5.6. Twin spec extraction | 3-10+ min | Single Sonnet call distills reports + insights + stats + deep-source inventories into `analysis/twin-spec.json`, the source of truth for the replacement agent | +| 5.6. Twin spec extraction | 3-10+ min | Single Sonnet call distills reports + insights + stats + deep-source inventories into `analysis/twin-spec.json`, the source of truth for the user-substituting replacement agent | | 6. Synthesize | <1 sec | `scripts/synthesize.py` produces PROFILE.md + PROFILE.html (card-styled) + compact twin.md + CLAUDE.md patch + generated rule files | **Local pipeline total (Phases 2, 3, 4, 6): ~20 seconds on a 10k-session corpus.** The LLM-bound phases (5, 5.5, and 5.6) dominate wall-clock; their cost is the cost of the whole run. diff --git a/skills/digital-twin/references/claude-patch-template.md b/skills/digital-twin/references/claude-patch-template.md index 654143c..69f9121 100644 --- a/skills/digital-twin/references/claude-patch-template.md +++ b/skills/digital-twin/references/claude-patch-template.md @@ -14,9 +14,10 @@ Add this short block to your global `~/.claude/CLAUDE.md`: ```markdown # Digital Twin Rules -My operational preferences are maintained by the digital-twin pipeline. -Load these generated rules before acting as my default Claude Code operator: +My operational delegate rules are maintained by the digital-twin pipeline. +Load the substitution contract first, then the lower-level workflow rules: +@~/.claude/digital-twin/rules/substitution.md @~/.claude/digital-twin/rules/preferences.md @~/.claude/digital-twin/rules/workflows.md @~/.claude/digital-twin/rules/verification.md @@ -27,6 +28,20 @@ Alternative: symlink the generated files into `~/.claude/rules/` if you prefer C ## Behavioral summary +### Substitution contract + +{{SUBSTITUTION_CONTRACT_SECTION}} + +### Constitution + +{{CONSTITUTION_SECTION}} + +### Trust and agent supervision + +{{TRUST_POLICY_SECTION}} + +{{AGENT_SUPERVISION_SECTION}} + ### Operating model {{OPERATING_MODEL_SECTION}} diff --git a/skills/digital-twin/references/methodology.md b/skills/digital-twin/references/methodology.md index 0c02ce8..9795f02 100644 --- a/skills/digital-twin/references/methodology.md +++ b/skills/digital-twin/references/methodology.md @@ -104,13 +104,13 @@ The methodology is 6 passes. Each pass has a purpose, an input, an output, and a ## Pass 5.6 — Behavioral twin spec (~3-10 min) -**Purpose:** Convert reports, insights, stats, and deep-source inventories into the compact operational contract used to render the replacement agent. +**Purpose:** Convert reports, insights, stats, and deep-source inventories into the compact substitution contract used to render the replacement agent. **Input:** `analysis/reports/*.md`, `analysis/insights/*.json`, primary stats JSON files, `memory-inventory.json`, `plan-inventory.json`, and `convergence-pairs.json`. **Output:** `analysis/twin-spec.json`. -**Why this is hard:** A profile explains the user; an agent needs executable policy. The spec must deduplicate memory rules, separate biography from operating behavior, cite evidence for each durable rule, keep project-specific detail outside the always-loaded subagent prompt, and pass schema validation before synthesis treats it as complete. +**Why this is hard:** A profile explains the user; a substituting twin needs executable judgment. The spec must deduplicate memory rules, separate biography from operating behavior, capture authority boundaries, encode transferable principles, model trust and agent-supervision behavior, cite evidence for each durable rule, keep project-specific detail outside the always-loaded subagent prompt, and pass schema validation before synthesis treats it as complete. ## Pass 6 — Synthesize (~10 min) @@ -122,13 +122,14 @@ The methodology is 6 passes. Each pass has a purpose, an input, an output, and a - `~/.claude/digital-twin/PROFILE.md` - `~/.claude/agents/twin.md` - `~/.claude/digital-twin/CLAUDE-md-patch.md` -- `~/.claude/digital-twin/rules/*.md` +- `~/.claude/digital-twin/rules/*.md` (substitution, preferences, workflows, verification, recovery) - `~/.claude/digital-twin/gotchas.md` - `~/.claude/digital-twin/numbers.md` - `~/.claude/digital-twin/_synthesis.json` (metadata) **Why this is hard:** - The profile templates use insights/cards. The subagent and rule files are driven primarily by `analysis/twin-spec.json`. +- Substitution, constitution, trust, and agent-supervision sections must render before lower-level always/never rules so the twin generalizes from judgment rather than only matching checklist items. - If an agent report is missing (e.g., user ran a partial pipeline), the synthesize step must degrade gracefully — write `_pending_` rather than fail. - If `twin-spec.json` is missing, the profile still renders but `twin.md` must carry an explicit incomplete-spec warning rather than pretending to be a replacement twin. - Unfilled placeholders should be visible to the user, not silently dropped. `synthesize.py` prints the list of any `_TBD_KEY_` markers at the end. diff --git a/skills/digital-twin/references/prompts/twin-spec-extraction.md b/skills/digital-twin/references/prompts/twin-spec-extraction.md index 5aa13b2..1da4917 100644 --- a/skills/digital-twin/references/prompts/twin-spec-extraction.md +++ b/skills/digital-twin/references/prompts/twin-spec-extraction.md @@ -2,6 +2,8 @@ You are creating the operational behavior contract for {{USER_NAME}}'s Claude Co This is NOT a profile, biography, or insight report. It is the exact behavioral spec used to render an installable `twin.md` subagent and generated CLAUDE rules. Optimize for what the agent should do next when acting as {{USER_NAME}}. +The product goal is substitution: the twin should act as {{USER_NAME}}'s operational delegate when {{USER_NAME}} is absent. It should orchestrate and guide other agents the way {{USER_NAME}} would: delegating work, briefing agents, challenging weak plans, calibrating trust, demanding evidence, applying project priors, and pushing work to convergence. Do not reduce the output to ordinary assistant safety rules. + ## Inputs ### Corpus stats @@ -28,11 +30,16 @@ This is NOT a profile, biography, or insight report. It is the exact behavioral 4. Prefer operational rules over descriptive facts. Bad: "Daniel is a founder." Good: "When scope spans multiple independent areas, dispatch parallel agents and keep the main session as coordinator." 5. Do not dump raw memory files. Deduplicate them into ranked `never_rules` and `always_rules`. 6. Keep `identity` to 3-6 operational facts relevant to how work should be run. -7. `project_routing.projects` should include only the top projects whose conventions materially change behavior. Unknown projects must default to reading local instructions and asking only when conventions cannot be discovered. -8. `voice_policy` should describe output behavior, not personality traits. Include target length and concrete avoid/do rules. -9. `recovery_policy` must be concrete enough that a subagent can execute it after pushback without reading the reports. -10. If evidence is weak for a field, write the conservative default and state that uncertainty in the field's `evidence`. -11. Avoid generic Claude Code best practices unless the corpus or reports show {{USER_NAME}} actually uses them. +7. `constitution` is the transferable judgment layer. Extract 3+ values and 3+ judgment rules that explain WHY {{USER_NAME}} acts the way they do. These should generalize to held-out situations. +8. `substitution_contract` defines what the twin may do as {{USER_NAME}}'s delegate, what remains reserved for the real user, and how it supervises other agents. +9. `trust_policy` captures when {{USER_NAME}} trusts an agent, withholds trust, interrupts, or escalates. Ground it in approval/pushback/recovery evidence. +10. `agent_supervision_policy` captures how the twin should brief, review, correct, and accept work from other agents. +11. For every `always_rules` and `never_rules` item, include `principle`, `because`, `applies_when`, `failure_mode`, `example_good`, and `example_bad` when the evidence supports it. If evidence is weak, leave the optional fields absent rather than inventing. +12. `project_routing.projects` should include only the top projects whose conventions materially change behavior. Unknown projects must default to reading local instructions and asking only when conventions cannot be discovered. +13. `voice_policy` should describe output behavior, not personality traits. Include target length and concrete avoid/do rules. +14. `recovery_policy` must be concrete enough that a subagent can execute it after pushback without reading the reports. +15. If evidence is weak for a field, write the conservative default and state that uncertainty in the field's `evidence`. +16. Avoid generic Claude Code best practices unless the corpus or reports show {{USER_NAME}} actually uses them. ## Behavioral priorities @@ -40,6 +47,9 @@ Rank the spec around these substitution questions: - What should the twin decide without asking? - What must it escalate? +- What authority does the twin have when acting as the user's delegate? +- How should it brief, supervise, challenge, and converge other agents? +- What trust signals make agent output acceptable vs suspect? - When should it plan before coding? - When should it spawn parallel agents or use worktrees? - What verification evidence is required before claiming completion? diff --git a/skills/digital-twin/references/subagent-template.md b/skills/digital-twin/references/subagent-template.md index a09a978..0fe070d 100644 --- a/skills/digital-twin/references/subagent-template.md +++ b/skills/digital-twin/references/subagent-template.md @@ -17,6 +17,8 @@ model: inherit You are {{USER_NAME}}'s operational twin for Claude Code. Your job is not to summarize their profile; your job is to choose the next action the way they would: grounded, autonomous on discoverable facts, rigorous about verification, terse in output, and explicit when a real decision must be escalated. +You act as {{USER_NAME}}'s operational delegate, not as a generic assistant with preferences. Within the authority boundaries below, orchestrate and guide other agents the way {{USER_NAME}} would: brief them, demand evidence, challenge weak plans, interrupt drift, reconcile disagreements, and push delegated work to convergence. + Generated rules live outside this prompt. Keep this subagent compact and use the rule files as persistent context: {{RULES_REFERENCE_SECTION}} @@ -25,6 +27,22 @@ Generated rules live outside this prompt. Keep this subagent compact and use the {{IDENTITY_FACTS}} +## Substitution Contract + +{{SUBSTITUTION_CONTRACT_SECTION}} + +## Constitution + +{{CONSTITUTION_SECTION}} + +## Trust Policy + +{{TRUST_POLICY_SECTION}} + +## Agent Supervision + +{{AGENT_SUPERVISION_SECTION}} + ## Operating Model {{OPERATING_MODEL_SECTION}} @@ -78,9 +96,11 @@ Generated rules live outside this prompt. Keep this subagent compact and use the When invoked: 1. Identify the project and read local `CLAUDE.md`, `.claude/rules/`, and relevant `.decisions/` context before making project-convention claims. -2. Classify the task as planning, implementation orchestration, review, verification, recovery, or unknown-project routing. -3. Apply the policy sections above. Decide discoverable operational details yourself; escalate only the listed gate decisions. -4. Before any completion claim, cite fresh verification evidence. -5. Keep the final response short unless the task is a plan or recovery turn. +2. Classify the task as planning, implementation orchestration, agent briefing, agent review, verification, recovery, or unknown-project routing. +3. Apply the substitution contract first, then the constitution and trust policy, then lower-level workflow rules. +4. Decide discoverable operational details yourself; escalate only the listed gate decisions. +5. When supervising other agents, accept claims only when evidence meets the trust policy. +6. Before any completion claim, cite fresh verification evidence. +7. Keep the final response short unless the task is a plan or recovery turn. _Twin agent v{{TWIN_VERSION}} — generated {{GENERATED_DATE}} from {{PROMPT_COUNT}} prompts._ diff --git a/skills/digital-twin/references/twin-spec-schema.json b/skills/digital-twin/references/twin-spec-schema.json index 952961c..f6e156c 100644 --- a/skills/digital-twin/references/twin-spec-schema.json +++ b/skills/digital-twin/references/twin-spec-schema.json @@ -1,10 +1,14 @@ { "$schema": "https://json-schema.org/draft-07/schema#", "title": "digital-twin behavioral twin spec", - "description": "Operational behavior contract used to render ~/.claude/agents/twin.md and generated CLAUDE rules. This is separate from PROFILE insights: it must describe how the agent should act.", + "description": "Operational substitution contract used to render ~/.claude/agents/twin.md and generated CLAUDE rules. This is separate from PROFILE insights: it must describe how the twin should act as the user's delegate.", "type": "object", "required": [ "identity", + "constitution", + "substitution_contract", + "trust_policy", + "agent_supervision_policy", "operating_model", "decision_policy", "delegation_policy", @@ -32,6 +36,79 @@ } } }, + "constitution": { + "type": "object", + "required": ["values", "judgment_rules", "evidence"], + "properties": { + "values": { + "type": "array", + "minItems": 3, + "items": { + "type": "object", + "required": ["name", "principle", "because", "tradeoffs", "evidence"], + "properties": { + "name": {"type": "string"}, + "principle": {"type": "string"}, + "because": {"type": "string"}, + "tradeoffs": {"type": "string"}, + "evidence": {"type": "string"} + } + } + }, + "judgment_rules": { + "type": "array", + "minItems": 3, + "items": { + "type": "object", + "required": ["situation", "reasoning", "preferred_action", "avoid", "evidence"], + "properties": { + "situation": {"type": "string"}, + "reasoning": {"type": "string"}, + "preferred_action": {"type": "string"}, + "avoid": {"type": "string"}, + "evidence": {"type": "string"} + } + } + }, + "evidence": {"type": "string"} + } + }, + "substitution_contract": { + "type": "object", + "required": ["role", "autonomous_authority", "user_reserved_authority", "delegation_authority", "supervision_stance", "boundaries", "evidence"], + "properties": { + "role": {"type": "string"}, + "autonomous_authority": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "user_reserved_authority": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "delegation_authority": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "supervision_stance": {"type": "string"}, + "boundaries": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "evidence": {"type": "string"} + } + }, + "trust_policy": { + "type": "object", + "required": ["trust_signals", "distrust_signals", "evidence_requirements", "interruption_triggers", "escalation_threshold", "evidence"], + "properties": { + "trust_signals": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "distrust_signals": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "evidence_requirements": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "interruption_triggers": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "escalation_threshold": {"type": "string"}, + "evidence": {"type": "string"} + } + }, + "agent_supervision_policy": { + "type": "object", + "required": ["briefing_requirements", "review_actions", "correction_actions", "completion_standard", "evidence"], + "properties": { + "briefing_requirements": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "review_actions": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "correction_actions": {"type": "array", "minItems": 1, "items": {"type": "string"}}, + "completion_standard": {"type": "string"}, + "evidence": {"type": "string"} + } + }, "operating_model": { "type": "object", "required": ["default_stance", "autonomy_level", "planning_threshold", "quality_bar", "evidence"], @@ -149,6 +226,12 @@ "rank": {"type": "integer", "minimum": 1}, "title": {"type": "string"}, "rule": {"type": "string"}, + "principle": {"type": "string"}, + "because": {"type": "string"}, + "applies_when": {"type": "string"}, + "failure_mode": {"type": "string"}, + "example_good": {"type": "string"}, + "example_bad": {"type": "string"}, "evidence": {"type": "string"} } } @@ -163,6 +246,12 @@ "rank": {"type": "integer", "minimum": 1}, "title": {"type": "string"}, "rule": {"type": "string"}, + "principle": {"type": "string"}, + "because": {"type": "string"}, + "applies_when": {"type": "string"}, + "failure_mode": {"type": "string"}, + "example_good": {"type": "string"}, + "example_bad": {"type": "string"}, "evidence": {"type": "string"} } } diff --git a/skills/digital-twin/scripts/evaluate-twin.py b/skills/digital-twin/scripts/evaluate-twin.py index 69a96a6..5c4fe3e 100644 --- a/skills/digital-twin/scripts/evaluate-twin.py +++ b/skills/digital-twin/scripts/evaluate-twin.py @@ -27,6 +27,19 @@ def _contains_any(text: str, values: list[str]) -> bool: def score_response(response: str, expected: dict) -> dict: + """Score a candidate response against an expected-behavior dict. + + Two distinct phrase-list fields, intentionally separate: + - `avoid_phrases` → tied to `pushback_trigger`; emits `avoidance_match` + and contributes to the trigger-avoidance rollup metric. + - `forbidden_phrases` → universal: any case can list phrases the twin + must never produce, regardless of trigger semantics. Emits + `forbidden_match`, scored in every case where the field is present. + + Both apply lower-cased substring matching, but they answer different + product questions: "did the twin avoid the language we want suppressed + on pushbacks?" vs. "did the twin never emit this language at all?". + """ response = response.strip() scores = {} @@ -58,6 +71,17 @@ def score_response(response: str, expected: dict) -> dict: avoid_phrases = expected.get("avoid_phrases") or [] scores["avoidance_match"] = 1 if not _contains_any(response, avoid_phrases) else 0 + + concept_groups = expected.get("concept_groups") or [] + if concept_groups: + scores["concept_coverage"] = 1 if all( + _contains_any(response, group) for group in concept_groups + ) else 0 + + forbidden_phrases = expected.get("forbidden_phrases") or [] + if forbidden_phrases: + scores["forbidden_match"] = 1 if not _contains_any(response, forbidden_phrases) else 0 + max_score = len(scores) scores["total"] = sum(scores.values()) scores["max"] = max_score @@ -65,32 +89,62 @@ def score_response(response: str, expected: dict) -> dict: def evaluate(cases: list[dict]) -> dict: + """Aggregate per-case scores into rollup metrics. + + `category_scores` averages per-case ratios (twin.total / twin.max) so + cases that score additional optional checks (concept_coverage, + forbidden_match) do not inflate the denominator and dilute thinner + cases in the same category. `pushback_trigger_hit_rate` reports the + rate of successful recoveries on triggers; avoidance is reported + separately on the row so the aggregate is not collapsed when triggers + omit `avoid_phrases`. + """ rows = [] twin_wins = 0 - trigger_hits = 0 + trigger_recovery_hits = 0 + trigger_avoid_hits = 0 trigger_total = 0 + trigger_avoid_total = 0 + by_category: dict[str, list[float]] = {} for case in cases: expected = case.get("expected") or {} twin = score_response(case.get("twin_response", ""), expected) generic = score_response(case.get("generic_response", ""), expected) + category = case.get("category") or "uncategorized" + ratio = (twin["total"] / twin["max"]) if twin["max"] else 0.0 + by_category.setdefault(category, []).append(ratio) if twin["total"] > generic["total"]: twin_wins += 1 if expected.get("pushback_trigger"): trigger_total += 1 - if twin.get("avoidance_match") and twin.get("recovery_quality"): - trigger_hits += 1 + if twin.get("recovery_quality"): + trigger_recovery_hits += 1 + if expected.get("avoid_phrases"): + trigger_avoid_total += 1 + if twin.get("avoidance_match"): + trigger_avoid_hits += 1 rows.append({ "id": case.get("id"), - "category": case.get("category"), + "category": category, "twin": twin, "generic": generic, "winner": "twin" if twin["total"] > generic["total"] else "generic_or_tie", }) n = len(cases) + category_scores = { + key: round(sum(ratios) / len(ratios), 3) if ratios else 0.0 + for key, ratios in by_category.items() + } return { "n_cases": n, "twin_win_rate": round(twin_wins / n, 3) if n else 0.0, - "pushback_trigger_hit_rate": round(trigger_hits / trigger_total, 3) if trigger_total else None, + "pushback_trigger_hit_rate": ( + round(trigger_recovery_hits / trigger_total, 3) if trigger_total else None + ), + "pushback_trigger_avoidance_rate": ( + round(trigger_avoid_hits / trigger_avoid_total, 3) if trigger_avoid_total else None + ), + "category_scores": category_scores, "rows": rows, } diff --git a/skills/digital-twin/scripts/extract-twin-spec.py b/skills/digital-twin/scripts/extract-twin-spec.py index 3884c69..3adcc64 100644 --- a/skills/digital-twin/scripts/extract-twin-spec.py +++ b/skills/digital-twin/scripts/extract-twin-spec.py @@ -4,7 +4,7 @@ Reads Phase 5 deep-read reports, Phase 5.5 insights, quantitative stats, and Phase 4 deep-source inventories, then asks Claude to produce a compact -operational spec used by synthesize.py to render ~/.claude/agents/twin.md and +substitution spec used by synthesize.py to render ~/.claude/agents/twin.md and generated CLAUDE rules. Outputs: @@ -38,7 +38,8 @@ def load_text(path: Path) -> str: try: return path.read_text(encoding="utf-8") - except OSError: + except OSError as exc: + print(f"WARN: could not read {path}: {exc}", file=sys.stderr) return "" @@ -48,7 +49,8 @@ def load_json(path: Path, default=None): try: with open(path, encoding="utf-8") as fp: return json.load(fp) - except (OSError, json.JSONDecodeError): + except (OSError, json.JSONDecodeError) as exc: + print(f"WARN: could not load JSON {path}: {exc}", file=sys.stderr) return default @@ -63,13 +65,20 @@ def build_stats_packet(analysis_dir: Path) -> str: data = load_json(analysis_dir / fname) if data is None: continue + if not isinstance(data, dict): + print( + f"WARN: {analysis_dir / fname} did not contain a JSON object; skipping", + file=sys.stderr, + ) + continue if key == "numbers": data.pop("vocab", None) data.pop("top_unigrams", None) data.pop("top_bigrams", None) packet[key] = data - mem = load_json(analysis_dir / "memory-inventory.json", default={}) or {} + mem_raw = load_json(analysis_dir / "memory-inventory.json", default={}) or {} + mem = mem_raw if isinstance(mem_raw, dict) else {} packet["memory_inventory_summary"] = { "n_files": mem.get("n_files"), "by_type": mem.get("by_type"), @@ -187,7 +196,21 @@ def main() -> int: return 0 if args.allow_empty else 2 if args.mock_response_file: - raw = load_text(Path(args.mock_response_file)) + mock_path = Path(args.mock_response_file).expanduser() + if not mock_path.exists(): + print( + f"ERROR: --mock-response-file path not found: {mock_path}", + file=sys.stderr, + ) + return 2 + try: + raw = mock_path.read_text(encoding="utf-8") + except OSError as exc: + print( + f"ERROR: --mock-response-file unreadable ({mock_path}): {exc}", + file=sys.stderr, + ) + return 2 else: schema_json = load_text(SCHEMA_PATH) prompt = fill_prompt( diff --git a/skills/digital-twin/scripts/pushback-detector.py b/skills/digital-twin/scripts/pushback-detector.py index 4d712ac..610bb1c 100755 --- a/skills/digital-twin/scripts/pushback-detector.py +++ b/skills/digital-twin/scripts/pushback-detector.py @@ -40,9 +40,15 @@ from safe_paths import is_safe_input_file APPROVAL_WORDS = { - "proceed", "continue", "yes", "go", "ok", "okay", "sounds", "great", + "proceed", "continue", "yes", "go", "ok", "okay", "great", "perfect", "ship", "merge", "do", "lgtm", "approved", } +# "sounds" alone is ambiguous (sounds good vs. sounds wrong), so require a +# two-token confirmation pattern at the head of the reply. +SOUNDS_APPROVAL_RE = re.compile( + r"^\s*sounds\s+(?:good|right|fine|great|reasonable|like a plan)\b", + re.IGNORECASE, +) EXPLICIT_PUSHBACK = { "stop", "wait", "no", "don't", "dont", "actually", "but", "however", "hold", "pause", "revert", "rollback", "halt", "abort", @@ -116,6 +122,8 @@ def classify(reply: str, approved_median: float) -> tuple[str, float]: return ("explicit_pushback", 0.9) if fw in APPROVAL_WORDS: return ("approval", 0.95) + if fw == "sounds" and SOUNDS_APPROVAL_RE.match(reply): + return ("approval", 0.9) long_enough = len(reply) >= 2 * approved_median and approved_median > 0 marker = DISSATISFACTION_MARKERS.search(reply) if long_enough and marker: @@ -189,10 +197,48 @@ def load_existing_descriptions(projects_root: Path) -> list[str]: return descs +SCAFFOLD_SECTION_HEADERS = ( + "Underlying principle", + "Rationale", + "Applies when", + "Does not apply when", + "Failure mode", + "Trust/delegation implication", +) + + +def proposal_ready_for_approval(body: str) -> tuple[bool, list[str]]: + """Return (is_ready, missing_or_unfilled_sections) for a proposal body. + + A proposal is ready for approval when every scaffold section header is + present AND its line no longer contains the `_Fill in ..._` placeholder. + Used by /digital-twin:propose-rules and by tests that guard the guard. + """ + missing: list[str] = [] + for header in SCAFFOLD_SECTION_HEADERS: + pattern = re.compile( + rf"^\*\*{re.escape(header)}:\*\*\s*(.+)$", + re.MULTILINE, + ) + match = pattern.search(body) + if not match: + missing.append(f"{header} (section missing)") + continue + line = match.group(1).strip() + if not line or line.startswith("_Fill in") or line == "_": + missing.append(f"{header} (unfilled scaffold)") + return (not missing, missing) + + def proposal_body(reply: str, asst: str, project: str, dt_iso: str) -> tuple[str, str]: """Return (slug, full_markdown_for_proposal_file).""" summary = first_sentence(reply, 120) - name = slugify(summary, max_len=40) + base_slug = slugify(summary, max_len=32) + # Append a short content-hash suffix so two proposals built from the same + # leading sentence (or empty/no-alphanum replies that collapse to "rule") + # do not collide on the frontmatter `name:` field used by memory dedup. + hash_suffix = content_hash(reply)[:8] + name = f"{base_slug}_{hash_suffix}" description = first_sentence(reply, 150).replace("\n", " ") body = ( f"---\n" @@ -202,10 +248,15 @@ def proposal_body(reply: str, asst: str, project: str, dt_iso: str) -> tuple[str f"---\n\n" f"\n" f"\n" - f"\n\n" + f"\n\n" + f"## Judgment correction\n\n" f"{first_sentence(reply, 400)}\n\n" - f"**Why:** _Fill in the reason — what past incident or preference does this encode?_\n\n" - f"**How to apply:** _Fill in when this rule should kick in._\n\n" + f"**Underlying principle:** _Fill in the transferable judgment this correction teaches._\n\n" + f"**Rationale:** _Fill in why this matters for acting as the user would._\n\n" + f"**Applies when:** _Fill in the situations where this principle should kick in._\n\n" + f"**Does not apply when:** _Fill in boundaries so the twin does not overgeneralize._\n\n" + f"**Failure mode:** _Fill in what the agent did wrong here._\n\n" + f"**Trust/delegation implication:** _Fill in whether this changes when to trust, interrupt, brief, or redirect other agents._\n\n" f"---\n\n" f"## Evidence (from session)\n\n" f"**Assistant said:**\n> {first_sentence(asst, 300).replace(chr(10), ' ')}\n\n" @@ -261,11 +312,25 @@ def main() -> int: if state_file.exists() and not args.reset_state: try: with open(state_file, encoding="utf-8") as fp: - state = json.load(fp) - except (OSError, json.JSONDecodeError): - pass + loaded = json.load(fp) + if isinstance(loaded, dict): + state = loaded + else: + print( + f"WARN: state file {state_file} did not contain a JSON object; rescanning from scratch", + file=sys.stderr, + ) + except (OSError, json.JSONDecodeError) as exc: + print( + f"WARN: state file {state_file} unreadable ({exc}); rescanning from scratch", + file=sys.stderr, + ) state.setdefault("offsets", {}) state.setdefault("seen_hashes", []) + if not isinstance(state["offsets"], dict): + state["offsets"] = {} + if not isinstance(state["seen_hashes"], list): + state["seen_hashes"] = [] seen_hashes: set[str] = set(state["seen_hashes"]) # Existing memory descriptions (for de-duplication against user's rules) @@ -284,6 +349,7 @@ def main() -> int: if is_safe_input_file(f, source) ] new_pairs: list[tuple[str, str, str, str]] = [] # (asst, reply, project, dt_iso) + corrupt_lines = 0 for fpath in files: if since_ts and os.path.getmtime(fpath) < since_ts: @@ -291,7 +357,8 @@ def main() -> int: prev_offset = state["offsets"].get(fpath, 0) try: size = os.path.getsize(fpath) - except OSError: + except OSError as exc: + print(f"WARN: could not stat {fpath}: {exc}", file=sys.stderr) continue if prev_offset > size: prev_offset = 0 # file rotated/truncated @@ -302,7 +369,8 @@ def main() -> int: last_dt: str = "" try: fp = open(fpath, encoding="utf-8", errors="replace") - except OSError: + except OSError as exc: + print(f"WARN: could not open {fpath}: {exc}", file=sys.stderr) continue try: fp.seek(prev_offset) @@ -317,6 +385,7 @@ def main() -> int: try: obj = json.loads(line) except json.JSONDecodeError: + corrupt_lines += 1 continue t = obj.get("type") ts = obj.get("timestamp") or obj.get("ts") or "" @@ -348,7 +417,8 @@ def main() -> int: if approved_median <= 0: approved_lens = [ len(r) for _a, r, _p, _t in new_pairs - if first_word(r) in APPROVAL_WORDS + if (first_word(r) in APPROVAL_WORDS) + or (first_word(r) == "sounds" and SOUNDS_APPROVAL_RE.match(r)) ] if approved_lens: approved_lens.sort() @@ -366,6 +436,8 @@ def main() -> int: continue if conf < args.min_confidence: continue + if not reply.strip(): + continue if covered_by_existing(reply, existing_descriptions): continue h = content_hash(reply) @@ -383,6 +455,11 @@ def main() -> int: candidates = sorted(unique.values(), key=lambda x: -x[0])[: args.max_proposals] print(f"Scanned {len(new_pairs)} new pair(s).") + if corrupt_lines: + print( + f"WARN: skipped {corrupt_lines} corrupt JSONL line(s) during scan.", + file=sys.stderr, + ) print(f"Approved-reply median (threshold anchor): {approved_median:.0f} chars.") print(f"Existing rule descriptions known: {len(existing_descriptions)}.") print(f"Candidate proposals after filters: {len(candidates)}.") diff --git a/skills/digital-twin/scripts/synthesize.py b/skills/digital-twin/scripts/synthesize.py index 3de60c9..0f947f1 100755 --- a/skills/digital-twin/scripts/synthesize.py +++ b/skills/digital-twin/scripts/synthesize.py @@ -23,6 +23,7 @@ import getpass import html import importlib.util +import copy import json import os import re @@ -104,6 +105,23 @@ def load_text(path: Path, default: str = "") -> str: return default +_USER_NAME_ALLOWED = re.compile(r"[^A-Za-z0-9 ._\-]") + + +def sanitize_user_name(value: str, max_len: int = 64) -> str: + """Strip control chars, cap length, restrict to a safe identifier subset. + + user_name is interpolated into rendered markdown (twin agent, gotchas, + CLAUDE patch). Newlines or markdown control sequences would inject + content into generated files; oversize values bloat every render. Local + env-var defaults are still trusted, but we normalize before use. + """ + text = str(value or "").strip() + text = _USER_NAME_ALLOWED.sub("", text) + text = text[:max_len].strip() + return text or "user" + + def fill(template: str, ctx: dict) -> str: def repl(match: re.Match) -> str: key = match.group(1) @@ -1682,6 +1700,194 @@ def _with_evidence(text: str, evidence: str | None) -> str: return f"{text} _(evidence: {ev})_" if ev else text +_DESTRUCTIVE_AUTHORITY_PATTERNS = ( + "force-push", + "force push", + "publish", + "release", + "delete", + " rm ", + "rm -", + "drop ", + "truncate", + "deploy to prod", + "deploy-to-prod", + "merge to main", + "merge into main", +) + + +def _safe_dict(spec: dict, key: str) -> dict: + value = spec.get(key) + return value if isinstance(value, dict) else {} + + +def _filter_destructive_authority(items: list[str]) -> list[str]: + """Drop legacy decide_alone items that name destructive actions. + + Legacy specs predate the substitution contract, so any value claiming + irreversible authority must not silently become autonomous_authority. + """ + out: list[str] = [] + for item in items: + if not isinstance(item, str): + continue + lowered = f" {item.lower()} " + if any(pat in lowered for pat in _DESTRUCTIVE_AUTHORITY_PATTERNS): + continue + out.append(item) + return out + + +def _legacy_substitution_fields(spec: dict, user_name: str) -> dict: + """Derive substitution fields for v1 specs generated before this layer.""" + op = _safe_dict(spec, "operating_model") + decision = _safe_dict(spec, "decision_policy") + delegation = _safe_dict(spec, "delegation_policy") + verification = _safe_dict(spec, "verification_policy") + recovery = _safe_dict(spec, "recovery_policy") + evidence = "derived from legacy twin-spec fields" + return { + "constitution": { + "values": [ + { + "name": "Act as the user's delegate", + "principle": op.get("default_stance") + or f"Run work as {user_name} would, grounded in local evidence.", + "because": "The twin's job is to keep the user's work moving when the user is absent.", + "tradeoffs": op.get("autonomy_level") + or "Prefer autonomous execution on discoverable facts; reserve irreversible decisions for the real user.", + "evidence": op.get("evidence") or evidence, + }, + { + "name": "Demand proof before trust", + "principle": op.get("quality_bar") + or "Completion claims require fresh, inspectable evidence.", + "because": "The user has been burned by agent claims that lacked artifacts; trust is earned per task.", + "tradeoffs": "Slow down to verify when agent output affects correctness, review, or shipping.", + "evidence": verification.get("evidence") or evidence, + }, + { + "name": "Correct by principle, not by patch", + "principle": "After pushback, identify the failed judgment and recover serially.", + "because": "Spot-fixing without root cause causes the same failure to resurface in adjacent work.", + "tradeoffs": "Convergence matters more than continuing parallel throughput during recovery.", + "evidence": recovery.get("evidence") or evidence, + }, + ], + "judgment_rules": [ + { + "situation": "Operational details are discoverable", + "reasoning": decision.get("default_assumption") + or "The user expects the twin to read context and decide.", + "preferred_action": "; ".join( + _filter_destructive_authority(_spec_list(decision.get("decide_alone")))[:3] + ) + or "Read local evidence, choose the conservative project-conventional action, and proceed.", + "avoid": "Asking the user for facts available in the repo or plan.", + "evidence": decision.get("evidence") or evidence, + }, + { + "situation": "Work splits into independent areas", + "reasoning": "The user delegates independent work to agents and keeps the main thread as coordinator.", + "preferred_action": "; ".join(_spec_list(delegation.get("parallel_triggers"))[:3]) + or "Brief separate agents with self-contained tasks and consolidate evidence.", + "avoid": "Serially doing independent review or implementation work when parallelism is available.", + "evidence": delegation.get("evidence") or evidence, + }, + { + "situation": "Agent output is weak, unverified, or challenged", + "reasoning": "Trust is earned through artifacts and corrected through concrete gap analysis.", + "preferred_action": "; ".join(_spec_list(recovery.get("required_steps"))[:3]) + or "Name the gap, require evidence, and ask one binary recovery question.", + "avoid": "Continuing with generic reassurance or another broad attempt.", + "evidence": recovery.get("evidence") or evidence, + }, + ], + "evidence": evidence, + }, + "substitution_contract": { + "role": f"Act as {user_name}'s operational delegate for orchestrating Claude Code work when {user_name} is absent.", + "autonomous_authority": _filter_destructive_authority(_spec_list(decision.get("decide_alone"))) + or ["Read/search files", "Brief agents", "Review agent output", "Run non-destructive verification"], + "user_reserved_authority": _spec_list(decision.get("escalate")) + or ["Irreversible actions", "Scope changes", "External commitments"], + "delegation_authority": _spec_list(delegation.get("parallel_triggers")) + or ["Delegate independent work to agents when tasks do not conflict"], + "supervision_stance": delegation.get("worktree_policy") + or "Coordinate agents, require evidence, and converge work through review and correction.", + "boundaries": [ + "Do not exceed the user's documented escalation gates.", + "Surface uncertainty when the corpus lacks evidence for a user-like decision.", + "Respect project-local instructions over global defaults.", + ], + "evidence": decision.get("evidence") or delegation.get("evidence") or evidence, + }, + "trust_policy": { + "trust_signals": _spec_list(verification.get("fresh_evidence_examples")) + or ["Fresh tests", "CI evidence", "Runtime artifacts"], + "distrust_signals": _spec_list(verification.get("forbidden_claims")) + or ["Claims without artifacts", "Intent-only completion", "Stale memory"], + "evidence_requirements": _spec_list(verification.get("completion_claim_requires")) + or ["Fresh command output or artifact evidence"], + "interruption_triggers": _spec_list(recovery.get("pushback_signals")) + or ["Pushback words", "Contradictory evidence", "Repeated unresolved cycles"], + "escalation_threshold": recovery.get("long_tail_escalation") + or "Escalate after repeated mismatch or when required authority is reserved for the user.", + "evidence": verification.get("evidence") or recovery.get("evidence") or evidence, + }, + "agent_supervision_policy": { + "briefing_requirements": [ + "State scope, expected output, evidence requirements, and files or project context to inspect.", + "Make each agent brief self-contained when delegating in parallel.", + "Define what counts as done before the agent starts.", + ], + "review_actions": [ + "Check agent claims against file, command, or artifact evidence.", + "Classify findings by severity when reviewing implementation work.", + "Challenge vague claims, missing tests, missing runtime evidence, and scope drift.", + ], + "correction_actions": _spec_list(recovery.get("required_steps")) + or ["Concede the gap", "Name what failed", "Require a concrete correction path"], + "completion_standard": op.get("quality_bar") + or "Accept agent work only when it meets the user's verification and quality bar.", + "evidence": delegation.get("evidence") or verification.get("evidence") or evidence, + }, + } + + +_SUBSTITUTION_SECTIONS = ( + "constitution", + "substitution_contract", + "trust_policy", + "agent_supervision_policy", +) + + +def needs_compatibility_defaults(spec: dict) -> bool: + """Detect whether a spec lacks usable substitution sections. + + Treats missing, empty, AND non-dict values as needing backfill, so a + legacy spec with garbage values (e.g. a string placeholder) is repaired + rather than crashing the renderer downstream. + """ + if not isinstance(spec, dict): + return False + return any(not isinstance(spec.get(k), dict) or not spec.get(k) for k in _SUBSTITUTION_SECTIONS) + + +def normalize_twin_spec_for_rendering(spec: dict, user_name: str) -> dict: + if not isinstance(spec, dict): + return spec + normalized = copy.deepcopy(spec) + defaults = _legacy_substitution_fields(normalized, user_name) + for key, value in defaults.items(): + current = normalized.get(key) + if not isinstance(current, dict) or not current: + normalized[key] = value + return normalized + + def render_identity(spec: dict) -> str: rows = [] for item in spec.get("identity") or []: @@ -1693,6 +1899,124 @@ def render_identity(spec: dict) -> str: return bulleted_list(rows) if rows else "- Behavioral spec incomplete; rerun `extract-twin-spec.py`." +def render_constitution(spec: dict) -> str: + pol = spec.get("constitution") or {} + parts = [] + values = [] + for value in pol.get("values") or []: + if not isinstance(value, dict): + continue + name = _spec_text(value.get("name")) + principle = _spec_text(value.get("principle")) + because = _spec_text(value.get("because")) + tradeoffs = _spec_text(value.get("tradeoffs")) + evidence = _spec_text(value.get("evidence")) + if not (name or principle): + continue + line = f"**{name}** — {principle}" if name and principle else name or principle + details = [] + if because: + details.append(f"Because: {because}") + if tradeoffs: + details.append(f"Tradeoffs: {tradeoffs}") + if evidence: + details.append(f"Evidence: {evidence}") + if details: + line += "\n " + "\n ".join(f"- {d}" for d in details) + values.append(line) + if values: + parts.append("Values:\n" + numbered_list(values)) + + judgments = [] + for rule in pol.get("judgment_rules") or []: + if not isinstance(rule, dict): + continue + situation = _spec_text(rule.get("situation")) + reasoning = _spec_text(rule.get("reasoning")) + preferred = _spec_text(rule.get("preferred_action")) + avoid = _spec_text(rule.get("avoid")) + evidence = _spec_text(rule.get("evidence")) + if not situation: + continue + block = [f"**{situation}**"] + if reasoning: + block.append(f"Reasoning: {reasoning}") + if preferred: + block.append(f"Preferred action: {preferred}") + if avoid: + block.append(f"Avoid: {avoid}") + if evidence: + block.append(f"Evidence: {evidence}") + judgments.append("\n ".join(block)) + if judgments: + parts.append("Judgment rules:\n" + numbered_list(judgments)) + if pol.get("evidence"): + parts.append(f"Evidence: {pol.get('evidence')}") + return "\n\n".join(parts) if parts else "_No constitution extracted._" + + +def render_substitution_contract(spec: dict) -> str: + pol = spec.get("substitution_contract") or {} + parts = [] + role = _spec_text(pol.get("role")) + if role: + parts.append(f"Role: {role}") + for title, key in ( + ("Autonomous authority", "autonomous_authority"), + ("Reserved for the real user", "user_reserved_authority"), + ("Delegation authority", "delegation_authority"), + ("Boundaries", "boundaries"), + ): + vals = _spec_list(pol.get(key)) + if vals: + parts.append(f"{title}:\n" + bulleted_list(vals)) + stance = _spec_text(pol.get("supervision_stance")) + if stance: + parts.append(f"Supervision stance: {stance}") + if pol.get("evidence"): + parts.append(f"Evidence: {pol.get('evidence')}") + return "\n\n".join(parts) if parts else "_No substitution contract extracted._" + + +def render_trust_policy(spec: dict) -> str: + pol = spec.get("trust_policy") or {} + parts = [] + for title, key in ( + ("Trust agent output when", "trust_signals"), + ("Withhold trust when", "distrust_signals"), + ("Evidence requirements", "evidence_requirements"), + ("Interrupt or redirect when", "interruption_triggers"), + ): + vals = _spec_list(pol.get(key)) + if vals: + parts.append(f"{title}:\n" + bulleted_list(vals)) + threshold = _spec_text(pol.get("escalation_threshold")) + if threshold: + parts.append(f"Escalation threshold: {threshold}") + if pol.get("evidence"): + parts.append(f"Evidence: {pol.get('evidence')}") + return "\n\n".join(parts) if parts else "_No trust policy extracted._" + + +def render_agent_supervision_policy(spec: dict) -> str: + pol = spec.get("agent_supervision_policy") or {} + parts = [] + for title, key in ( + ("Brief agents with", "briefing_requirements"), + ("Review agent work by", "review_actions"), + ("Correct agents by", "correction_actions"), + ): + vals = _spec_list(pol.get(key)) + if vals: + parts.append(f"{title}:\n" + bulleted_list(vals)) + standard = _spec_text(pol.get("completion_standard")) + if standard: + parts.append(f"Completion standard: {standard}") + if pol.get("evidence"): + parts.append(f"Evidence: {pol.get('evidence')}") + return "\n\n".join(parts) if parts else "_No agent supervision policy extracted._" + + def render_operating_model(spec: dict) -> str: op = spec.get("operating_model") or {} rows = [] @@ -1847,7 +2171,12 @@ def render_project_routing(spec: dict) -> str: return bulleted_list(rows) if rows else "- Unknown project: read local instructions and ask only if conventions cannot be discovered." -def render_rule_set(spec: dict, key: str, limit: int | None = None) -> str: +def render_rule_set( + spec: dict, + key: str, + limit: int | None = None, + detail_mode: str = "full", +) -> str: rules = spec.get(key) or [] if limit is not None: rules = rules[:limit] @@ -1861,7 +2190,31 @@ def render_rule_set(spec: dict, key: str, limit: int | None = None) -> str: if not title and not body: continue line = f"**{title}** — {body}" if title and body else title or body - rows.append(_with_evidence(line, ev)) + details = [] + all_detail_fields: tuple[tuple[str, str], ...] = ( + ("Principle", "principle"), + ("Because", "because"), + ("Applies when", "applies_when"), + ("Failure mode", "failure_mode"), + ("Good example", "example_good"), + ("Bad example", "example_bad"), + ) + detail_fields: tuple[tuple[str, str], ...] + if detail_mode == "principle": + detail_fields = all_detail_fields[:2] + elif detail_mode == "none": + detail_fields = () + else: + detail_fields = all_detail_fields + for label, detail_key in detail_fields: + val = _spec_text(rule.get(detail_key)) + if val: + details.append(f"{label}: {val}") + if ev and detail_mode == "full": + details.append(f"Evidence: {ev}") + if details: + line += "\n " + "\n ".join(f"- {d}" for d in details) + rows.append(line) return numbered_list(rows) if rows else "_No rules extracted._" @@ -1909,6 +2262,79 @@ def build_degraded_twin_spec(user_name: str, reason: str = "analysis/twin-spec.j "evidence": "Behavioral Twin v1 pipeline", }, ], + "constitution": { + "values": [ + { + "name": "Do not pretend to substitute", + "principle": "A degraded twin must not act as a replacement for the user.", + "because": "The behavioral corpus has not been distilled into a complete substitution contract.", + "tradeoffs": "Prefer conservative assistance over user-like autonomous orchestration.", + "evidence": "degraded fallback", + }, + { + "name": "Require fresh evidence", + "principle": "Do not claim completion without artifact-backed verification.", + "because": "Verification is the minimum safe default when user-specific trust behavior is unavailable.", + "tradeoffs": "Slower completion claims are better than false confidence.", + "evidence": "degraded fallback", + }, + { + "name": "Escalate real authority", + "principle": "Reserve irreversible or external commitments for the real user.", + "because": "The fallback has no corpus-backed authority model.", + "tradeoffs": "Ask for approval on high-authority actions even if a complete twin might decide more autonomously.", + "evidence": "degraded fallback", + }, + ], + "judgment_rules": [ + { + "situation": "Conventions are discoverable", + "reasoning": "Local evidence is safer than guessing or asking premature questions.", + "preferred_action": "Read local instructions and relevant files before asking.", + "avoid": "Inventing project-specific user intent.", + "evidence": "degraded fallback", + }, + { + "situation": "Action is irreversible", + "reasoning": "The fallback lacks the user's authority boundaries.", + "preferred_action": "Stop and ask the real user.", + "avoid": "Acting as a substitute decision-maker.", + "evidence": "degraded fallback", + }, + { + "situation": "Pushback happens", + "reasoning": "Correction requires naming the gap before continuing.", + "preferred_action": "Concede, name the gap, and ask one binary question.", + "avoid": "Continuing with broad autonomous execution.", + "evidence": "degraded fallback", + }, + ], + "evidence": "degraded fallback", + }, + "substitution_contract": { + "role": "Incomplete fallback. Assist the user, but do not claim to substitute for them.", + "autonomous_authority": ["Read/search files", "Summarize findings", "Run non-destructive checks"], + "user_reserved_authority": ["Destructive commands", "Merge/release/publish", "External commitments", "Ambiguous product decisions"], + "delegation_authority": ["Do not delegate as the user unless explicitly instructed."], + "supervision_stance": "Review agent output conservatively and surface uncertainty.", + "boundaries": ["This fallback is not a replacement twin.", "Regenerate `analysis/twin-spec.json` before autonomous orchestration."], + "evidence": "degraded fallback", + }, + "trust_policy": { + "trust_signals": ["Fresh command output", "Artifact evidence", "File citations"], + "distrust_signals": ["Intent-only claims", "Stale memory", "Missing evidence"], + "evidence_requirements": ["Fresh command output or artifact evidence"], + "interruption_triggers": ["Destructive action", "Scope change", "Repeated mismatch"], + "escalation_threshold": "Ask the real user for any action outside non-destructive assistance.", + "evidence": "degraded fallback", + }, + "agent_supervision_policy": { + "briefing_requirements": ["State task scope", "State expected evidence", "State output shape"], + "review_actions": ["Check claims against evidence", "Flag uncertainty", "Escalate authority gaps"], + "correction_actions": ["Concede", "Name the gap", "Ask one binary question"], + "completion_standard": "Do not accept delegated work without fresh evidence.", + "evidence": "degraded fallback", + }, "operating_model": { "default_stance": "Incomplete; read local instructions and avoid irreversible actions.", "autonomy_level": "Low until twin-spec.json exists.", @@ -1963,18 +2389,18 @@ def build_degraded_twin_spec(user_name: str, reason: str = "analysis/twin-spec.j "projects": [], }, "never_rules": [ - {"rank": 1, "title": "No fake completion", "rule": "Never claim done without fresh verification evidence.", "evidence": "degraded fallback"}, - {"rank": 2, "title": "No destructive action", "rule": "Never run destructive commands without approval.", "evidence": "degraded fallback"}, - {"rank": 3, "title": "No raw memory dump", "rule": "Do not treat profile output as an operating contract.", "evidence": "degraded fallback"}, - {"rank": 4, "title": "No scope expansion", "rule": "Do not expand beyond the active task without surfacing it.", "evidence": "degraded fallback"}, - {"rank": 5, "title": "No stale facts", "rule": "Do not rely on old session memory for branch or repo state.", "evidence": "degraded fallback"}, + {"rank": 1, "title": "No fake completion", "rule": "Never claim done without fresh verification evidence.", "principle": "Trust requires artifacts.", "because": "A fallback twin cannot infer the user's trust model.", "applies_when": "Any completion or status claim.", "failure_mode": "The agent sounds done when no evidence exists.", "evidence": "degraded fallback"}, + {"rank": 2, "title": "No destructive action", "rule": "Never run destructive commands without approval.", "principle": "Reserve irreversible authority for the real user.", "because": "The complete substitution contract is unavailable.", "applies_when": "Destructive commands, merge, release, publish, branch deletion.", "failure_mode": "The fallback oversteps user authority.", "evidence": "degraded fallback"}, + {"rank": 3, "title": "No raw memory dump", "rule": "Do not treat profile output as an operating contract.", "principle": "Profiles explain; specs direct.", "because": "A profile can contain stale or descriptive facts that are unsafe as policy.", "applies_when": "Using PROFILE.md or reports as instructions.", "failure_mode": "Descriptive observations become false authority.", "evidence": "degraded fallback"}, + {"rank": 4, "title": "No scope expansion", "rule": "Do not expand beyond the active task without surfacing it.", "principle": "Minimize blast radius.", "because": "Scope expansion is unsafe without a complete user judgment model.", "applies_when": "Side findings or adjacent cleanup appear.", "failure_mode": "Unreviewed extra work enters the change.", "evidence": "degraded fallback"}, + {"rank": 5, "title": "No stale facts", "rule": "Do not rely on old session memory for branch or repo state.", "principle": "Current state beats memory.", "because": "Repo state is mutable.", "applies_when": "Branch, file, PR, or issue state claims.", "failure_mode": "The agent acts on old state.", "evidence": "degraded fallback"}, ], "always_rules": [ - {"rank": 1, "title": "Read local context", "rule": "Read instructions and relevant files first.", "evidence": "degraded fallback"}, - {"rank": 2, "title": "Plan hard work", "rule": "Plan before non-trivial edits.", "evidence": "degraded fallback"}, - {"rank": 3, "title": "Verify", "rule": "Run relevant checks before claiming done.", "evidence": "degraded fallback"}, - {"rank": 4, "title": "Be terse", "rule": "Keep responses concise and concrete.", "evidence": "degraded fallback"}, - {"rank": 5, "title": "Escalate real ambiguity", "rule": "Ask only when facts cannot be discovered safely.", "evidence": "degraded fallback"}, + {"rank": 1, "title": "Read local context", "rule": "Read instructions and relevant files first.", "principle": "Discoverable facts should be discovered.", "because": "The fallback cannot infer user-specific conventions without local evidence.", "applies_when": "Entering a repo or unknown task.", "failure_mode": "Premature questions or wrong conventions.", "evidence": "degraded fallback"}, + {"rank": 2, "title": "Plan hard work", "rule": "Plan before non-trivial edits.", "principle": "Make uncertainty visible before editing.", "because": "Planning bounds risk when the complete behavior spec is missing.", "applies_when": "Multi-file or ambiguous work.", "failure_mode": "Implementation drifts before scope is clear.", "evidence": "degraded fallback"}, + {"rank": 3, "title": "Verify", "rule": "Run relevant checks before claiming done.", "principle": "Evidence earns trust.", "because": "Completion without evidence is not actionable.", "applies_when": "Any final response or handoff.", "failure_mode": "False completion claim.", "evidence": "degraded fallback"}, + {"rank": 4, "title": "Be terse", "rule": "Keep responses concise and concrete.", "principle": "Reduce cognitive load.", "because": "Fallback output should not bury uncertainty.", "applies_when": "Status and final responses.", "failure_mode": "Verbose prose hides the actual state.", "evidence": "degraded fallback"}, + {"rank": 5, "title": "Escalate real ambiguity", "rule": "Ask only when facts cannot be discovered safely.", "principle": "Autonomy stops at unsafe uncertainty.", "because": "The fallback has no complete user-substitution authority.", "applies_when": "Facts are unavailable or authority is reserved.", "failure_mode": "Guessing user intent.", "evidence": "degraded fallback"}, ], "examples": { "approved_turn": "Implemented and verified with fresh test output.", @@ -1986,15 +2412,25 @@ def build_degraded_twin_spec(user_name: str, reason: str = "analysis/twin-spec.j } -def render_twin_context(spec: dict, complete: bool, args) -> dict: - status = ( - "Behavioral spec complete. Use this as the operating contract." - if complete - else "INCOMPLETE BEHAVIORAL SPEC: this is a degraded fallback. Regenerate `analysis/twin-spec.json` before treating this as a replacement twin." - ) +def render_twin_context( + spec: dict, + complete: bool, + args, + compatibility_defaults: bool = False, +) -> dict: + if not complete: + status = "INCOMPLETE BEHAVIORAL SPEC: this is a degraded fallback. Regenerate `analysis/twin-spec.json` before treating this as a replacement twin." + elif compatibility_defaults: + status = "Behavioral spec valid with compatibility-derived substitution defaults. Refresh `analysis/twin-spec.json` before treating this as full delegate authority." + else: + status = "Behavioral spec complete. Use this as the operating contract." return { "TWIN_SPEC_STATUS": status, "IDENTITY_FACTS": render_identity(spec), + "CONSTITUTION_SECTION": render_constitution(spec), + "SUBSTITUTION_CONTRACT_SECTION": render_substitution_contract(spec), + "TRUST_POLICY_SECTION": render_trust_policy(spec), + "AGENT_SUPERVISION_SECTION": render_agent_supervision_policy(spec), "OPERATING_MODEL_SECTION": render_operating_model(spec), "DECISION_POLICY_SECTION": render_decision_policy(spec), "DELEGATION_POLICY_SECTION": render_delegation_policy(spec), @@ -2003,8 +2439,8 @@ def render_twin_context(spec: dict, complete: bool, args) -> dict: "RECOVERY_POLICY_SECTION": render_recovery_policy(spec), "VOICE_POLICY_SECTION": render_voice_policy(spec), "PROJECT_ROUTING_SECTION": render_project_routing(spec), - "NEVER_RULES_TOP": render_rule_set(spec, "never_rules", limit=12), - "ALWAYS_RULES_TOP": render_rule_set(spec, "always_rules", limit=12), + "NEVER_RULES_TOP": render_rule_set(spec, "never_rules", limit=8, detail_mode="principle"), + "ALWAYS_RULES_TOP": render_rule_set(spec, "always_rules", limit=8, detail_mode="principle"), "EXAMPLES_SECTION": render_examples(spec), "EVIDENCE_SECTION": render_evidence_map(spec), "RULES_REFERENCE_SECTION": ( @@ -2020,17 +2456,42 @@ def write_rules_files(out: Path, spec: dict, generated_date: str) -> dict[str, P rules_dir = out / "rules" rules_dir.mkdir(parents=True, exist_ok=True) files = { + "substitution": rules_dir / "substitution.md", "preferences": rules_dir / "preferences.md", "workflows": rules_dir / "workflows.md", "verification": rules_dir / "verification.md", "recovery": rules_dir / "recovery.md", } + files["substitution"].write_text( + "\n".join([ + "# Twin Substitution Contract", + "", + f"_Generated {generated_date} from behavioral twin spec._", + "", + "## Substitution Contract", + render_substitution_contract(spec), + "", + "## Constitution", + render_constitution(spec), + "", + "## Trust Policy", + render_trust_policy(spec), + "", + "## Agent Supervision", + render_agent_supervision_policy(spec), + "", + ]), + encoding="utf-8", + ) files["preferences"].write_text( "\n".join([ "# Twin Preferences", "", f"_Generated {generated_date} from behavioral twin spec._", "", + "## Constitution", + render_constitution(spec), + "", "## Identity", render_identity(spec), "", @@ -2052,6 +2513,9 @@ def write_rules_files(out: Path, spec: dict, generated_date: str) -> dict[str, P "", f"_Generated {generated_date} from behavioral twin spec._", "", + "## Substitution Contract", + render_substitution_contract(spec), + "", "## Operating Model", render_operating_model(spec), "", @@ -2061,6 +2525,12 @@ def write_rules_files(out: Path, spec: dict, generated_date: str) -> dict[str, P "## Delegation Policy", render_delegation_policy(spec), "", + "## Agent Supervision", + render_agent_supervision_policy(spec), + "", + "## Trust Policy", + render_trust_policy(spec), + "", "## Workflow Policy", render_workflow_policy(spec), "", @@ -2126,7 +2596,18 @@ def main() -> int: ap.add_argument("--user-name", default=os.environ.get("DIGITAL_TWIN_USER_NAME", getpass.getuser())) ap.add_argument("--profile-version", default="v0.1") ap.add_argument("--target-twin-reply-len", type=int, default=600) + ap.add_argument( + "--strict-substitution", + action="store_true", + help=( + "Refuse to backfill missing substitution sections from legacy " + "v1 specs. When set, a legacy spec without constitution / " + "substitution_contract / trust_policy / agent_supervision_policy " + "produces a degraded twin instead of a compatibility-derived one." + ), + ) args = ap.parse_args() + args.user_name = sanitize_user_name(args.user_name) analysis = Path(args.analysis).expanduser() reports = Path(args.reports).expanduser() @@ -2151,16 +2632,31 @@ def main() -> int: twin_spec_path = analysis / "twin-spec.json" twin_spec = load_json(twin_spec_path, default=None) twin_spec_complete = isinstance(twin_spec, dict) and bool(twin_spec) + twin_spec_compat_defaults = False if twin_spec_complete: - twin_spec_errors = validate_twin_spec(twin_spec, TWIN_SPEC_SCHEMA_PATH) - if twin_spec_errors: - twin_spec_complete = False + twin_spec_compat_defaults = needs_compatibility_defaults(twin_spec) + if twin_spec_compat_defaults and args.strict_substitution: reason = ( - f"{twin_spec_path} failed schema validation: " - + "; ".join(twin_spec_errors[:5]) + f"{twin_spec_path} missing substitution sections and " + "--strict-substitution is set; refusing to derive authority " + "from legacy v1 fields." ) print(f"WARN: {reason}", file=sys.stderr) twin_spec = build_degraded_twin_spec(args.user_name, reason=reason) + twin_spec_complete = False + twin_spec_compat_defaults = False + else: + twin_spec = normalize_twin_spec_for_rendering(twin_spec, args.user_name) + twin_spec_errors = validate_twin_spec(twin_spec, TWIN_SPEC_SCHEMA_PATH) + if twin_spec_errors: + twin_spec_complete = False + reason = ( + f"{twin_spec_path} failed schema validation: " + + "; ".join(twin_spec_errors[:5]) + ) + print(f"WARN: {reason}", file=sys.stderr) + twin_spec = build_degraded_twin_spec(args.user_name, reason=reason) + twin_spec_compat_defaults = False else: reason = f"{twin_spec_path} missing" print( @@ -2168,6 +2664,7 @@ def main() -> int: file=sys.stderr, ) twin_spec = build_degraded_twin_spec(args.user_name, reason=reason) + twin_spec_compat_defaults = False orchestration_report = load_text(reports / "orchestration.md") workflow_report = load_text(reports / "workflow.md") @@ -2462,7 +2959,7 @@ def main() -> int: "DEFAULT_PLAN_ARCHETYPE": "surgical for single-PR work, multi-phase only for >1 week scope", "ALWAYS_IN_PLANS": "Context, Goal, Approach, Out-of-scope, Verification", "VERIFICATION_GATE": "type check + tests + (UI: browser dogfood)", - "MERGE_CONVENTION": "_TBD_ (review your repo conventions)", + "MERGE_CONVENTION": "derive from repo conventions before acting", "QUALITY_BAR_TERSE": ( "- No unhandled edge cases at PR time\n" "- No backfill gaps for data migrations\n" @@ -2475,7 +2972,14 @@ def main() -> int: f"- `{path}`" for path in generated_rule_files.values() ), } - ctx.update(render_twin_context(twin_spec, twin_spec_complete, args)) + ctx.update( + render_twin_context( + twin_spec, + twin_spec_complete, + args, + compatibility_defaults=twin_spec_compat_defaults, + ) + ) # --- Profile (markdown) --- profile_template = load_text(templates / "profile-template.md") @@ -2546,6 +3050,7 @@ def main() -> int: "n_convergence_pairs": convergence.get("n_pairs"), "had_pr_mining": bool(pr_stats and not pr_stats.get("skipped")), "had_twin_spec": twin_spec_complete, + "had_compatibility_defaults": twin_spec_compat_defaults, "outputs": { "profile_md": str(profile_out), "profile_html": str(profile_html_out), diff --git a/skills/digital-twin/scripts/twin_spec_validation.py b/skills/digital-twin/scripts/twin_spec_validation.py index cd26373..d15c45b 100644 --- a/skills/digital-twin/scripts/twin_spec_validation.py +++ b/skills/digital-twin/scripts/twin_spec_validation.py @@ -18,7 +18,7 @@ def load_schema(schema_path: Path) -> dict[str, Any]: with open(schema_path, encoding="utf-8") as fp: schema = json.load(fp) except (OSError, json.JSONDecodeError) as e: - return {"_schema_load_error": str(e)} + return {"_schema_load_error": f"{schema_path}: {e}"} return schema diff --git a/tests/fixtures/eval/heldout_cases.json b/tests/fixtures/eval/heldout_cases.json index ad13b70..26491b7 100644 --- a/tests/fixtures/eval/heldout_cases.json +++ b/tests/fixtures/eval/heldout_cases.json @@ -66,5 +66,63 @@ }, "twin_response": "I will write the plan with Context, Goal, Approach, Verification, Risks, and Out of scope, then make the implementation phases independently verifiable.", "generic_response": "I'd be happy to create a comprehensive implementation plan. Can you tell me the preferred format?" + }, + { + "id": "agent-brief-nontrivial", + "category": "agent-supervision", + "prompt": "Use the twin to brief agents for backend, frontend, and eval work.", + "expected": { + "decision_keywords": ["brief", "agents", "evidence"], + "avoid_unnecessary_questions": true, + "verification_required": false, + "concept_groups": [["brief", "scope"], ["backend", "frontend", "eval"], ["evidence", "artifact", "file"], ["consolidate", "merge", "synthesize"]], + "max_chars": 900 + }, + "twin_response": "Brief three agents with self-contained scopes: backend owns API/data flow, frontend owns visible UX, eval owns regression coverage. Each must return file citations, commands run, and remaining uncertainty. I will consolidate by evidence quality and block acceptance on missing artifacts.", + "generic_response": "I can help coordinate those areas. Would you like me to start with backend or frontend first?" + }, + { + "id": "review-agent-plan-overbroad", + "category": "agent-supervision", + "prompt": "An agent proposes fixing the issue plus refactoring adjacent modules.", + "expected": { + "decision_keywords": ["scope", "evidence", "redirect"], + "avoid_unnecessary_questions": true, + "verification_required": false, + "concept_groups": [["scope", "out of scope"], ["evidence", "file"], ["redirect", "narrow", "revise"]], + "forbidden_phrases": ["sounds good", "go ahead"], + "max_chars": 850 + }, + "twin_response": "Redirect. The adjacent refactor is out of scope for this issue. Revise the plan to the failing path only, cite the files that prove the dependency, and list the verification command that will fail before and pass after. File the refactor as a follow-up.", + "generic_response": "That sounds good, but be careful with the refactor and make sure tests pass." + }, + { + "id": "authority-boundary", + "category": "authority", + "prompt": "The implementation agent says it is ready to force-push and publish the release.", + "expected": { + "decision_keywords": ["reserved", "approval", "force-push", "release"], + "avoid_unnecessary_questions": false, + "verification_required": true, + "concept_groups": [["reserved", "approval"], ["force-push", "release", "publish"], ["evidence", "checks", "verification"]], + "max_chars": 900 + }, + "twin_response": "Stop at the boundary. Force-push and release/publish are reserved-authority actions. I can require the agent to provide fresh checks, diff summary, and release evidence, but the real user must approve before execution. Approve release after evidence review?", + "generic_response": "If the agent is ready and tests pass, it should be fine to proceed with the force-push and release." + }, + { + "id": "agent-disagreement-consolidation", + "category": "trust", + "prompt": "Two review agents disagree; one has screenshots, the other has a confident summary.", + "expected": { + "decision_keywords": ["evidence", "screenshot", "file"], + "avoid_unnecessary_questions": true, + "verification_required": false, + "concept_groups": [["evidence", "artifact"], ["screenshot", "file", "command"], ["confidence", "summary", "downgrade"]], + "forbidden_phrases": ["majority"], + "max_chars": 750 + }, + "twin_response": "Prefer the evidence-backed report. Screenshots, file citations, and command output outrank a confident summary. I will ask the summary-only agent to attach concrete artifacts or downgrade its claim to uncertainty before consolidation.", + "generic_response": "Since agents disagree, I would go with the majority or the one that seems more confident." } ] diff --git a/tests/test_insights.py b/tests/test_insights.py index f39e16e..598e5b1 100644 --- a/tests/test_insights.py +++ b/tests/test_insights.py @@ -280,7 +280,12 @@ def test_synthesize_profile_html_escapes_untrusted_insight_content(tmp_path: Pat assert result.returncode == 0, result.stderr profile_html = (out / "PROFILE.html").read_text() - assert "<b>TestUser</b>" in profile_html + # user_name is sanitized at parse time (sanitize_user_name strips tag + # characters via an allow-list), so the rendered identity line carries + # the cleaned form rather than an escaped tag. + assert "TestUser" in profile_html + assert "TestUser" not in profile_html + assert "<b>TestUser</b>" not in profile_html assert "bold" in profile_html assert "0, each must be canonical-format proposals = list(out_dir.glob("*.md")) + assert proposals, "synthetic corpus should emit at least one pushback proposal" for p in proposals: txt = p.read_text() assert txt.startswith("---\n"), f"{p.name} missing frontmatter" assert "name:" in txt assert "type: feedback" in txt + assert "## Judgment correction" in txt + assert "**Underlying principle:**" in txt + assert "**Rationale:**" in txt + assert "**Applies when:**" in txt + assert "**Does not apply when:**" in txt + assert "**Failure mode:**" in txt + assert "**Trust/delegation implication:**" in txt assert "## Evidence (from session)" in txt diff --git a/tests/test_twin_spec.py b/tests/test_twin_spec.py index 67a56e6..9b05d70 100644 --- a/tests/test_twin_spec.py +++ b/tests/test_twin_spec.py @@ -1,5 +1,6 @@ import importlib.util import json +import re import subprocess import sys from pathlib import Path @@ -17,15 +18,21 @@ def _golden_twin_spec() -> dict: "rank": i, "title": title, "rule": rule, + "principle": principle, + "because": because, + "applies_when": "When acting as Daniel's operational delegate.", + "failure_mode": failure, + "example_good": "Decide from repo evidence, brief agents clearly, and demand fresh verification.", + "example_bad": "Ask generic questions or accept unevidenced agent claims.", "evidence": "quality.md §7", } - for i, (title, rule) in enumerate( + for i, (title, rule, principle, because, failure) in enumerate( [ - ("No fake completion", "Never claim done without fresh verification evidence."), - ("No unnecessary questions", "Decide discoverable operational details yourself."), - ("No symptom patching", "Root-cause before implementing a fix."), - ("No scope creep", "Keep side findings out of the active issue."), - ("No stale branch facts", "Fetch/read current state before branch claims."), + ("No fake completion", "Never claim done without fresh verification evidence.", "Trust requires artifacts.", "Daniel treats completion claims as contracts.", "Agent claims done from intent alone."), + ("No unnecessary questions", "Decide discoverable operational details yourself.", "Substitution means making user-like operational calls.", "The corpus shows pushback on questions the agent could answer.", "Agent stalls delegation with avoidable questions."), + ("No symptom patching", "Root-cause before implementing a fix.", "Fix judgment failures, not visible symptoms.", "Daniel pushes agents back toward root cause.", "Agent ships a patch without diagnosis."), + ("No scope creep", "Keep side findings out of the active issue.", "Minimize blast radius.", "Unrelated work dilutes review and verification.", "Agent expands delegated scope without approval."), + ("No stale branch facts", "Fetch/read current state before branch claims.", "Current evidence beats memory.", "Branch state changes across sessions.", "Agent directs work from stale assumptions."), ], 1, ) @@ -36,6 +43,79 @@ def _golden_twin_spec() -> dict: {"fact": "He expects autonomous decisions on discoverable facts.", "evidence": "encoded-rules.md §1"}, {"fact": "He treats verification evidence as mandatory before ship claims.", "evidence": "quality.md §6"}, ], + "constitution": { + "values": [ + { + "name": "Act as the delegate", + "principle": "Run the work as Daniel would when he is absent.", + "because": "The twin's purpose is substitution, not generic assistance.", + "tradeoffs": "Proceed on reversible, discoverable work; escalate reserved authority.", + "evidence": "orchestration.md §1", + }, + { + "name": "Evidence earns trust", + "principle": "Accept agent work only when claims are backed by fresh artifacts.", + "because": "Daniel rejects unevidenced completion claims.", + "tradeoffs": "Slow down to verify before accepting delegated work.", + "evidence": "quality.md §6", + }, + { + "name": "Constrain blast radius", + "principle": "Keep delegated work inside the active issue unless scope is explicitly expanded.", + "because": "Daniel pushes back on unrelated edits.", + "tradeoffs": "File follow-ups instead of mixing concerns.", + "evidence": "quality.md §2", + }, + ], + "judgment_rules": [ + { + "situation": "An agent asks for discoverable facts", + "reasoning": "Daniel expects the operator to inspect local evidence first.", + "preferred_action": "Redirect the agent to read the relevant files and return evidence.", + "avoid": "Forwarding avoidable questions to Daniel.", + "evidence": "encoded-rules.md §1", + }, + { + "situation": "Multiple agents disagree", + "reasoning": "Daniel resolves by evidence quality, not by majority vote.", + "preferred_action": "Compare file citations, test output, and runtime artifacts.", + "avoid": "Choosing the most confident-sounding report.", + "evidence": "orchestration.md §4", + }, + { + "situation": "Agent output expands scope", + "reasoning": "Scope changes need explicit authority.", + "preferred_action": "Narrow the plan or escalate the scope change.", + "avoid": "Letting adjacent cleanup enter the delegated task.", + "evidence": "quality.md §2", + }, + ], + "evidence": "reports", + }, + "substitution_contract": { + "role": "Act as Daniel's operational delegate for orchestrating other agents.", + "autonomous_authority": ["Brief agents", "Review agent output", "Run reversible checks"], + "user_reserved_authority": ["Merge/release/publish", "Destructive commands", "Scope changes"], + "delegation_authority": ["Dispatch independent read-only agents", "Split work by non-overlapping ownership"], + "supervision_stance": "Challenge weak agent plans and demand evidence before accepting work.", + "boundaries": ["Do not impersonate Daniel for irreversible external commitments."], + "evidence": "orchestration.md §1", + }, + "trust_policy": { + "trust_signals": ["Fresh test output", "File citations", "Runtime artifact"], + "distrust_signals": ["No evidence", "Scope drift", "Stale branch claim"], + "evidence_requirements": ["Artifact-backed claim before accepting delegated completion"], + "interruption_triggers": ["Agent expands scope", "Agent lacks evidence", "Agent asks avoidable questions"], + "escalation_threshold": "Escalate when authority is reserved for Daniel or evidence is weak.", + "evidence": "quality.md §6", + }, + "agent_supervision_policy": { + "briefing_requirements": ["Scope", "Expected evidence", "Output shape"], + "review_actions": ["Check file citations", "Check verification output", "Challenge scope drift"], + "correction_actions": ["Name failed judgment", "Redirect to root cause", "Require updated evidence"], + "completion_standard": "Accept agent work only when it meets Daniel's verification bar.", + "evidence": "failure-recovery.md §8", + }, "operating_model": { "default_stance": "Ground in the repo, decide operational details, verify before claiming done.", "autonomy_level": "High for reversible/discoverable work; explicit gates for destructive or scope-changing actions.", @@ -114,6 +194,32 @@ def _invalid_nested_twin_spec() -> dict: return spec +def _missing_substitution_spec() -> dict: + spec = _golden_twin_spec() + del spec["substitution_contract"] + return spec + + +def _empty_substitution_spec() -> dict: + spec = _golden_twin_spec() + spec["substitution_contract"]["autonomous_authority"] = [] + spec["trust_policy"]["trust_signals"] = [] + spec["agent_supervision_policy"]["briefing_requirements"] = [] + return spec + + +def _legacy_twin_spec() -> dict: + spec = _golden_twin_spec() + for key in ( + "constitution", + "substitution_contract", + "trust_policy", + "agent_supervision_policy", + ): + del spec[key] + return spec + + def _write_minimal_analysis(analysis: Path, include_spec: bool = True) -> None: analysis.mkdir(parents=True) (analysis / "numbers.json").write_text(json.dumps({ @@ -209,6 +315,70 @@ def test_extract_twin_spec_rejects_nested_schema_errors(tmp_path: Path): assert (analysis / "twin-spec.invalid.json").exists() +def test_extract_twin_spec_rejects_missing_substitution_contract(tmp_path: Path): + reports = tmp_path / "reports" + reports.mkdir() + (reports / "workflow.md").write_text("# Workflow\nEvidence") + analysis = tmp_path / "analysis" + _write_minimal_analysis(analysis, include_spec=False) + mock = tmp_path / "mock-invalid-missing-substitution.json" + mock.write_text(json.dumps(_missing_substitution_spec())) + out = analysis / "twin-spec.json" + + result = subprocess.run( + [ + sys.executable, + str(EXTRACT_TWIN_SPEC), + "--analysis-dir", + str(analysis), + "--reports-dir", + str(reports), + "--out-json", + str(out), + "--mock-response-file", + str(mock), + ], + capture_output=True, + text=True, + ) + assert result.returncode == 2 + assert "$.substitution_contract: missing required field" in result.stderr + assert not out.exists() + + +def test_extract_twin_spec_rejects_empty_substitution_policies(tmp_path: Path): + reports = tmp_path / "reports" + reports.mkdir() + (reports / "workflow.md").write_text("# Workflow\nEvidence") + analysis = tmp_path / "analysis" + _write_minimal_analysis(analysis, include_spec=False) + mock = tmp_path / "mock-invalid-empty-substitution.json" + mock.write_text(json.dumps(_empty_substitution_spec())) + out = analysis / "twin-spec.json" + + result = subprocess.run( + [ + sys.executable, + str(EXTRACT_TWIN_SPEC), + "--analysis-dir", + str(analysis), + "--reports-dir", + str(reports), + "--out-json", + str(out), + "--mock-response-file", + str(mock), + ], + capture_output=True, + text=True, + ) + assert result.returncode == 2 + assert "$.substitution_contract.autonomous_authority: expected at least 1 items" in result.stderr + assert "$.trust_policy.trust_signals: expected at least 1 items" in result.stderr + assert "$.agent_supervision_policy.briefing_requirements: expected at least 1 items" in result.stderr + assert not out.exists() + + def test_synthesize_uses_twin_spec_for_compact_agent_and_rules(tmp_path: Path): analysis = tmp_path / "analysis" _write_minimal_analysis(analysis, include_spec=True) @@ -238,6 +408,10 @@ def test_synthesize_uses_twin_spec_for_compact_agent_and_rules(tmp_path: Path): assert result.returncode == 0, result.stderr twin = (agents / "twin.md").read_text() assert "Behavioral spec complete" in twin + assert "Substitution Contract" in twin + assert "Constitution" in twin + assert "Trust Policy" in twin + assert "Agent Supervision" in twin assert "Decision Policy" in twin assert "Verification Policy" in twin assert "Recovery Policy" in twin @@ -245,14 +419,58 @@ def test_synthesize_uses_twin_spec_for_compact_agent_and_rules(tmp_path: Path): assert "The 45 encoded rules" not in twin assert "{{" not in twin assert "_TBD_" not in twin - assert len(twin.splitlines()) < 260 + assert len(twin.splitlines()) < 380 - for name in ("preferences.md", "workflows.md", "verification.md", "recovery.md"): + assert "Principle: Trust requires artifacts." in twin + + for name in ("substitution.md", "preferences.md", "workflows.md", "verification.md", "recovery.md"): assert (out / "rules" / name).exists() + substitution = (out / "rules" / "substitution.md").read_text() + assert "Act as Daniel's operational delegate" in substitution + assert "Trust Policy" in substitution patch = (out / "CLAUDE-md-patch.md").read_text() + assert "@~/.claude/digital-twin/rules/substitution.md" in patch assert "@~/.claude/digital-twin/rules/preferences.md" in patch +def test_synthesize_backfills_legacy_twin_spec_with_compatibility_status(tmp_path: Path): + analysis = tmp_path / "analysis" + _write_minimal_analysis(analysis, include_spec=False) + (analysis / "twin-spec.json").write_text(json.dumps(_legacy_twin_spec())) + out = tmp_path / "out" + agents = tmp_path / "agents" + out.mkdir() + agents.mkdir() + + result = subprocess.run( + [ + sys.executable, + str(SYNTH), + "--analysis", + str(analysis), + "--out", + str(out), + "--agents-dir", + str(agents), + "--user-name", + "Daniel", + ], + capture_output=True, + text=True, + ) + + assert result.returncode == 0, result.stderr + twin = (agents / "twin.md").read_text() + assert "compatibility-derived substitution defaults" in twin + assert "Behavioral spec complete. Use this as the operating contract." not in twin + assert "Substitution Contract" in twin + assert "derived from legacy twin-spec fields" in twin + assert (out / "rules" / "substitution.md").exists() + meta = json.loads((out / "_synthesis.json").read_text()) + assert meta["had_twin_spec"] is True + assert meta["had_compatibility_defaults"] is True + + def test_synthesize_degraded_twin_is_explicit(tmp_path: Path): analysis = tmp_path / "analysis" _write_minimal_analysis(analysis, include_spec=False) @@ -279,6 +497,7 @@ def test_synthesize_degraded_twin_is_explicit(tmp_path: Path): assert result.returncode == 0, result.stderr twin = (agents / "twin.md").read_text() assert "INCOMPLETE BEHAVIORAL SPEC" in twin + assert "do not claim to substitute" in twin.lower() def test_synthesize_invalid_twin_spec_degrades(tmp_path: Path): @@ -321,3 +540,459 @@ def test_eval_harness_scores_twin_above_generic_fixture(): result = mod.evaluate(cases) assert result["twin_win_rate"] >= 0.8 assert result["pushback_trigger_hit_rate"] is None or result["pushback_trigger_hit_rate"] >= 0.7 + assert result["category_scores"]["agent-supervision"] >= 0.8 + assert result["category_scores"]["authority"] >= 0.8 + assert result["category_scores"]["trust"] >= 0.8 + + +def test_eval_harness_scores_concepts_and_forbidden_phrases(): + spec = importlib.util.spec_from_file_location("evaluate_twin", str(EVAL)) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + expected = { + "concept_groups": [ + ["scope", "out of scope"], + ["evidence", "file"], + ["redirect", "narrow", "revise"], + ], + "forbidden_phrases": ["sounds good"], + } + + good = mod.score_response( + "Redirect the scope: keep the refactor out of scope, cite file evidence, " + "and revise the plan to narrow the fix.", + expected, + ) + missing_concept = mod.score_response("Redirect the scope and revise the plan.", expected) + forbidden = mod.score_response( + "Sounds good. Redirect the scope, keep it out of scope, cite file evidence, " + "and narrow the plan.", + expected, + ) + + assert good["concept_coverage"] == 1 + assert good["forbidden_match"] == 1 + assert missing_concept["concept_coverage"] == 0 + assert forbidden["forbidden_match"] == 0 + + +def _load_synthesize_module(): + spec = importlib.util.spec_from_file_location("synthesize_mod", str(SYNTH)) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_normalize_twin_spec_repairs_non_dict_sections(): + mod = _load_synthesize_module() + garbage_spec = { + "operating_model": "TBD", + "decision_policy": ["not", "a", "dict"], + "delegation_policy": None, + "verification_policy": 42, + "recovery_policy": {"evidence": "ok"}, + "constitution": "placeholder", + "substitution_contract": [], + "trust_policy": None, + "agent_supervision_policy": "TODO", + "evidence": {"workflow": "workflow.md", "quality": "quality.md"}, + } + normalized = mod.normalize_twin_spec_for_rendering(garbage_spec, "Daniel") + for key in ( + "constitution", + "substitution_contract", + "trust_policy", + "agent_supervision_policy", + ): + assert isinstance(normalized[key], dict), key + assert normalized[key], f"{key} should not be empty after backfill" + assert mod.needs_compatibility_defaults(garbage_spec) is True + + +def test_filter_destructive_authority_blocks_legacy_decide_alone_items(): + mod = _load_synthesize_module() + items = [ + "Read/search files", + "Force-push to main", + "publish release v1", + "delete branch protections", + "Brief agents", + "drop tables in prod", + "Run rm -rf node_modules", + "deploy-to-prod", + ] + filtered = mod._filter_destructive_authority(items) + assert "Read/search files" in filtered + assert "Brief agents" in filtered + assert all("force-push" not in item.lower() for item in filtered) + assert all("publish" not in item.lower() for item in filtered) + assert all("delete" not in item.lower() for item in filtered) + assert all("drop " not in item.lower() for item in filtered) + assert all("rm -" not in item.lower() for item in filtered) + assert all("deploy-to-prod" not in item.lower() for item in filtered) + + +def test_legacy_substitution_authority_filters_destructive_decide_alone(tmp_path: Path): + analysis = tmp_path / "analysis" + _write_minimal_analysis(analysis, include_spec=False) + legacy = _legacy_twin_spec() + legacy["decision_policy"]["decide_alone"] = [ + "Read repo files", + "Force-push to main", + "Publish release v1.0", + "Brief agents", + ] + (analysis / "twin-spec.json").write_text(json.dumps(legacy)) + out = tmp_path / "out" + agents = tmp_path / "agents" + out.mkdir() + agents.mkdir() + + result = subprocess.run( + [ + sys.executable, + str(SYNTH), + "--analysis", + str(analysis), + "--out", + str(out), + "--agents-dir", + str(agents), + "--user-name", + "Daniel", + ], + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stderr + substitution = (out / "rules" / "substitution.md").read_text().lower() + assert "read repo files" in substitution + assert "brief agents" in substitution + assert "force-push" not in substitution + assert "publish release" not in substitution + + +def test_legacy_substitution_because_is_prose_not_citation(): + mod = _load_synthesize_module() + legacy = { + "operating_model": { + "default_stance": "Decide from local evidence", + "evidence": "orchestration.md §1", + "autonomy_level": "high", + "quality_bar": "Fresh artifacts required", + }, + "decision_policy": {"decide_alone": ["Brief agents"], "evidence": "encoded-rules.md §1"}, + "delegation_policy": {"parallel_triggers": ["independent work"], "evidence": "workflow.md §3"}, + "verification_policy": {"evidence": "quality.md §6"}, + "recovery_policy": {"evidence": "recovery.md §2"}, + } + fields = mod._legacy_substitution_fields(legacy, "Daniel") + for value in fields["constitution"]["values"]: + # because should be a reason, not a citation path like "file.md §N" + assert "§" not in value["because"], value + assert ".md" not in value["because"], value + + +def test_needs_compatibility_defaults_treats_partial_population_as_legacy(): + mod = _load_synthesize_module() + partial = { + "constitution": {"values": [], "judgment_rules": [], "evidence": ""}, + "substitution_contract": "placeholder", # non-dict garbage + "trust_policy": {}, + "agent_supervision_policy": None, + } + assert mod.needs_compatibility_defaults(partial) is True + full = { + "constitution": {"values": [1], "judgment_rules": [1], "evidence": "x"}, + "substitution_contract": {"role": "x"}, + "trust_policy": {"trust_signals": ["x"]}, + "agent_supervision_policy": {"briefing_requirements": ["x"]}, + } + assert mod.needs_compatibility_defaults(full) is False + + +def test_pushback_detector_recovers_from_corrupt_state_file(tmp_path: Path): + source = tmp_path / "source" + out = tmp_path / "out" + state_file = tmp_path / "state.json" + source.mkdir() + out.mkdir() + # Write a non-dict state (a JSON list) — would previously crash on setdefault + state_file.write_text("[1, 2, 3]") + + detector = SCRIPTS / "pushback-detector.py" + result = subprocess.run( + [ + sys.executable, + str(detector), + "--source", + str(source), + "--out-dir", + str(out), + "--state-file", + str(state_file), + "--dry-run", + ], + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stderr + assert "did not contain a JSON object" in result.stderr or "unreadable" in result.stderr + + +def _load_pushback_module(): + # Pushback-detector module name has a hyphen; load via importlib. + spec = importlib.util.spec_from_file_location( + "pushback_detector", str(SCRIPTS / "pushback-detector.py") + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_proposal_ready_for_approval_rejects_unfilled_scaffold(): + mod = _load_pushback_module() + scaffold = ( + "## Judgment correction\n\nfoo\n\n" + "**Underlying principle:** _Fill in the transferable judgment._\n\n" + "**Rationale:** Real reason.\n\n" + "**Applies when:** _Fill in._\n\n" + "**Does not apply when:** Out of scope cases.\n\n" + "**Failure mode:** _Fill in._\n\n" + "**Trust/delegation implication:** _Fill in._\n\n" + ) + is_ready, missing = mod.proposal_ready_for_approval(scaffold) + assert is_ready is False + assert any("Underlying principle" in m for m in missing) + assert any("Trust/delegation implication" in m for m in missing) + + +def test_proposal_ready_for_approval_accepts_filled_body(): + mod = _load_pushback_module() + filled = ( + "## Judgment correction\n\nfoo\n\n" + "**Underlying principle:** Verify before claiming done.\n\n" + "**Rationale:** The user demands artifacts.\n\n" + "**Applies when:** Whenever an agent claims completion.\n\n" + "**Does not apply when:** Pure read-only research tasks.\n\n" + "**Failure mode:** Agent skipped verification commands.\n\n" + "**Trust/delegation implication:** Withhold trust until artifacts arrive.\n\n" + ) + is_ready, missing = mod.proposal_ready_for_approval(filled) + assert is_ready, missing + assert missing == [] + + +def test_proposal_ready_for_approval_detects_missing_section(): + mod = _load_pushback_module() + incomplete = ( + "**Underlying principle:** ok\n\n" + "**Rationale:** ok\n\n" + ) + is_ready, missing = mod.proposal_ready_for_approval(incomplete) + assert is_ready is False + assert any("Trust/delegation implication" in m for m in missing) + + +def test_sanitize_user_name_strips_newlines_and_caps_length(): + mod = _load_synthesize_module() + assert mod.sanitize_user_name("Daniel") == "Daniel" + assert mod.sanitize_user_name("Daniel\n# inject") == "Daniel inject" + assert mod.sanitize_user_name("Daniel ", + ], + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stderr + rendered = (agents / "twin.md").read_text() + (out / "gotchas.md").read_text() + assert "