rc11/rc12 #508 fix over-fires: 11-message Telegram finals on research/audit runs (plan body + post-approval text both bloated + Layer E concatenates)

[nathanschram]

Symptom

Staging @hetz_lba1_bot running 0.35.3rc12. Plan-mode research/audit completions now produce final Telegram messages of 25k-42k characters — split into 8-12 Telegram messages. User report (Nathan, 2026-05-12): "I had a summary from Claude Code yesterday which was 11 Telegram messages long!! What I really want back is to have Claude Code provide summaries like we have here in command line — summaries of plans (not the entire plan), summaries of recommendations and/or findings and/or next steps (where relevant)."

The rc11 fix for #508 has the right intent but over-corrects: instead of the original 584-char "Plan approved" repro, we now ship the entire plan body + a paraphrased restatement + the structured ## Summary block, all concatenated.

Evidence from staging logs (journalctl --user -u untether.service --since '48 hours ago')

chat	date	answer_len	approximate Telegram messages
aushistory (-5189982707)	2026-05-11 02:37	14,267	~4
aushistory	2026-05-11 03:06	28,195	~8
aushistory	2026-05-11 03:34	35,463	~10
aushistory	2026-05-11 03:59	42,634	~12 ← user's repro
scout (-5243261989)	2026-05-12 03:06	26,176	~8
scout	2026-05-12 03:32	27,558	~8

Pre-fix comparison (rc10, 2026-05-10 04:16, scout): 584 chars — "Plan approved — research is complete..."

Telegram MCP search_messages for the literal 📋 Plan (approved): returned hits on every recent plan-mode completion in both chats — confirming Layer E IS the load-bearing over-firer (not preamble alone).

Root cause — three stacked over-shoots

1. Preamble A1 (src/untether/runner_bridge.py:293-304) tells Claude: "For research/audit tasks where no further work is expected after approval, expand the bullets into a substantive summary." — plan body now 2,000-5,000 chars instead of 3-5 bullets.

2. Preamble A2 (same paragraph) instructs: "your next assistant message ... MUST repeat the substantive findings or decisions" — Claude paraphrases the plan, producing another 500-2,000 chars.

3. Layer E (src/untether/runners/claude.py:_prepend_exitplanmode_plan) prepends the plan body with the substring-skip rule body in final_answer. Because A2 told Claude to paraphrase rather than literal-copy, the substring check fails on every well-behaved run and Layer E concatenates the full plan body in front of the paraphrased post-approval text — doubling the content.

Fix (rc13)

Replace verbose-driving preamble clauses with CLI-style brevity; replace the broken substring skip with a length gate; cap the plan-body re-emit when it does fire.

A1 — plan body shape

When calling ExitPlanMode, your plan parameter MUST be a concise 3-5 bullet summary of findings, decisions, or proposed changes — never just a file path. Keep it short: the plan is shown to the user for approval, not as the final deliverable.

A2 — post-approval text shape

After ExitPlanMode is approved, your next assistant message — which becomes the user's final Telegram message — MUST contain a brief CLI-style summary: 3-7 bullets or 1-2 short paragraphs covering key findings, recommendations, decisions, and next steps. Aim for ~500-1500 characters. Do not re-paste the full plan content — the user has already seen it during approval. Brevity is the goal.

A3 — ## Summary ### Plan/Document Created block

[Path AND a 3-5 bullet headline summary — the user has already seen the plan during approval, so this is a pointer + headline, not a re-paste of the full content]

Layer E (_prepend_exitplanmode_plan)

• Replace body in final_answer substring check with len(final_answer or "") < 600 length gate. The substring check remains as a secondary skip (cheap; covers exact-copy edge cases).
• When prepending, cap plan body to first 1500 chars + …\n\n(plan truncated — shown in full during approval) to prevent runaway when the gate does fire.

Tests

• tests/test_preamble.py: update A1/A2/A3 assertions to match new wording; add coverage for substring-skip, length-gate, and body-cap branches of _prepend_exitplanmode_plan.
• tests/test_claude_runner.py: existing test_translate_result_prepends_exitplanmode_plan_into_answer covered the prepend path — extend to confirm a longer (>600 char) post-approval answer does NOT trigger the prepend, and that body-cap fires for >1500-char plans.

Verification

• Integration test via @untether_dev_bot on a Claude plan-mode chat with a research-task prompt ("audit X and tell me Y"). Expect:
  • Final message ≤ 1-2 Telegram messages (~600-3500 chars)
  • 📋 Plan (approved): does NOT appear when A2 produces a real summary
  • Original final Telegram message becomes too short for research/investigation tasks under plan mode #508 fall-back (584-char acknowledgement) still triggers Layer E if Claude produces nothing post-approval

Target release

v0.35.3rc13. Ships on dev from fix/v0.35.3rc13-plan-summary-overfire.

References

• final Telegram message becomes too short for research/investigation tasks under plan mode #508 — original "final message too short" fix landed in rc11 (commits 24b6684 + 51c52c8)
• bug: runner.current_stream singleton race leaks data across concurrent Claude chats (rc11 #508 demonstrates; other read sites share defect) #510 / rc12 — moved the prepend to per-stream path (race fix; unaffected by this tuning fix)
• User report: 2026-05-12 conversation referencing 11-message summary

[nathanschram]

rc13 fix landed on fix/v0.35.3rc13-plan-summary-overfire

Status: PR #516 open against dev. Tests green (2652 passed, 2 skipped, 82.38% coverage). ruff format + check clean. Live integration test PASSED ✅ on @untether_dev_bot (chat -5284581592) — both primed and unprimed prompts.

Implementation

Layer A (preamble) — runner_bridge.py _DEFAULT_PREAMBLE:

• A1: "concise 3–5 bullet summary; the plan is shown for approval, not as the final deliverable" (dropped rc11's "expand into substantive summary" license)
• A2: "brief CLI-style summary: 3–7 bullets or 1–2 short paragraphs, ~500–1500 chars; do NOT re-paste the full plan content" (dropped rc11's "MUST repeat substantive findings")
• A3: "Path AND a 3–5 bullet headline summary — the user has already seen the plan during approval, so this is a pointer + headline, not a re-paste of the full content"

Layer E (_prepend_exitplanmode_plan in runners/claude.py):

• Substring check (body in final_answer) replaced with length gate len(final_answer) < 600. Substring check stays as a cheap secondary skip.
• Plan body capped at 1500 chars + …\n\n(plan truncated — shown in full during approval) when Layer E fires.

Tests (7 new/updated in tests/test_preamble.py, 2 new in tests/test_claude_runner.py)

• Regression-locks the rc11 verbosity-driving phrases ("expand the bullets into a substantive summary", "MUST repeat the substantive findings or decisions", "do not require the user to open the file") out of _DEFAULT_PREAMBLE.
• Confirms A1/A2/A3 wording present.
• _prepend_exitplanmode_plan length-gate, body-cap, substring-skip, no-plan, empty/None branches.
• test_translate_result_skips_prepend_when_answer_substantive — runner-level proof that a 600+ char post-approval result doesn't trigger Layer E.
• test_translate_result_caps_long_plan_body_when_prepending — runner-level proof that 3500-char plan bodies get capped to 1500 + truncation marker.

Live verification on @untether_dev_bot (chat -5284581592)

Primed test (with "keep it short" hint):

• runner.completed action_count=5 answer_len=882
• Telegram message ≈ 1 chunk; clean 3-bullet CLI summary; no 📋 Plan (approved): literal.

Unprimed test (default research-task prompt):

• runner.completed action_count=4 answer_len=1019
• Preamble does the job on its own — 1019 chars, well within target.
• Final message:

  - Engines (6): Claude Code, Codex, OpenCode, Pi, Gemini CLI, AMP — runners in src/untether/runners/, registered via pyproject.toml entry points.
  - runner_bridge.py: glue between engine runners and TelegramPresenter — injects the agent preamble, owns per-run ProgressEdits and RunContext threading, runs auto-continue (suppressed on signal deaths), and prepends ExitPlanMode plan bodies to the final answer.
  - Recent v0.35.3 rc11/rc12 fixes: #508 plan-mode final-message regression · #507 dead-ScheduleWakeup post-result idle shortcut · ...
  
  Summary
  
  Completed
  - Read-only lookup answered from CLAUDE.md + CHANGELOG.md.
  - Plan written and approved at ~/.claude/plans/...
  
  Next Steps
  - None — read-only Q&A.
  

Compare to rc12 (same chat, prior session)

Yesterday's rc12 final on the same chat: ~4000+ chars with 📋 Plan (approved): prepend + full plan body re-paste. rc13 brings this back to ~1000 chars with no prepend.

Commits

• 6040ab3 fix: rc11/rc12 #508 fix over-fires: 11-message Telegram finals on research/audit runs (plan body + post-approval text both bloated + Layer E concatenates) #515 — rc13 plan-summary over-fire (CLI-style brevity restored)

PR #516 targets dev. Once merged, will publish 0.35.3rc13 to TestPyPI; staging upgrade via scripts/staging.sh install 0.35.3rc13.

[nathanschram]

Shipped to staging

• PR fix: #515 — rc13 plan-summary over-fire (CLI-style brevity restored) #516 squash-merged to dev as commit 1b192f1
• Dev CI: all required gates green (pytest 3.12/3.13/3.14, ruff format+check, ty, build, lockfile, install-test, bandit, CodeQL, release-validation, docs). pip-audit fails on pre-existing urllib3 CVE-2026-44431 / CVE-2026-44432 — unrelated to this PR; needs a separate Dependabot bump.
• TestPyPI publish: untether-0.35.3rc13 wheel + sdist + attestations uploaded successfully.
• scripts/staging.sh install 0.35.3rc13 installed via pipx; systemctl --user restart untether brought the service up cleanly on rc13.
• scripts/healthcheck.sh --version 0.35.3rc13 → 5/5 passed (service active, no restarts, version matches, no ERROR-level logs in last 60s, Bot API getMe 200).
• Staging cron scheduler (13 crons) + webhook server (1 webhook) restarted cleanly; startup banner delivered to admin chat 8351408485.

Awaiting real-world verification

Will close after Nathan's next research/audit run on @hetz_lba1_bot produces a CLI-style summary in the 500–1500 char target range (≤ 2 Telegram messages instead of the 8–12 we were seeing). The two dev-bot tests (882 chars primed, 1019 chars unprimed) prove the preamble + length gate are doing the work; the only remaining unknown is whether real research-task prompts in the wild (longer context, larger plan files) behave the same — staging will tell us within a day.

If the next real run still produces a 5k+ char final, re-open this issue and we'll consider:

• Adding a length-gate test against the post-approval result that fires a claude.plan_summary.over_threshold WARN structlog when an OK plan-mode run produces > N chars — so future regressions self-surface in untether-issue-watcher without waiting for user reports.
• Tightening the A1/A2 preamble wording further (e.g. explicit "no headers, no tables" constraints for research-mode finals).
• An opt-in [bridge] final_answer_max_chars config knob with a hard truncate + "…(truncated; full output in ~/.claude/plans/<name>.md)" suffix.