-
Notifications
You must be signed in to change notification settings - Fork 40
Warn when plan agent model refuses to tool-call #652
Description
Problem
Telemetry analysis of 2026-04-04 showed the `plan` agent has a 31% completion rate. Every abandoned plan session has the same signature: one generation, `finish_reason=stop`, zero tool calls.
Root cause: some models (observed: `ollama-cloud/qwen3-coder-next` with 5/5 failures, occasionally `openai/gpt-5.4`) respond to the plan agent's system prompt with text instead of tool calls. The user sees a text answer, no exploration happening, and aborts the session.
Representative telemetry burst (machine `450185a5`, 2026-04-04):
- 5 plan sessions with `qwen3-coder-next` → all `stop`, all abandoned in 10-38s
- 6th session switches to `minimax-m2.7` → immediately tool-calls, completes in 97s
Proposed Fix
When the plan agent finishes a step with `finish_reason=stop` AND no tool calls have ever been made in the session, show a one-shot warning:
- `log.warn` with session/model/provider context
- Emit `plan_no_tool_generation` telemetry event (per-model abandonment tracking)
- Inject a synthetic text part visible in the TUI recommending the user try a different model via `/model`
Synthetic text parts are filtered from LLM context replay, so this doesn't pollute future turns.
Acceptance Criteria
- Warning fires exactly once per session
- Only fires for `agent=plan`, `finish_reason=stop`, `sessionToolCallsMade=0`
- Other finish_reasons (`tool-calls`, `length`, `error`) don't trigger
- Other agents (`builder`, `analyst`, `explore`, `general`) don't trigger
- New telemetry event type `plan_no_tool_generation` with model_id, provider_id, tokens_output