feat(verify): force tool use on temporal-dismissal-prone OpenAI/Gemini calls#11
Draft
aRealGem wants to merge 2 commits into
Draft
feat(verify): force tool use on temporal-dismissal-prone OpenAI/Gemini calls#11aRealGem wants to merge 2 commits into
aRealGem wants to merge 2 commits into
Conversation
PR #10 (Rev-2 reliability checkpoint) squash-merged 2026-05-04 as b06dddf; 3 weeks idle since. Moved the row from WIP to Done, added a 2026-05-23 session note in STATUS.md, and bumped the new "temporal-dismissal substance fix" row to WIP (bundles two of the three (NEW 2026-05-04) P0 rows; intra-family tie-break stays in Backlog). Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…i calls
OpenAI: add tool_choice="required" at every web_search call site
(build_batch_payload, build_multi_batch_payload, _call_with_search live
path). Per OpenAI Responses API docs: "With tool_choice: 'auto', search
is optional. Use tool_choice: 'required' [...] when search must run."
Gemini: add tool_config.function_calling_config.mode="ANY" at every call
site (build_batch_payload, CreateCachedContentConfig for the cache-hit
path, GenerateContentConfig for the non-cached call/call_multi fallbacks).
Cached and non-cached paths land the nudge in different spots because
tool_config on a per-request GenerateContentConfig with cached_content is
a hard Google API error (see test_gemini_cache.py contract).
SYNTHESIS_SYSTEM temporal preamble (build_temporal_preamble) gains two
new rules:
5. VERIFY, DON'T DISMISS — a claim's date being past training cutoff
is NOT grounds for "Unverifiable"; Unverifiable is reserved for
claims the web genuinely cannot resolve. Carries a concrete worked
example using today's runtime-injected date.
6. INVOKE SEARCH BEFORE DECIDING — returning "Unverifiable" without
having invoked the search tool on a checkable claim is a contract
violation, not a humble admission of uncertainty.
Tests: +12 (test_openai_tool_choice.py x3, test_gemini_tool_config.py x5,
test_temporal_context.py x4). 798 passing total.
Targets the temporal-dismissal failure mode that scored 0/4 on the first
live run of the 4-case regression set (run cbc335a1, 2026-05-01). Expected
3/4 pass post-fix (TrumpRx + Venezuela + Helicoide); Rubio still gated on
the deferred intra-family consensus tie-break P0.
Caveat: Gemini's function_calling_config.mode="ANY" is documented for
function-call tools, not the built-in google_search grounding tool. If
ANY is a no-op for grounding, the temporal preamble revisions are the
bigger lever. Plumbing the knob anyway keeps the adapter surfaces
symmetric and is harmless either way.
Refs: eval/sotu-2026/temporal-regressions-runbook.md;
PROJECT_BOARD.md P0 rows (NEW 2026-05-04).
Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Substance-track fix for two of the three temporal-dismissal failure modes observed in the 2026-05-23 regression rerun (0/4 pass).
tool_choice: "required"on batch payload, multi-batch payload, and the live_call_with_searchpath. Forces the model to invokeweb_searchinstead of skipping straight to a no-search "Unverifiable" verdict.tool_config.function_calling_config.mode = "ANY"on batch payload, cached-content create config, and the non-cachedcall/call_multipaths (cached-retrieve path intentionally left alone).src/truthbot/verify/context/temporal.py): added rules 5 ("VERIFY, DON'T DISMISS") and 6 ("INVOKE SEARCH BEFORE DECIDING") to counter the "training cutoff -> Unverifiable" reflex.New / extended unit tests pin the three payload knobs and the two new preamble rules:
tests/test_openai_tool_choice.py(new)tests/test_gemini_tool_config.py(new; includes a regression guard that the cached-retrieve path does NOT settool_config)tests/test_temporal_context.py(extended)Validation
uv run pytest -q-> 798 passed, 18 deselected (live tests, correctly excluded), 1 xfailed. Green locally.data/temporal-regressions-mini-transcript.txt; runbook ateval/sotu-2026/temporal-regressions-runbook.md. Expected outcome once run: 3/4 pass. The Rubio case is expected to remain failing -- it is gated on the deferred intra-family consensus tie-break follow-up, which is out of scope for this PR.What is deliberately NOT in this PR
Follow-ups
Made with Cursor