Skip to content

fix: gate prompt caching by provider, not model name only#22

Open
sumleo wants to merge 1 commit into
ResearAI:mainfrom
sumleo:fix/provider-aware-prompt-caching
Open

fix: gate prompt caching by provider, not model name only#22
sumleo wants to merge 1 commit into
ResearAI:mainfrom
sumleo:fix/provider-aware-prompt-caching

Conversation

@sumleo

@sumleo sumleo commented Jun 17, 2026

Copy link
Copy Markdown

What

Make prompt caching provider-aware so it actually works for the documented anthropic and openrouter configurations, not just Bedrock.

Why

Prompt caching is currently wired up for Bedrock only, even though the README documents AI_PROVIDER=anthropic and AI_PROVIDER=openrouter as supported setups. Two issues combine so that caching silently no-ops outside Bedrock:

  1. frontend/app/api/chat/route.ts (cache breakpoints, previously ~L317/L337/L347) — the cache markers were hardcoded to the Bedrock namespace providerOptions.bedrock.cachePoint, and applied whenever supportsPromptCaching(modelId) returned true. With @ai-sdk/anthropic, the adapter expects providerOptions.anthropic.cacheControl = { type: "ephemeral" }; the bedrock namespace is just ignored, so no cache breakpoints are ever sent and nothing gets cached.

  2. frontend/lib/ai-providers.ts (supportsPromptCaching, previously ~L683) — gating was on the model-name substring only (claude/anthropic), with no provider dimension. So a Claude model on a provider that can't honor a Bedrock cache marker still reported caching as enabled.

Fix

  • supportsPromptCaching(modelId, provider?) now takes the resolved provider and only enables caching for providers whose AI SDK adapter understands a Claude cache marker (bedrock, anthropic, openrouter). The provider argument is optional and falls back to the old model-name-only behaviour, so existing callers keep working.
  • getAIModel now returns the resolved provider (it was already computed internally), so route.ts can pass it through.
  • New getCacheBreakpointProviderOptions(provider) emits the correct marker shape per provider:
    • bedrock{ bedrock: { cachePoint: { type: "default" } } } (unchanged)
    • anthropic / openrouter (and other Claude-capable adapters) → { anthropic: { cacheControl: { type: "ephemeral" } } }
  • route.ts uses that helper for all three cache breakpoints (last assistant message + the two system breakpoints) instead of the hardcoded Bedrock object.

The OpenRouter adapter reads anthropic.cacheControl from providerOptions and maps it to its own cache_control field, so the Anthropic marker shape covers both anthropic and openrouter.

Testing

The repo has no test runner (Biome only). npx biome ci on the two changed files is clean. Note: npm run check reports pre-existing errors/warnings across other files that are already present on main (identical counts before and after this change); this PR introduces none.

Related

Open PR #20 ("custom provider protocol support") touches the Python autofigure/* code and some autofigure frontend components, but not app/api/chat/route.ts or lib/ai-providers.ts, so there's no overlap with this fix.

@sumleo

sumleo commented Jun 18, 2026

Copy link
Copy Markdown
Author

Hi @ResearAI, gentle nudge on this when you have a moment. It's a small, self-contained prompt-caching fix, and I'm happy to rebase or tweak anything if that would make review easier. Thanks for the project and your time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant