Problem
After the storyboard migration (#424), comply() runs ALL storyboards that match a track. For media_buy, that's 5 storyboards with 27 total steps. Each step is an MCP call. The test agent rate-limits after ~20 calls, causing cascading failures across remaining storyboards.
The old comply ran ~10 hand-written scenarios per track, each making 2-5 MCP calls total. The storyboard approach is much more thorough but exceeds rate limits.
Current behavior
adcp comply test-mcp → mostly failures from rate limiting, despite media_buy_seller passing 9/9 when run standalone.
Options
-
One storyboard per track — for each track, designate a "primary" storyboard. comply() only runs that one. Others are available for adcp storyboard run but not included in comply.
-
Storyboard selection in ComplyOptions — comply(url, { storyboards: ['media_buy_seller', 'creative_template'] }) — let callers pick which storyboards to run instead of running all per track.
-
Rate-limit backoff — add exponential backoff in the runner when an MCP call fails with rate limiting. Slower but complete.
-
Deduplicate discovery — many storyboards start with get_products and list_creative_formats. Share discovered context across storyboards within a track so only the first one does discovery.
Option 4 is probably the right long-term fix. Option 1 is the quickest path to a working comply.
Related
Problem
After the storyboard migration (#424),
comply()runs ALL storyboards that match a track. Formedia_buy, that's 5 storyboards with 27 total steps. Each step is an MCP call. The test agent rate-limits after ~20 calls, causing cascading failures across remaining storyboards.The old comply ran ~10 hand-written scenarios per track, each making 2-5 MCP calls total. The storyboard approach is much more thorough but exceeds rate limits.
Current behavior
adcp comply test-mcp→ mostly failures from rate limiting, despitemedia_buy_sellerpassing 9/9 when run standalone.Options
One storyboard per track — for each track, designate a "primary" storyboard. comply() only runs that one. Others are available for
adcp storyboard runbut not included in comply.Storyboard selection in ComplyOptions —
comply(url, { storyboards: ['media_buy_seller', 'creative_template'] })— let callers pick which storyboards to run instead of running all per track.Rate-limit backoff — add exponential backoff in the runner when an MCP call fails with rate limiting. Slower but complete.
Deduplicate discovery — many storyboards start with
get_productsandlist_creative_formats. Share discovered context across storyboards within a track so only the first one does discovery.Option 4 is probably the right long-term fix. Option 1 is the quickest path to a working comply.
Related