Skip to content

Storyboard validation: 15 storyboards skip all tracks against training agent #1992

@bokelley

Description

@bokelley

Problem

When running storyboard validation against the public test agent, 15 of 21 storyboards show all tracks as "skip" — the scenarios are sent to testAllScenarios() but the orchestrator skips them because the agent doesn't expose the required tools.

Skipped storyboards and their missing scenarios:

Storyboard Scenarios sent Track(s) Why skipped
media_buy_seller full_sales_flow, create_media_buy, creative_lifecycle, creative_sync media_buy, creative Agent missing get_products, create_media_buy
media_buy_guaranteed_approval full_sales_flow, create_media_buy, creative_sync media_buy, creative Same
media_buy_non_guaranteed full_sales_flow, create_media_buy media_buy Same
media_buy_proposal_mode full_sales_flow, create_media_buy, creative_lifecycle, creative_sync media_buy, creative Same
media_buy_catalog_creative creative_lifecycle, full_sales_flow, create_media_buy media_buy, creative Same
media_buy_governance_escalation campaign_governance, full_sales_flow, campaign_governance_denied/conditions, create_media_buy media_buy, campaign_governance Same
creative_lifecycle creative_lifecycle, creative_flow creative Agent missing list_creative_formats, build_creative
creative_template creative_sync, creative_flow creative Agent missing sync_creatives, build_creative
creative_sales_agent creative_sync, creative_flow creative Same
content_standards governance_content_standards governance Agent missing list_content_standards
property_governance governance_property_lists, property_list_filters governance Agent missing create_property_list
si_session si_availability, si_session_lifecycle, si_handoff si Agent missing si_initiate_session
signal_marketplace signals_flow signals Agent missing get_signals
signal_owned signals_flow signals Same
social_platform sync_audiences, creative_sync audiences, creative Agent missing sync_audiences, sync_creatives

Root cause

The SCENARIO_REQUIREMENTS in @adcp/client gate scenarios based on which MCP tools the agent advertises. The public test agent apparently does not expose these tools when accessed via the public test token, or the tool names don't match what SCENARIO_REQUIREMENTS expects.

Investigation needed

  1. What tools does the test agent actually advertise? Compare against SCENARIO_REQUIREMENTS.
  2. Is this a test token permissions issue (the public token has limited tool access)?
  3. Or does the test agent genuinely not implement these tools?

The training agent task handlers implement 29 tasks (verified in task-handlers.ts), so the tools should be available. This is likely a discovery/advertisement issue rather than a missing implementation.

Context

Discovered during storyboard validation (#1985). Only 3 storyboards pass (campaign_governance_denied/conditions/delivery) and 2 fail (capability_discovery, creative_ad_server).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions