Question Regarding Tool Exposure Strategy in APEX-Agents Leaderboard Evaluation

[TheRoadQaQ]

I have a quick question regarding your APEX-Agents leaderboard setup using archipelago.

I noticed the use of:

• toolbelt_list_tools
• toolbelt_inspect_tool
• toolbelt_add_tool
• toolbelt_remove_tool

Could you share the reasoning behind this progressive tool exposure design?

From our perspective, revealing new tools by injecting updated tool definitions into the context may invalidate the KV cache (since tool schemas are placed at the beginning of the prompt), which could impact performance.

Was this trade-off intentional, or were alternative approaches considered?

For example, would it be feasible to only expose tool names initially, and return detailed specifications dynamically via tool call responses instead?

Thanks in advance for your insights!

[thisisvk45]

Not a maintainer, but I have been reading the codebase and the APEX-Agents paper recently and wanted to add a couple of observations that might be useful while you wait for an official response.

The KV-cache invalidation concern is real, but the harness already accepts cache invalidation as a baseline cost. Appendix D.1 of the paper notes that context summarization triggers at 70 percent utilization and retains only the last 10 messages. That summarization step is itself a full prefix rewrite, so the toolbelt mutations sit on top of an architecture that does not optimize for cache reuse to begin with. From that lens, progressive tool exposure may have been considered a marginal additional cost rather than a primary cache concern.

The likely upside of the current design is context-bloat control rather than cache behavior. APEX-Agents worlds expose 63 tools by default and up to 250 in the IB worlds with EDGAR plus equities plus fixed income. Inlining all of those schemas at trajectory start would consume a substantial chunk of context window before the agent does any work, which becomes a bigger problem at long horizons given the 250-step cap. The paper's failure analysis in Section 5.2 shows that successful trajectories use fewer steps and tools than failing ones, so the implicit goal seems to be keeping the context lean enough for the agent to plan over long horizons.

Your alternative of returning detailed specs only via tool-call responses is interesting and worth exploring as an ablation. It would preserve the prefix and shift the cost from the prompt to tool-call output tokens, which most providers price lower. The trade-off would be one extra round-trip per tool discovery, which the current design folds into the planning step. Curious to hear the maintainers' take on whether this trade-off was measured or assumed.