Skip to content

Backport multimodal tool outputs and LiteLLM content sanitation#24

Merged
bitifirefly merged 2 commits intodevelopfrom
backport/upstream-20260322-final-batch3
Mar 22, 2026
Merged

Backport multimodal tool outputs and LiteLLM content sanitation#24
bitifirefly merged 2 commits intodevelopfrom
backport/upstream-20260322-final-batch3

Conversation

@bitifirefly
Copy link
Copy Markdown

Summary

  • backport multimodal tool-output flow so read_file and web_fetch can return native image content blocks
  • add shared image helpers (detect_image_mime, build_image_content_blocks) and use them across tools
  • enrich system/subagent prompts to explicitly allow visual-resource reading via tool outputs
  • keep tool interfaces compatible with structured results (Tool.execute / ToolRegistry.execute -> Any)
  • strip internal _meta keys from multimodal content blocks before LiteLLM requests to avoid provider schema incompatibilities

Testing

  • ruff check opencane/utils/helpers.py opencane/agent/tools/filesystem.py opencane/agent/tools/web.py opencane/agent/context.py opencane/agent/subagent.py opencane/agent/tools/base.py opencane/agent/tools/registry.py opencane/providers/litellm_provider.py tests/test_filesystem_tools_workspace.py tests/test_web_fetch_security.py tests/test_litellm_message_sanitize.py
  • pytest -q tests/test_filesystem_tools_workspace.py tests/test_web_fetch_security.py tests/test_litellm_message_sanitize.py
  • pytest -q (431 passed)

@bitifirefly bitifirefly merged commit 458c6c4 into develop Mar 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant