Skip to content

Preserve image content blocks through tool-result conversion #30

@Anarchid

Description

@Anarchid

Context

When an MCPL/MCP tool returns image content blocks (e.g. a Zulip
attachment fetch or a screenshot tool), framework.ts collapses the
result to a JSON string before it ever reaches Membrane, so vision-
capable providers see a literal "{\"type\":\"image\",...}" text
chunk instead of an image.

Where

src/framework.ts, toMembraneToolResult — previously:

content = JSON.stringify(afResult.data);

This runs regardless of whether afResult.data is an MCP content
array carrying an image block.

Fix

Already implemented on Anarchid/agent-framework#tool-result-image-passthrough.

Detect MCP tool-result content arrays containing at least one image
block and convert them to Membrane's native ToolResultContentBlock[]
(text + base64 image). Everything else falls through to JSON, so
scalars/objects/text-only results keep their existing behavior.

Test should convert AF tool results to Membrane format when resuming stream still passes (all 19 framework tests green).

Dependency

Needs the matching Membrane change (antra-tess/membrane#15 — branch Anarchid/membrane#tool-result-image-passthrough) to actually reach the model — otherwise the adapter still JSON-stringifies the array at the wire.

Upstream context

Discovered while plumbing image attachments end-to-end for the
zulip-mcp + connectome-host triumvirate (clerk/miner/reviewer
pipeline). zulip-mcp side: Anarchid/zulip_mcp#attachment-fetch-images.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions