image-to-image and image-to-video support

[tombeckenham]

Thank you @yiss for describing this so well

Problem

generateImage() and generateVideo() are currently centered around text-prompt inputs, but several providers and models support image-conditioned generation workflows.

Examples include:

• image-to-image generation
• prompt + reference image generation
• multi-reference image generation
• image-to-video generation
• video generation from a starting frame
• model-specific image editing / transformation workflows

Today there is no obvious provider-agnostic way to pass image inputs into generateImage() and generateVideo().

TanStack AI already has a clean multimodal abstraction for content parts (ImagePart with source.type: 'data' | 'url'). It would be great if media generation APIs reused that same shape instead of introducing provider-specific one-offs for image-conditioned generation.

Why this matters

Modern image and video models are increasingly multimodal. Generation is no longer only text-to-image or text-to-video.

A unified way to pass image inputs would make it much easier for adapters to support workflows like:

• image editing
• reference-guided generation
• image-to-video
• multi-image composition

Proposal

Add an optional inputs field to both generateImage() and generateVideo() that accepts reusable multimodal content parts, ideally existing ImagePart values.

This would provide a consistent, provider-agnostic way to pass image-conditioned inputs into media generation APIs.

Example API

generateImage()

import { generateImage, type ImagePart } from '@tanstack/ai'

const reference: ImagePart = {
  type: 'image',
  source: {
    type: 'url',
    value: 'https://example.com/reference.png',
  },
}

await generateImage({
  adapter: openaiImage('gpt-image-1.5'),
  prompt: 'Turn this into a cinematic product photo',
  inputs: [reference],
})

generateVideo()

import { generateVideo, type ImagePart } from '@tanstack/ai'

const startingFrame: ImagePart = {
  type: 'image',
  source: {
    type: 'data',
    value: base64Image,
    mimeType: 'image/png',
  },
}

await generateVideo({
  adapter: googleVideo('veo-3.1'),
  prompt: 'Animate this still into a slow cinematic push-in with subtle motion',
  inputs: [startingFrame],
})

Multiple reference images

import { generateImage, type ImagePart } from '@tanstack/ai'

const product: ImagePart = {
  type: 'image',
  source: {
    type: 'url',
    value: 'https://example.com/product.png',
  },
}

const style: ImagePart = {
  type: 'image',
  source: {
    type: 'url',
    value: 'https://example.com/style.png',
  },
}

await generateImage({
  adapter: geminiImage('nano-banana'),
  prompt: 'Generate a new image of the product using the style of the second reference',
  inputs: [product, style],
})

Expected behavior

• generateImage() and generateVideo() should both accept image-conditioned inputs through the same field name.
• The input format should ideally reuse existing TanStack AI multimodal primitives such as ImagePart.
• Adapters should map those inputs into the provider-native request shape.
• Unsupported combinations can be rejected by adapters at runtime or by adapter-specific validation.
• Providers that only support text prompts should continue to work unchanged.

Open design questions

• Should the field be named inputs, references, or something else?
• Should it accept only ImagePart[], or broader content parts for future extensibility?
• Should generateVideo() support multiple input images as well, or only one initially?

Summary

Request: add a unified, provider-agnostic way to pass image-conditioned inputs into both generateImage() and generateVideo(), ideally by reusing existing multimodal content-part types such as ImagePart.

Originally posted by @yiss in #481

[tombeckenham]

Note: scan through the entire fal, gemini, grok, and openai APIs for image and video generation to determine how many ways an image, video, or audio can be input. Here's the cases I can think of right now:

Image inputs:

• Editing an image - change elements of an existing image
• Character references - use this person in an image or video
• Element References - place this object into an image or video
• Location References - set this image or video in this location
• Starting Frame - set as the starting frame for this video
• Ending Frame - set as the ending frame for this video
• Generate 3d from image

Video Inputs

• Edit a video
• Generate foley / sfx for a video
• Generate music or dialog for video (although pretty uncommon)
• Generate 3d from video
• Merge the video / trim the video etc

Audio Inputs

• Lip sync dialog in video
• Time video generation to music
• Add music to video
• Add sfx / foley to video

[tombeckenham]

Design decisions (agreed) — implementation in progress

Following research across fal.ai, OpenAI, Gemini, Grok, and OpenRouter, here's the design we're going with:

Public API

await generateImage({
  adapter: openaiImage('gpt-image-1'),
  prompt: 'Turn this into a cinematic product photo',
  imageInputs: [productRef, styleRef],
})

await generateVideo({
  adapter: googleVideo('veo-3.1'),
  prompt: 'Slow cinematic push-in',
  imageInputs: [startFrame],
})

• Field names: imageInputs?: ImagePart[], videoInputs?: VideoPart[], audioInputs?: AudioPart[] on both generateImage() and generateVideo() option types.
  • Naming rationale: imageInputs reads unambiguously alongside result.images (output), and the modality-typed split admits non-image conditioning (video-to-video, audio for lipsync) without future field renames.
• Intent disambiguation via ImagePart.metadata.role:
  • 'reference' (default if omitted), 'mask', 'control', 'start_frame', 'end_frame', 'character'
  • Adapters use the role to route to provider-specific fields; unknown/missing roles fall back to positional.

Per-provider behavior in this PR

Provider	Image generation	Video generation
OpenAI	gpt-image-1 / gpt-image-1-mini → images.edit() (up to 16). dall-e-2 → images.edit() (1 only). dall-e-3 → throw. role: 'mask' → mask param.	sora-2 / sora-2-pro → input_reference (single image only).
Gemini	Native (gemini-*-flash-image, nano-banana) → inject into contents[].parts[] as inline_data/file_data. Imagen → throw (text-only).	Veo not yet adapted; deferred.
fal	1 input → image_url, >1 → image_urls; role: 'mask' → mask_url; 'reference' → reference_image_urls; 'control' → control_image_url. Per-endpoint mapping is interim — will be replaced by a dedicated fal schemas library.	1 image → image_url (start frame); role: 'end_frame' → end_image_url; 'start_frame' → start_image_url; 'reference' → reference_image_urls.
Grok	Throws not-supported (needs Imagine API rewrite — follow-up issue).	n/a
OpenRouter	Throws not-supported (needs multimodal injection — follow-up issue).	n/a
Anthropic	n/a (no image generation API).	n/a

Cross-cutting

• Source coercion: ImagePart.source.type === 'url' is fetched and converted to a SDK-uploadable Blob/Buffer for OpenAI; passed through as-is for Gemini/fal. 'data' (base64) → Buffer/data-URI.
• Per-model gating: adapters throw clearly when a model doesn't support imageInputs so users see a fast, actionable error.
• E2E coverage: image-to-image (gpt-image-1) and image-to-video (sora-2) added to testing/e2e/.
• Docs: docs/media/image-generation.md and docs/media/video-generation.md gain reference / mask / multi-input examples.
• SKILL: packages/typescript/ai/skills/ai-core/media-generation/SKILL.md updated with the new pattern.

Open follow-ups (separate PRs)

• Grok Imagine API adapter (image-to-image + image-to-video via xAI's native endpoint, not the OpenAI-compat shim).
• OpenRouter multimodal injection for image-conditioned models routed through chat completions.
• Gemini Veo video adapter.
• Replace fal's interim per-endpoint table with the dedicated fal schemas library once it lands.

[yiss]

Hi @tombeckenham thank you for refining the API proposal. I'm happy to contribute to this if possible. I'd be happy to open a PR

[tombeckenham]

Do you want to give it a shot? I started one but happy for you to take a look