Skip to content

VLM support for FoundationModels protocol#97

Open
carinapeng wants to merge 4 commits into
apple:mainfrom
carinapeng:carina/fm-vlm-protocol
Open

VLM support for FoundationModels protocol#97
carinapeng wants to merge 4 commits into
apple:mainfrom
carinapeng:carina/fm-vlm-protocol

Conversation

@carinapeng

Copy link
Copy Markdown
Contributor

Purpose

Adds CoreAIVisionLanguageModel, a FoundationModels LanguageModel adapter for VLM bundles. This lets callers drive a vision-language model through a standard LanguageModelSession and pass images via the FoundationModels attachment API, the same surface used for first-party multimodal models.

Changes

  • New swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift:
    • CoreAIVisionLanguageModel: LanguageModel — loads a VLM bundle folder, declares the .vision capability, builds the backing engine.
    • CoreAIVLMExecutor: LanguageModelExecutor — pulls the image attachment + text from the Transcript, runs the vision encoder + decode via the existing CoreAISequentialVLMEngine, and streams tokens into the FM channel.
  • Thin adapter: reuses the existing engine (no export or runner changes).
  • Streaming (session.streamResponse { … }) works the same way. An image attachment is required; a text-only prompt throws unsupportedTranscriptContent.

API usage

import CoreAILanguageModels
import FoundationModels

let model = try await CoreAIVisionLanguageModel(resourcesAt: bundleURL)
let session = LanguageModelSession(model: model)

let response = try await session.respond(
    options: GenerationOptions(maximumResponseTokens: 256)
) {
    Attachment(cgImage)          // or Attachment(imageURL:) / CIImage / CVPixelBuffer
    "What is in this image?"
}
print(response.content)

Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
@carinapeng carinapeng requested a review from kevchengcodes July 13, 2026 18:44
Comment thread swift/Sources/CoreAILanguageModels/VLM/CoreAIVisionLanguageModel.swift Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants