Skip to content

Add Bedrock Converse client with --provider flag and worker subproces…#135

Open
Pidem wants to merge 2 commits into
MedARC-AI:mainfrom
Pidem:Bedrock-eval
Open

Add Bedrock Converse client with --provider flag and worker subproces…#135
Pidem wants to merge 2 commits into
MedARC-AI:mainfrom
Pidem:Bedrock-eval

Conversation

@Pidem

@Pidem Pidem commented Jun 22, 2026

Copy link
Copy Markdown

Adds native AWS Bedrock support via the Converse API. No proxy or external server required — just AWS credentials.

Changes

  • medarc_verifiers/bedrock/client.py — BedrockConverseClient implementing the verifiers Client interface using boto3's Converse API. Supports text, multi-turn, and tool-use messages. Runs the
    synchronous boto3 call via run_in_executor to avoid blocking the async event loop.
  • medarc_verifiers/bedrock/init.py — Monkeypatch that registers bedrock_converse as a valid ClientType in verifiers. Rebuilds RunRolloutRequest/RunGroupRequest Pydantic models so worker
    subprocesses also accept the new client type.
  • medarc_verifiers/init.py — Calls register() at import time so spawned env worker processes inherit the patched types.
  • medarc_verifiers/cli/_single_run.py — Adds --provider flag to single-run mode (was previously only available in bench mode).
  • medarc_verifiers/cli/verifiers_adapter.py — Adds "bedrock" to PROVIDER_CONFIGS with client_type: "bedrock_converse".
  • configs/medmarks-endpoints-bedrock.toml — Endpoint aliases for Nova Pro/Lite/Micro and Claude models on Bedrock.
  • docs/bedrock.md — Usage documentation.
  • pyproject.toml — bedrock dependency group (boto3>=1.35.0).

Usage

uv sync --group bedrock

AWS_PROFILE=my-profile uv run medarc-eval medqa
-m nova-pro
--endpoints-path configs/medmarks-endpoints-bedrock.toml
--provider bedrock
-n 25

Authentication uses the standard boto3 credential chain (AWS_PROFILE, AWS_REGION, ~/.aws/config, SSO, instance roles).

Tested

  • MedQA with us.amazon.nova-pro-v1:0 — 5 examples, 3 rollouts each, reward=0.8.

Note

This is a monkeypatch bridge until verifiers#1522 (PrimeIntellect-ai/verifiers#1522) is merged upstream, at which point medarc_verifiers/bedrock/ can be removed.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants