Skip to content

Add MedFact-Bench zero-shot evaluation environment#137

Open
KouateMuhamed wants to merge 1 commit into
MedARC-AI:mainfrom
KouateMuhamed:dataset/medfact-bench
Open

Add MedFact-Bench zero-shot evaluation environment#137
KouateMuhamed wants to merge 1 commit into
MedARC-AI:mainfrom
KouateMuhamed:dataset/medfact-bench

Conversation

@KouateMuhamed

@KouateMuhamed KouateMuhamed commented Jul 11, 2026

Copy link
Copy Markdown

Motivation

MedFact-Bench (Jin et al., Med-V1) is a 14,274-example benchmark for medical claim verification and evidence attribution across five component datasets: SciFact, HealthVer, MedAESQA, PubMedQA-Fact, and BioASQ-Fact. It fills a gap in MedMarks, which currently has no claim-versus-evidence verification environment. It is also directly useful for downstream MedARC fact-verification work, including AMFV, where verdict accuracy serves as a training and evaluation signal.

Summary

  • Adds the medfact_bench evaluation environment with pinned Hugging Face and local Parquet loading.
  • Implements the Med-V1 five-point score parser, three-way verdict mapping, and invalid-prediction handling.
  • Adds benchmark-specific raw-results reporting, including per-component accuracy, parsing diagnostics, coverage, class metrics, and paper-comparable macro-accuracy.
  • Adds MedFact-Bench to the smoke and verified benchmark configurations, with protocol and reference-result documentation.
  • Completes local end-to-end smoke validation with Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct.

Validation

uv run pytest tests/test_environments/test_medfact_bench.py
uv run pytest tests/test_environments/test_medfact_bench_reporting.py
uv run ruff check environments/medfact_bench tests/test_environments
uv run ruff format --check environments/medfact_bench tests/test_environments
uv run medarc-eval bench --config configs/medmarks-smoke.toml --dry-run
uv build --project environments/medfact_bench

Also verified:

  • Ruff check and format check pass.
  • medarc-eval bench --config configs/medmarks-smoke.toml --dry-run includes medfact_bench.

@KouateMuhamed KouateMuhamed marked this pull request as ready for review July 11, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant