Eval Recipes

Evaluate AI agents with benchmarking harnesses and online evaluation recipes.

Eval Recipes is a library dedicated to make it easier to keep up with the state-of-the-art in evaluating AI agents. It currently has two main components: a benchmarking harness for evaluating CLI agents (GitHub Copilot CLI, Claude Code, etc) on real-world tasks via containers and an online evaluation framework for LLM chat assistants. The common thread between these components is the concept of recipes which are a mix of code and LLM calls to achieve a desired tradeoff between flexibility and quality.

Installation

# Benchmarking requires certain prerequisites, see the full documentation for more details.
# With uv (add to project dependencies, pinned to a release tag)
uv add "eval-recipes @ git+https://github.com/microsoft/eval-recipes@v0.0.35"

# With pip
pip install "git+https://github.com/microsoft/eval-recipes@v0.0.35"

Warning

This library is very early and everything is subject to change. Consider pinning the dependency to a release.

Benchmarking

Eval Recipes provides a benchmarking harness for evaluating AI agents on real-world tasks in isolated Docker containers. It supports score based, comparison based, and third party benchmarks. We include tasks ranging from creating CLI applications to automations. Agents are automatically scored based on deterministic and semantic tests using a specialized auditing agent. Additional features include agent continuation (automatically providing follow-up prompts when needed), multi-trial evaluation for consistency measurement, and reporting with HTML dashboards.

Usage

1. Create agent definition(s). Examples are provided in data/agents.
2. Create task definition(s). Examples are provided in data/tasks.
3. Create a benchmark configuration. Examples are provided in data/benchmarks.

import asyncio
from pathlib import Path

from eval_recipes.benchmarking.loaders import load_agents, load_benchmark, load_tasks
from eval_recipes.benchmarking.pipelines.score_pipeline import ScorePipeline

agents = load_agents(Path("data/agents"))
tasks = load_tasks(Path("data/tasks"))
benchmark = load_benchmark(Path("data/benchmarks/full_benchmark.yaml"))

pipeline = ScorePipeline(
    benchmark=benchmark.score_benchmark,
    agents=agents,
    tasks=tasks,
    output_dir=Path(".benchmark_results"),
)
asyncio.run(pipeline.run())

See docs/BENCHMARKING.md for full details, including installation prerequisites.

---

Online Evaluations

Get Started Quick!

1. View notebooks directly on GitHub

Located in demos/.

2. Run interactive notebooks with marimo

Run demo notebooks (the .py files located at demos/) with marimo. Follow the installation section below if you do not have uv installed or environment variables configured.

uv run marimo edit demos/1_evaluate.py
# Select Y to run in a sandboxed venv

High Level API

The primary way of interacting with the package is the high-level API which takes in a list of messages (defined by OpenAI's responses API) and a list of custom tool definitions (built-in tools are not supported).

Each evaluation will output if it is deemed applicable to your input, an overall score from 0 to 100, and additional metadata specific to that evaluation.

Currently there are several built-in evaluations: claim_verification, tool_usage, guidance, and preference_adherence. For more details on how these evaluations work, check the Low Level API section below. Each evaluation can be additionally configured, such as selecting the LLM used. The full configurations are defined in schemas.py.

evaluate will return a list of EvaluationOutput instances corresponding to each evaluation.

Tip

All of the code examples in this readme can be pasted into a .py file and run as is!

import asyncio
from openai.types.chat.chat_completion_tool_param import ChatCompletionToolParam
from openai.types.responses import EasyInputMessageParam, ResponseInputParam
from eval_recipes.evaluate import evaluate
from eval_recipes.evaluations.check_criteria.check_criteria_evaluator import CheckCriteriaEvaluatorConfig
from eval_recipes.schemas import BaseEvaluatorConfig

async def main() -> None:
    messages: ResponseInputParam = [
        EasyInputMessageParam(
            role="system", content="You are a helpful assistant with search and document editing capabilities."
        ),
        EasyInputMessageParam(
            role="user",
            content="What material has the best elasticity for sports equipment? Please keep your response concise.",
        ),
        EasyInputMessageParam(
            role="assistant",
            content="Polyurethane elastomers offer excellent elasticity with 85% energy return and high durability.",
        ),
    ]

    tools: list[ChatCompletionToolParam] = [
        ChatCompletionToolParam(
            type="function",
            function={
                "name": "search",
                "description": "Search for information",
                "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
            },
        ),
    ]
    config_preference_adherence = BaseEvaluatorConfig(model="gpt-5-mini")  # Sample config
    check_criteria = CheckCriteriaEvaluatorConfig(criteria=["Your response should be at least one paragraph long."])
    result = await evaluate(
        messages=messages,
        tools=tools,
        evaluations=["check_criteria", "claim_verification", "guidance", "preference_adherence", "tool_usage"],
        evaluation_configs={"preference_adherence": config_preference_adherence, "check_criteria": check_criteria},
        max_concurrency=1,
    )
    print(result)

asyncio.run(main())

Custom Evaluations

You can create custom evaluators by implementing a class that follows the EvaluatorProtocol. This allows you to extend the evaluation framework with domain-specific metrics tailored to your needs.

Custom evaluators must implement:

1. An __init__ method that accepts an optional BaseEvaluatorConfig parameter. If a config is not provided, you must initialize a default.
2. An async evaluate method that takes messages and tools as input and returns an EvaluationOutput

Here is an example of a custom evaluator that scores based on the length of the assistant's response being used in conjunction with the preference_adherence evaluator:

import asyncio
from openai.types.chat.chat_completion_tool_param import ChatCompletionToolParam
from openai.types.responses import EasyInputMessageParam, ResponseInputParam
from eval_recipes.evaluate import evaluate
from eval_recipes.schemas import BaseEvaluatorConfig, EvaluationOutput

class ResponseLengthEvaluator:
    """Custom evaluator that scores based on response brevity."""
    def __init__(self, config: BaseEvaluatorConfig | None = None) -> None:
        self.config = config or BaseEvaluatorConfig()

    async def evaluate(self, messages: ResponseInputParam, tools: list[ChatCompletionToolParam]) -> EvaluationOutput:
        total_length = 0
        for message in reversed(messages):  # Only look at the last assistant message
            if ("role" in message and message["role"] == "assistant") and message.get("content"):
                total_length += len(str(message["content"]))
                break

        score = max(0, 100 - int(total_length // 25))  # Decrease score as length increases
        return EvaluationOutput(eval_name="response_length", applicable=True, score=score, metadata={})

async def main() -> None:
    messages: ResponseInputParam = [
        EasyInputMessageParam(
            role="user",
            content="What material has the best elasticity for sports equipment? Please keep your response concise.",
        ),
        EasyInputMessageParam(
            role="assistant",
            content="Polyurethane elastomers offer excellent elasticity with 85% energy return and high durability.",
        ),
    ]
    result = await evaluate(
        messages=messages,
        tools=[],
        evaluations=[ResponseLengthEvaluator, "preference_adherence"],
        evaluation_configs={"ResponseLengthEvaluator": BaseEvaluatorConfig(model="gpt-5-mini")},
        max_concurrency=1,
    )
    print(result)

asyncio.run(main())

Low Level API

LOW_LEVEL_API.md

Development Installation

Prerequisites

• uv
• prek

Setup

Create uv virtual environment and install dependencies:

uv sync --frozen --all-extras --all-groups

This project uses prek for git hooks. See .pre-commit-config.yaml for the full configuration.

prek install

Setup environment variables. This library requires either OpenAI or Azure OpenAI to be configured. You must set the correct environment variables in the .env file. The semantic_test evaluator additionally requires ANTHROPIC_API_KEY to be set, as it uses the Claude Agent SDK. Check utils.py create_client to troubleshoot any configuration issues.

cp .env.sample .env
# Configure API keys in .env

To update dependencies (updates the lock file):

uv sync --all-extras --all-groups

Run formatting, linting, type checking, and tests in one command:

uv run ruff format && uv run ruff check --fix && uv run ty check

Run tests:

uv run pytest

Run all pre-commit hooks manually:

prek run --all-files

Other

• Generating Jupyter Notebooks
• To re-create the Manim animation:
  • make install-all to install manim. See the docs if you have issues on a Linux-based system. Note this will also require ffmpeg to be installed.
  • uv run manim scripts/create_animation.py EvalRecipesAnimation -qh && ffmpeg -y -i media/videos/create_animation/1080p60/EvalRecipesAnimation.mp4 -vf "fps=30,scale=1920:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 demos/data/EvalRecipesAnimation.gif
• Validating Evaluations:
  • This script will run evaluations against a small "goldset" (see data/goldset) where we have inputs to evaluate with labels of what the scores should be (defined in data/goldset/labels.yaml).

Attributions

The built-in claim_verification evaluation is based on these two papers: Claimify and VeriTrail. This is not an official implementation of either and please cite the original papers if you use this evaluation in your work.