Quickstart

Quickstart

Get EvalCI running in under 5 minutes.

→ Full docs: https://synapsekit.github.io/synapsekit-docs/docs/evalci/quickstart

---

Step 1 — Write an eval case

Create a file named eval_*.py in your test directory:

# tests/evals/eval_rag.py
from synapsekit import eval_case
from myapp.pipeline import rag_pipeline

@eval_case(min_score=0.80, max_cost_usd=0.01, max_latency_ms=3000)
async def eval_rag_relevancy():
    result = await rag_pipeline.ask("What is SynapseKit?")
    score = await result.score_relevancy(
        reference="SynapseKit is a Python framework for building LLM applications."
    )
    return {"score": score, "cost_usd": result.cost_usd, "latency_ms": result.latency_ms}

---

Step 2 — Add the workflow

# .github/workflows/eval.yml
name: EvalCI

on:
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: SynapseKit/evalci@v1
        with:
          path: tests/evals
          threshold: "0.80"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

---

Step 3 — Add your API key

1. Go to your repo → Settings → Secrets and variables → Actions
2. Click New repository secret
3. Name: OPENAI_API_KEY, Value: your key
4. Click Add secret

---

Step 4 — Open a PR

Push and open a pull request. EvalCI posts a results comment and passes/fails the check automatically.

---

Local testing

pip install synapsekit[openai]
synapsekit test tests/evals/ --threshold 0.80