Skip to content

Troubleshooting

Jump to bottom
gaafa edited this page Apr 9, 2026 · 1 revision

Troubleshooting

No eval files found

Error: No eval files found in 'tests/evals'

EvalCI only discovers files matching eval_*.py or *_eval.py. Rename your files:

test_rag.py    ❌  →  eval_rag.py    ✅
test_agent.py  ❌  →  eval_agent.py  ✅

TypeError: float() argument must be a string or a real number, not 'coroutine'

Cause: Bug in synapsekit < 1.5.2 — async @eval_case functions were not awaited correctly.

Fix: Pin to synapsekit 1.5.2 or later:

- uses: SynapseKit/evalci@v1
  with:
    synapsekit-version: "1.5.2"

PR comment not posted

  • Check that github-token is set (default ${{ github.token }} works for most repos)
  • The workflow must be triggered by a pull_request event — push events don't post comments
  • Check Actions permissions: repo Settings → Actions → General → Workflow permissions → Read and write

Import error for LLM provider

Error: ModuleNotFoundError: No module named 'openai'

Set the correct extras for your provider:

- uses: SynapseKit/evalci@v1
  with:
    extras: "openai"        # for OpenAI
    # extras: "anthropic"   # for Anthropic
    # extras: "openai,anthropic"  # for both

Action exits 1 but all cases pass

Check if fail-on-regression is set and you have a baseline snapshot that's being compared against. Either remove --fail-on-regression or update your baseline snapshot.

Still stuck?

Clone this wiki locally