feat: test harness with LLM judge, replacing smartest framework

[yanekyuk]

Summary

• New protocol/src/lib/test-harness/ library that replaces the smartest framework with a leaner approach that feels like standard bun tests
• createTestHarness() factory wires real adapters (Drizzle DB, EmbedderAdapter, RedisCacheAdapter) against a test database with setup/reset/teardown lifecycle
• assertMatchesSchema(value, zodSchema) for deterministic Zod validation
• assertLLMEvaluate(value, config) for scored semantic criteria via LLM judge — supports per-criterion required flag, per-criterion min threshold, and overall minScore
• LLM judge uses google/gemini-2.5-flash via OpenRouter with structured output (Zod response format)
• Migrated 2 existing smartest test files (opportunity evaluator stress test, direct-connection graph test)
• Graceful no-op when OPENROUTER_API_KEY is missing (warns, doesn't fail)

New Features

• createTestHarness() — real adapter injection for integration tests
• assertMatchesSchema() — Zod schema validation assertion
• assertLLMEvaluate() — scored semantic criteria with LLM judge
• callJudge() — core LLM judge function with structured output

Refactors

• Migrated opportunity.evaluator.smartest.spec.ts → stress test block in opportunity.evaluator.spec.ts
• Migrated opportunity.graph.direct-connection.smartest.spec.ts → opportunity.graph.direct-connection.spec.ts

Documentation

• Updated .env.example with DATABASE_TEST_URL and TEST_JUDGE_MODEL (replacing SMARTEST_VERIFIER_MODEL/SMARTEST_GENERATOR_MODEL)

Deferred

• Smartest framework (lib/smartest/) stays until remaining 7 test files are migrated (follow-up)
• queue and graphs properties on TestHarness deferred until graph-level integration tests need them

Test plan

• Run bun test src/lib/test-harness/tests/judge.spec.ts — 2 tests (LLM scoring, missing API key)
• Run bun test src/lib/test-harness/tests/assertions.spec.ts — 4 tests (schema valid/invalid, LLM pass/fail)
• Run bun test src/lib/test-harness/tests/harness.spec.ts — 3 tests (db connection, embedder, reset)
• Run bun test src/lib/protocol/agents/tests/opportunity.evaluator.spec.ts — verify stress test block
• Run bun test src/lib/protocol/graphs/tests/opportunity.graph.direct-connection.spec.ts — verify migrated test
• Verify tsc --noEmit passes

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7373f34a-148e-4c55-9c11-b9e1c1e3c72f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}