feat: BDD E2E contract tests for service behavioral spec

[onmete]

Summary

• Adds a pytest-bdd test suite under tests/e2e/ that verifies the service's behavioral contract across all 6 providers
• Gherkin .feature files map 1:1 to the behavioral specs in .ai/spec/ — the scenario is both the requirement and the proof
• Extracts container lifecycle into a shared scripts/start-containers.sh used by both make eval and make e2e

What's in the suite

Feature	Verifies spec	Scenarios
schema_compliance.feature	query-api.md	/analyze, /execute, /verify schema conformance + response wrapping
chat_sse.feature	chat-api.md	SSE event ordering, conversation continuity, tool call fields
skill_invocation.feature	provider-contract.md	Skill discovery, tool execution, token verification

8 BDD tests total, parametrized across all 6 providers when containers are running.

Design decisions for team discussion

1. BDD format — follows the ASDLC verification layer convention (Gherkin Given/When/Then). Scenarios are human-readable and usable as acceptance criteria directly.

2. Separate from evals — tests/e2e/ is standalone with no imports from evals/. The two suites test different things:

   • evals/ = model quality (non-deterministic, expensive)
   • tests/e2e/ = service contract (structural, cheap)

3. Cheap model defaults — tests/e2e/config.env sets haiku/flash/4.1-mini as defaults. Contract tests validate structure, not quality — no need for expensive models.

4. Shared container lifecycle — scripts/start-containers.sh replaces the container management from evals/run.sh. Both make eval and make e2e use it.

How to run

make e2e                              # all providers
make e2e E2E_ARGS="-k claude"         # single provider
ANTHROPIC_MODEL=claude-opus-4-6 make e2e  # override model

Test plan

• Team agrees on BDD approach for E2E verification
• Run make e2e against at least one provider to validate
• Verify make eval still works with the refactored container script
• Review feature file scenarios for completeness

Made with Cursor

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.