Skip to content

feat(providers): add deterministic test provider for staging/CI (#459)#512

Merged
Cedarich merged 2 commits into
Pulsefy:mainfrom
kimanicode:issue/459-deterministic-test-provider
May 29, 2026
Merged

feat(providers): add deterministic test provider for staging/CI (#459)#512
Cedarich merged 2 commits into
Pulsefy:mainfrom
kimanicode:issue/459-deterministic-test-provider

Conversation

@kimanicode
Copy link
Copy Markdown
Contributor

Summary

Provides deterministic AI outputs for predictable integration testing across the entire platform by extending the existing TestProvider pattern to all key endpoints — OCR, proof-of-life, PII anonymization, and verification. All test providers are fixture-driven and activated purely via environment variable, requiring no code changes to enable in CI or staging.

Closes #459


Changes

Python AI Service (app/ai-service/)

New fixture files:

  • fixtures/ocr_responses.json — 6 deterministic OCR results (varied fields, quality levels, edge cases)
  • fixtures/proof_of_life_responses.json — 6 deterministic proof-of-life results (pass/fail, with/without burst frames)
  • fixtures/anonymize_responses.json — 6 deterministic PII scrubbing results (varying PII types and counts)

Extended TestProvider to additional services (all bypass system dependencies when TEST_PROVIDER_MODE=true):

  • services/ocr.py — returns fixture OCR data instead of calling Tesseract
  • services/pii_scrubber.py — returns fixture anonymization data instead of calling spaCy
  • proof_of_life.py — returns fixture POL data instead of calling OpenCV

New stability tests (tests/test_test_provider_stability.py — 17 tests):

  • Same input → same output across all endpoints
  • Output structure validation
  • Cross-endpoint fixture independence
  • TestProvider unit-level determinism (key generation, caching)

NestJS Backend (app/backend/)

Added VERIFICATION_MODE=test to verification.service.ts:

  • Returns fixture-driven deterministic results using SHA-256 hash of claim ID
  • 6 fixture responses covering low / medium / high risk levels
  • Same claim ID always produces the same fixture result

New stability tests (4 tests in verification.service.spec.ts):

  • Deterministic results across repeated calls
  • Valid fixture score ranges and risk levels
  • Same claim ID → same fixture across parallel calls

How to Enable

No code changes required — purely environment variable driven.

CI / Staging / Testnet:

TEST_PROVIDER_MODE=true
VERIFICATION_MODE=test

Checklist

  • Branch name follows issue/<number>-<slug>
  • Scope limited to files listed in the issue
  • 21 stability tests added across Python service and NestJS backend
  • Enabled via env vars — no code changes needed to activate
  • pnpm-lock.yaml / package-lock.json not modified

@Cedarich Cedarich merged commit c44d966 into Pulsefy:main May 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deterministic “Test Provider” for Staging/Testnet

2 participants