test: validate emitted result JSON against result.schema.json#127
Merged
Conversation
Add tests/test_result_schema.py with 15 tests enforcing the contract between HarnessResult.to_dict() and schemas/result.schema.json. - 7 positive tests covering every (mode, result) combination the harness can produce: dry_run/not_run, trace/pass, trace/fail, live/pass, trace/error, plus assertion-without-evidence and empty-assertions-list edge cases. - 8 negative tests confirming the schema actually rejects bad shapes (invalid mode/result enums, extra top-level/assertion/trace properties, missing required fields). Without these the positive tests alone would still pass if someone accidentally loosened additionalProperties or dropped an enum. The fixture validator uses Draft202012Validator (matching the schema's declared draft) and calls check_schema() on load to catch schema syntax errors. Adds jsonschema>=4 to dev dependencies. CI already runs the dev extras so this works in CI without further changes. Closes #87
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds `tests/test_result_schema.py` enforcing the contract between `HarnessResult.to_dict()` and `schemas/result.schema.json`.
7 positive tests cover every (mode × result) combination the harness can produce: `dry_run/not_run`, `trace/pass`, `trace/fail`, `live/pass`, `trace/error`, plus `assertion-without-evidence` and `empty-assertions-list` edge cases.
8 negative tests confirm the schema actually rejects bad shapes — invalid `mode`/`result` enums, extra top-level / assertion / trace properties, missing required fields. Without these, the positive tests alone would still pass if someone accidentally loosened `additionalProperties` or dropped an enum.
The validator uses `Draft202012Validator` matching the schema's declared draft, and calls `check_schema()` at fixture load so schema syntax errors fail loudly.
Why this matters for v0.1.0
The result JSON shape is part of the public contract (downstream CI gates parse it). Without these tests, schema and dataclass can drift silently — particularly easy when adding a new field to `HarnessResult` but forgetting the schema, or vice versa.
Dependencies
Adds `jsonschema>=4` to `[project.optional-dependencies].dev`. CI already installs `[dev]`, so no workflow change needed.
Test plan
Closes #87