Add manifest-driven layered artifact generation command#131
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Code Review
This pull request introduces a new script and associated tests for regenerating the layered admissibility artifact, along with a corresponding npm command. The feedback provided focuses on enhancing the robustness of path handling by anchoring file paths to the repository root in both the script and the test suite, ensuring consistent behavior across different execution environments.
| from src.validation.degradation_curve_generator import DegradationCurveGenerator | ||
|
|
||
| CURVE_ID = "coding_workflow_pr_review_curve_v1" | ||
| OUTPUT_PATH = Path("artifacts/layered_admissibility_results.json") |
There was a problem hiding this comment.
The OUTPUT_PATH is currently defined as a relative path. Since this script is intended to be a deterministic entrypoint and already calculates REPO_ROOT, it is more robust to anchor the output path to the repository root. This ensures the script behaves consistently regardless of the current working directory.
| OUTPUT_PATH = Path("artifacts/layered_admissibility_results.json") | |
| OUTPUT_PATH = REPO_ROOT / "artifacts" / "layered_admissibility_results.json" |
| """Regenerate layered admissibility artifact from manifest-ordered fixtures.""" | ||
|
|
||
| generator = DegradationCurveGenerator() | ||
| fixtures = generator.fixtures_for_layered_admissibility_curve() |
There was a problem hiding this comment.
The call to fixtures_for_layered_admissibility_curve() relies on a default relative path for the manifest file. To ensure portability and consistency with the REPO_ROOT logic, it's better to pass an absolute path anchored to the repository root.
| fixtures = generator.fixtures_for_layered_admissibility_curve() | |
| fixtures = generator.fixtures_for_layered_admissibility_curve(manifest_path=REPO_ROOT / "fixtures" / "manifest.json") |
| from scripts.generate_layered_admissibility_artifact import ( | ||
| CURVE_ID, | ||
| generate_layered_admissibility_artifact, | ||
| ) |
There was a problem hiding this comment.
Import REPO_ROOT from the script to allow anchoring paths in the test file, ensuring tests can be run from any directory.
| from scripts.generate_layered_admissibility_artifact import ( | |
| CURVE_ID, | |
| generate_layered_admissibility_artifact, | |
| ) | |
| from scripts.generate_layered_admissibility_artifact import ( | |
| CURVE_ID, | |
| REPO_ROOT, | |
| generate_layered_admissibility_artifact, | |
| ) |
| ) | ||
| from src.validation.degradation_curve_generator import DegradationCurveGenerator | ||
|
|
||
| ARTIFACT_PATH = Path("artifacts/layered_admissibility_results.json") |
There was a problem hiding this comment.
Anchor ARTIFACT_PATH to the repository root using the imported REPO_ROOT. This improves test robustness when executed from different working directories.
| ARTIFACT_PATH = Path("artifacts/layered_admissibility_results.json") | |
| ARTIFACT_PATH = REPO_ROOT / "artifacts" / "layered_admissibility_results.json" |
|
|
||
| def test_generated_artifact_has_no_new_top_level_fields() -> None: | ||
| generator = DegradationCurveGenerator() | ||
| curve = generator.generate(generator.fixtures_for_layered_admissibility_curve(), curve_id=CURVE_ID) |
There was a problem hiding this comment.
Pass the absolute manifest path to fixtures_for_layered_admissibility_curve to ensure the test can locate the manifest regardless of the current working directory.
| curve = generator.generate(generator.fixtures_for_layered_admissibility_curve(), curve_id=CURVE_ID) | |
| fixtures = generator.fixtures_for_layered_admissibility_curve(manifest_path=REPO_ROOT / "fixtures" / "manifest.json") | |
| curve = generator.generate(fixtures, curve_id=CURVE_ID) |
Motivation
artifacts/layered_admissibility_results.jsonfromfixtures/manifest.jsonusing the existing manifest-driven generator while preserving exact rational scoring and byte-stable output.Description
scripts/generate_layered_admissibility_artifact.pywhich importsDegradationCurveGenerator, callsfixtures_for_layered_admissibility_curve(), generates the curve withcurve_id = coding_workflow_pr_review_curve_v1, and writesartifacts/layered_admissibility_results.jsonvia the generator's deterministicwrite_jsonbehavior.tests/test_generate_layered_admissibility_artifact.pyverifying the generated JSON equals the committed artifact, no new top-level fields are introduced, and the moderate fixture retainsoverall_admissibility_score = 0.8333333333333334."generate:layered-admissibility": "python scripts/generate_layered_admissibility_artifact.py"topackage.jsonto provide a simple CLI shim.Summary: Added a deterministic, manifest-driven artifact regeneration script, focused tests, and a small npm entry.
Changed files:
scripts/generate_layered_admissibility_artifact.py,tests/test_generate_layered_admissibility_artifact.py,package.json.Determinism guarantees: the script uses manifest ordering from
fixtures/manifest.json, reusesDegradationCurveGenerator.write_jsonfor stable serialization, and introduces no timestamps, hostnames, nondeterministic IDs, or environment-derived fields.Testing
python scripts/generate_layered_admissibility_artifact.pywhich producedartifacts/layered_admissibility_results.jsonwith the committed content.pytest tests/test_admissibility_scorer.py -q(10 passed),pytest tests/test_degradation_curve_generator.py -q(13 passed),pytest tests/test_fixture_manifest.py -q(8 passed),pytest tests/test_svg_curve_renderer.py -q(7 passed), and the full test suite vianpm run check(all tests passed; full pytest run: 196 passed).Validation commands run:
python scripts/generate_layered_admissibility_artifact.py,pytest tests/test_admissibility_scorer.py -q,pytest tests/test_degradation_curve_generator.py -q,pytest tests/test_fixture_manifest.py -q,pytest tests/test_svg_curve_renderer.py -q,npm run check.Risks: Low — this is a thin, deterministic wrapper around existing generator logic and focused regression tests; no core scoring, fixture, or schema changes were made.
Next: Open PR for review and keep the change small and focused; run CI to confirm cloud validation.
Codex Task