Skip to content

Add manifest-driven layered artifact generation command#131

Merged
ProfRandom92 merged 1 commit into
mainfrom
codex/add-deterministic-artifact-regeneration-command
May 19, 2026
Merged

Add manifest-driven layered artifact generation command#131
ProfRandom92 merged 1 commit into
mainfrom
codex/add-deterministic-artifact-regeneration-command

Conversation

@ProfRandom92
Copy link
Copy Markdown
Owner

Motivation

  • Provide a single deterministic entrypoint to regenerate artifacts/layered_admissibility_results.json from fixtures/manifest.json using the existing manifest-driven generator while preserving exact rational scoring and byte-stable output.

Description

  • Added scripts/generate_layered_admissibility_artifact.py which imports DegradationCurveGenerator, calls fixtures_for_layered_admissibility_curve(), generates the curve with curve_id = coding_workflow_pr_review_curve_v1, and writes artifacts/layered_admissibility_results.json via the generator's deterministic write_json behavior.
  • Added tests/test_generate_layered_admissibility_artifact.py verifying the generated JSON equals the committed artifact, no new top-level fields are introduced, and the moderate fixture retains overall_admissibility_score = 0.8333333333333334.
  • Added npm script "generate:layered-admissibility": "python scripts/generate_layered_admissibility_artifact.py" to package.json to provide a simple CLI shim.

Summary: Added a deterministic, manifest-driven artifact regeneration script, focused tests, and a small npm entry.

Changed files: scripts/generate_layered_admissibility_artifact.py, tests/test_generate_layered_admissibility_artifact.py, package.json.

Determinism guarantees: the script uses manifest ordering from fixtures/manifest.json, reuses DegradationCurveGenerator.write_json for stable serialization, and introduces no timestamps, hostnames, nondeterministic IDs, or environment-derived fields.

Testing

  • Ran python scripts/generate_layered_admissibility_artifact.py which produced artifacts/layered_admissibility_results.json with the committed content.
  • Ran unit tests used for validation: pytest tests/test_admissibility_scorer.py -q (10 passed), pytest tests/test_degradation_curve_generator.py -q (13 passed), pytest tests/test_fixture_manifest.py -q (8 passed), pytest tests/test_svg_curve_renderer.py -q (7 passed), and the full test suite via npm run check (all tests passed; full pytest run: 196 passed).

Validation commands run: python scripts/generate_layered_admissibility_artifact.py, pytest tests/test_admissibility_scorer.py -q, pytest tests/test_degradation_curve_generator.py -q, pytest tests/test_fixture_manifest.py -q, pytest tests/test_svg_curve_renderer.py -q, npm run check.

Risks: Low — this is a thin, deterministic wrapper around existing generator logic and focused regression tests; no core scoring, fixture, or schema changes were made.

Next: Open PR for review and keep the change small and focused; run CI to confirm cloud validation.


Codex Task

@vercel
Copy link
Copy Markdown

vercel Bot commented May 19, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
comptextv7 Ready Ready Preview, Comment May 19, 2026 2:57pm

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new script and associated tests for regenerating the layered admissibility artifact, along with a corresponding npm command. The feedback provided focuses on enhancing the robustness of path handling by anchoring file paths to the repository root in both the script and the test suite, ensuring consistent behavior across different execution environments.

from src.validation.degradation_curve_generator import DegradationCurveGenerator

CURVE_ID = "coding_workflow_pr_review_curve_v1"
OUTPUT_PATH = Path("artifacts/layered_admissibility_results.json")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The OUTPUT_PATH is currently defined as a relative path. Since this script is intended to be a deterministic entrypoint and already calculates REPO_ROOT, it is more robust to anchor the output path to the repository root. This ensures the script behaves consistently regardless of the current working directory.

Suggested change
OUTPUT_PATH = Path("artifacts/layered_admissibility_results.json")
OUTPUT_PATH = REPO_ROOT / "artifacts" / "layered_admissibility_results.json"

"""Regenerate layered admissibility artifact from manifest-ordered fixtures."""

generator = DegradationCurveGenerator()
fixtures = generator.fixtures_for_layered_admissibility_curve()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The call to fixtures_for_layered_admissibility_curve() relies on a default relative path for the manifest file. To ensure portability and consistency with the REPO_ROOT logic, it's better to pass an absolute path anchored to the repository root.

Suggested change
fixtures = generator.fixtures_for_layered_admissibility_curve()
fixtures = generator.fixtures_for_layered_admissibility_curve(manifest_path=REPO_ROOT / "fixtures" / "manifest.json")

Comment on lines +6 to +9
from scripts.generate_layered_admissibility_artifact import (
CURVE_ID,
generate_layered_admissibility_artifact,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Import REPO_ROOT from the script to allow anchoring paths in the test file, ensuring tests can be run from any directory.

Suggested change
from scripts.generate_layered_admissibility_artifact import (
CURVE_ID,
generate_layered_admissibility_artifact,
)
from scripts.generate_layered_admissibility_artifact import (
CURVE_ID,
REPO_ROOT,
generate_layered_admissibility_artifact,
)

)
from src.validation.degradation_curve_generator import DegradationCurveGenerator

ARTIFACT_PATH = Path("artifacts/layered_admissibility_results.json")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Anchor ARTIFACT_PATH to the repository root using the imported REPO_ROOT. This improves test robustness when executed from different working directories.

Suggested change
ARTIFACT_PATH = Path("artifacts/layered_admissibility_results.json")
ARTIFACT_PATH = REPO_ROOT / "artifacts" / "layered_admissibility_results.json"


def test_generated_artifact_has_no_new_top_level_fields() -> None:
generator = DegradationCurveGenerator()
curve = generator.generate(generator.fixtures_for_layered_admissibility_curve(), curve_id=CURVE_ID)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pass the absolute manifest path to fixtures_for_layered_admissibility_curve to ensure the test can locate the manifest regardless of the current working directory.

Suggested change
curve = generator.generate(generator.fixtures_for_layered_admissibility_curve(), curve_id=CURVE_ID)
fixtures = generator.fixtures_for_layered_admissibility_curve(manifest_path=REPO_ROOT / "fixtures" / "manifest.json")
curve = generator.generate(fixtures, curve_id=CURVE_ID)

@ProfRandom92 ProfRandom92 merged commit ea997d4 into main May 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant