Skip to content

Add deterministic incident response page triage fixture family#133

Merged
ProfRandom92 merged 1 commit into
mainfrom
codex/add-incident-response-page-triage-fixture-family
May 19, 2026
Merged

Add deterministic incident response page triage fixture family#133
ProfRandom92 merged 1 commit into
mainfrom
codex/add-incident-response-page-triage-fixture-family

Conversation

@ProfRandom92
Copy link
Copy Markdown
Owner

Summary:

  • Add a second deterministic fixture family incident_response_page_triage with four levels (baseline, mild, moderate, severe) and deterministic manifest registration to demonstrate admissibility degradation beyond the coding-workflow family.

Changed files:

  • fixtures/manifest.json (registered four new fixtures in deterministic order)
  • fixtures/incident_response_page_triage_v1/** (baseline bundle: original/reconstructed traces, state, dependency graph, original/contracts, expected/*)
  • fixtures/incident_response_page_triage_mild_v1/** (mild bundle)
  • fixtures/incident_response_page_triage_moderate_v1/** (moderate bundle)
  • fixtures/incident_response_page_triage_degraded_v1/** (severe bundle)
  • tests/test_fixture_manifest.py (updated EXPECTED_FIXTURE_ORDER to include new fixture ids)
  • tests/test_incident_response_fixture_contract_bundle.py (new targeted validator tests for baseline and severe fixtures)

Testing:

  • Ran pytest tests/test_fixture_manifest.py -q and it passed (8 passed).
  • Ran pytest tests/test_contract_validator.py -q and it passed (10 passed).
  • Ran pytest tests/test_fixture_contract_bundle.py -q and pytest tests/test_negative_fixture_contract_bundle.py -q and both passed.
  • Ran pytest tests/test_incident_response_fixture_contract_bundle.py -q and it passed (2 tests).
  • Ran full test suite pytest -q and observed all tests passing (199 passed).
  • Ran npm run check and it completed successfully.

Risks:

  • The mild level is implemented as a deterministic weak degradation that yields one contract failure (RECOVERY_PATH_INVALID) because current validators do not express non-contract drift; this is an intentional, conservative choice to stay within existing validator semantics.
  • Large fixture additions increase repository size and require keeping manifest ordering updated for future fixture additions.

Next:

  • Keep changes small and, if desired, add further deterministic families following the same contract-bundle pattern in separate PRs.
  • Optionally regenerate layered artifacts (e.g., artifacts/layered_admissibility_results.json) in a follow-up PR only if multi-family artifact inclusion is explicitly requested.

Codex Task

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new family of deterministic fixtures for incident response triage, ranging from baseline to severe degradation levels. Each fixture includes dependency graphs, state information, and operational/relational contracts. Feedback highlights that the 'moderate' and 'mild' fixtures are currently redundant and should be differentiated to provide a meaningful progression. Additionally, the new test suite should be expanded to cover all degradation levels and refactored to use the fixture metadata for validation instead of assuming all contracts fail in the severe case.

Comment thread fixtures/manifest.json
Comment on lines +113 to +129
"fixture_id": "incident_response_page_triage_moderate_v1",
"fixture_version": "1.0.0",
"category": "incident_response",
"family": "incident_response_page_triage",
"degradation_level": "moderate",
"path": "fixtures/incident_response_page_triage_moderate_v1",
"expected_admissible": false,
"contracts": [
"alert_ack_before_mitigation",
"no_orphan_mitigation_steps",
"rollback_reachable",
"root_cause_links_incident"
],
"expected_failure_labels": [
"RECOVERY_PATH_INVALID"
]
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The incident_response_page_triage_moderate_v1 fixture is currently identical to the mild version in both its reconstructed artifacts and expected failure labels. To provide a meaningful progression of degradation within this family, the moderate level should ideally introduce additional failures (e.g., CAUSAL_DEPENDENCY_LOSS) or different artifact drift. If this is intentional for now, consider differentiating them to avoid redundancy in the test suite.

Comment on lines +43 to +57
def test_incident_response_severe_emits_only_expected_failure_labels() -> None:
original, reconstructed, contracts = _payload(SEVERE_ROOT)
expected_admissibility = _load_json(SEVERE_ROOT / "expected/admissibility.json")
expected_failures = _load_json(SEVERE_ROOT / "expected/failures.json")

results = ContractValidator().validate_contracts(original=original, reconstructed=reconstructed, contracts=contracts)

assert expected_admissibility["expected_admissible"] is False
assert all(not result.passed for result in results)

observed_contracts = sorted(result.contract_id for result in results)
assert observed_contracts == sorted(expected_admissibility["must_fail_contracts"])

observed_labels = sorted({result.failure_label for result in results if result.failure_label is not None})
assert observed_labels == sorted(expected_failures["expected_failures"])
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current test suite only validates the baseline and severe (degraded) fixtures, leaving mild and moderate untested. Additionally, the severe test assumes all contracts fail (all(not result.passed for result in results)), which is fragile if the fixture is updated to include passing contracts. I recommend refactoring these tests to iterate through all degradation levels and verify the passed status of each contract against the must_hold_contracts and must_fail_contracts lists defined in the fixture's admissibility.json.

@ProfRandom92 ProfRandom92 merged commit 36b7b22 into main May 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant