Local source: coven-github/issues/10-add-structured-review-output-and-publication-gates.md
Summary
Agent output should be structured, validated, rechecked against live GitHub state, and passed through deterministic publication policy before comments, Check Runs, branches, or PRs are created. The current flow accepts a result envelope from coven-code and publishes partial outputs, but the review/fix publication contract is still under-specified.
Current Evidence
crates/worker/src/lib.rs runs coven-code --headless, reads a result path, updates Check Runs, and opens draft PRs when the result has commits and a branch.
README.md marks coven-code --headless invocation, PR creation, Check Runs, and Cave polling as partial.
- The revised architecture separates agent judgment from deterministic publication authority.
DESIGN.md calls for deterministic gates before repair or merge work.
Problem
An agent can produce stale, speculative, duplicate, or out-of-scope findings. It can also propose changes that no longer apply after the branch moves. Publication should not be a direct reflection of model output; it should be a controlled adapter decision.
Impact
- Bad or stale findings can be posted as authoritative review comments.
- Draft PRs can be opened against stale branches or wrong bases.
- Agent output can leak secrets or private context into public comments.
- Teams cannot configure severity thresholds, advisory-only modes, or write behavior.
Proposed Design
Define a strict result schema:
- task status and exit reason;
- reviewed target refs;
- findings with severity, confidence, path, line/range, evidence, and suggested validation;
- proposed patches/branch metadata;
- memory usage/proposals;
- warnings and redaction events.
Add publication gates:
- target ref is still current;
- finding path/line still exists;
- finding severity/confidence meets policy;
- no secret patterns in output;
- actor/trigger allows publication;
- branch protection and repo policy allow draft PR creation;
- memory writes are approved or held for review.
Acceptance Criteria
- Worker rejects result envelopes that do not match schema.
- Publication does not proceed when target refs are stale.
- Findings are filtered for duplicate, stale, low-confidence, and out-of-scope cases.
- Secret redaction runs before Check Run, comment, PR body, and task API output.
- Repo policy can choose advisory comment, Check Run only, request changes, or draft PR modes.
- Tests cover stale branch, malformed result, secret in output, low-confidence finding, and draft PR allowed/denied policy.
Test Notes
Create schema-level tests and mocked publication tests. Feed intentionally malformed or unsafe result JSON and assert that the adapter records failure without publishing unsafe GitHub output.
Local source:
coven-github/issues/10-add-structured-review-output-and-publication-gates.mdSummary
Agent output should be structured, validated, rechecked against live GitHub state, and passed through deterministic publication policy before comments, Check Runs, branches, or PRs are created. The current flow accepts a result envelope from
coven-codeand publishes partial outputs, but the review/fix publication contract is still under-specified.Current Evidence
crates/worker/src/lib.rsrunscoven-code --headless, reads a result path, updates Check Runs, and opens draft PRs when the result has commits and a branch.README.mdmarkscoven-code --headlessinvocation, PR creation, Check Runs, and Cave polling as partial.DESIGN.mdcalls for deterministic gates before repair or merge work.Problem
An agent can produce stale, speculative, duplicate, or out-of-scope findings. It can also propose changes that no longer apply after the branch moves. Publication should not be a direct reflection of model output; it should be a controlled adapter decision.
Impact
Proposed Design
Define a strict result schema:
Add publication gates:
Acceptance Criteria
Test Notes
Create schema-level tests and mocked publication tests. Feed intentionally malformed or unsafe result JSON and assert that the adapter records failure without publishing unsafe GitHub output.