Skip to content

Add structured review output and deterministic publication gates #11

@romgenie

Description

@romgenie

Local source: coven-github/issues/10-add-structured-review-output-and-publication-gates.md

Summary

Agent output should be structured, validated, rechecked against live GitHub state, and passed through deterministic publication policy before comments, Check Runs, branches, or PRs are created. The current flow accepts a result envelope from coven-code and publishes partial outputs, but the review/fix publication contract is still under-specified.

Current Evidence

  • crates/worker/src/lib.rs runs coven-code --headless, reads a result path, updates Check Runs, and opens draft PRs when the result has commits and a branch.
  • README.md marks coven-code --headless invocation, PR creation, Check Runs, and Cave polling as partial.
  • The revised architecture separates agent judgment from deterministic publication authority.
  • DESIGN.md calls for deterministic gates before repair or merge work.

Problem

An agent can produce stale, speculative, duplicate, or out-of-scope findings. It can also propose changes that no longer apply after the branch moves. Publication should not be a direct reflection of model output; it should be a controlled adapter decision.

Impact

  • Bad or stale findings can be posted as authoritative review comments.
  • Draft PRs can be opened against stale branches or wrong bases.
  • Agent output can leak secrets or private context into public comments.
  • Teams cannot configure severity thresholds, advisory-only modes, or write behavior.

Proposed Design

Define a strict result schema:

  • task status and exit reason;
  • reviewed target refs;
  • findings with severity, confidence, path, line/range, evidence, and suggested validation;
  • proposed patches/branch metadata;
  • memory usage/proposals;
  • warnings and redaction events.

Add publication gates:

  • target ref is still current;
  • finding path/line still exists;
  • finding severity/confidence meets policy;
  • no secret patterns in output;
  • actor/trigger allows publication;
  • branch protection and repo policy allow draft PR creation;
  • memory writes are approved or held for review.

Acceptance Criteria

  • Worker rejects result envelopes that do not match schema.
  • Publication does not proceed when target refs are stale.
  • Findings are filtered for duplicate, stale, low-confidence, and out-of-scope cases.
  • Secret redaction runs before Check Run, comment, PR body, and task API output.
  • Repo policy can choose advisory comment, Check Run only, request changes, or draft PR modes.
  • Tests cover stale branch, malformed result, secret in output, low-confidence finding, and draft PR allowed/denied policy.

Test Notes

Create schema-level tests and mocked publication tests. Feed intentionally malformed or unsafe result JSON and assert that the adapter records failure without publishing unsafe GitHub output.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions