Skip to content

Make doctor suggest observed check names when required checks are never observed #57

Description

@coygeek

Summary

deploybot doctor can warn that a configured required check was never observed, but it does not surface the check names it did observe. When a policy accidentally uses a workflow name where GitHub exposes matrix job check names, the current warning is correct but not actionable enough.

Evidence

Using the example policy against this repository:

[queue]
required_checks = ["CI"]

deploybot doctor --json reports:

{
  "check": "required-checks",
  "status": "warn",
  "detail": "Never observed: CI",
  "hint": "Use the exact GitHub check display names."
}

However, inspecting an open PR shows the observed check names are the matrix jobs:

"checks": {
  "test (3.11)": "passed",
  "test (3.12)": "passed",
  "test (3.13)": "passed"
}

This means required_checks = ["CI"] is probably the wrong value for this repo's PR gate, even though CI is the workflow name.

Impact

This is a sharp onboarding edge:

  • pipeline.ci_workflows wants workflow names such as CI.
  • queue.required_checks wants exact PR check display names such as test (3.11).

When these are confused, ready PRs can appear blocked with a generic missing-check reason even though the relevant matrix checks have passed.

The user or agent then has to discover the correct check display names manually from PR rollups.

Proposed fix

When doctor warns that required checks were never observed, include a short list of observed check names in the warning detail or hint.

For example:

{
  "check": "required-checks",
  "status": "warn",
  "detail": "Never observed: CI",
  "hint": "Use exact GitHub check display names. Observed on recent PRs: test (3.11), test (3.12), test (3.13)."
}

If there are too many observed checks, cap the list and report the count:

Observed on recent PRs: test (3.11), test (3.12), test (3.13), ... (12 total)

Acceptance criteria

  • When a configured required check is never observed, doctor --json includes the observed candidate check names it sampled.
  • Text-mode doctor also shows those candidates or points to a command that prints them.
  • The warning distinguishes workflow names from check display names.
  • The implementation remains advisory: unobserved required checks are still warn, not fail.
  • Tests cover a config with required_checks = ["CI"] and observed check names like test (3.11).

Verification

  • Add/update unit tests in tests/test_doctor.py.
  • Run python -m unittest discover -s tests.
  • Run ruff check ..

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions