Add bug-agent pipeline with explicit comment trigger#8640
Add bug-agent pipeline with explicit comment trigger#8640
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a multi-stage automated bug pipeline: three GitHub Actions workflows and accompanying agent guidance. New workflows are 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
polmichel
force-pushed
the
pmi-20260318-bug-analyst-agent
branch
from
c72ed6b to
6812369
Compare
polmichel
force-pushed
the
pmi-20260318-bug-analyst-agent
branch
from
dbfdf67 to
232dcc7
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
.github/workflows/bug-agent-review.yml (1)
31-44: Consider prompt injection risk from PR body interpolation.The PR body (
github.event.pull_request.body) is directly interpolated into the prompt. While the fix agent controls this content in the expected flow, a malicious actor could manually create a PR withAGENT_FIX_COMPLETEin the body and inject adversarial instructions. Consider whether this risk is acceptable given the agent's limited permissions (read + PR comments).Additionally, pinning
anthropics/claude-code-actionto a specific version or commit SHA rather than@mainwould improve reproducibility and protect against upstream changes.🔒 Proposed version pinning
- - uses: anthropics/claude-code-action@main + - uses: anthropics/claude-code-action@v1 # or pin to specific SHA🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/bug-agent-review.yml around lines 31 - 44, The workflow currently interpolates untrusted github.event.pull_request.body into the Claude prompt and pins anthropics/claude-code-action to `@main`; to fix, sanitize or validate the PR body before injection (e.g., strip/escape control tokens like AGENT_FIX_COMPLETE or any instruction-like lines, or limit insertion to a safe summary from steps.prompt.outputs.PROMPT) so the prompt cannot be manipulated by a malicious PR, and pin the action reference (replace anthropics/claude-code-action@main) to a specific release tag or commit SHA to avoid upstream changes; update the workflow where steps.prompt.outputs.PROMPT and github.event.pull_request.body are combined and the uses entry for anthropics/claude-code-action accordingly..github/workflows/bug-agent-analyst.yml (1)
34-46: Consider prompt injection risk and action version pinning.Similar to the review workflow, the issue body is directly interpolated into the prompt. While less risky since it requires a
buglabel and explicit/bug-analyzetrigger, consider whether adversarial issue content could manipulate the agent.For consistency across all three workflows, pin
anthropics/claude-code-actionto a specific version:🔒 Proposed version pinning
- - uses: anthropics/claude-code-action@main + - uses: anthropics/claude-code-action@v1 # or pin to specific SHA🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/bug-agent-analyst.yml around lines 34 - 46, The workflow currently uses anthropics/claude-code-action@main and injects github.event.issue.body and steps.prompt.outputs.PROMPT directly into the `prompt`, creating prompt-injection risk and an unpinned action; change the action reference from `anthropics/claude-code-action@main` to a specific released tag (e.g., a stable semver tag) and sanitize/escape or validate `github.event.issue.body` (and any content fed into `steps.prompt.outputs.PROMPT`) before interpolation—implement a step that strips or encodes dangerous characters and enforces the `bug` label and `/bug-analyze` trigger as preconditions so only validated issue text reaches the `prompt`.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/bug-agent-analyst.yml:
- Around line 17-21: The workflow's permissions block currently grants
"contents: read" which prevents pushes; update the permissions under the
"permissions:" key by replacing "contents: read" with "contents: write" so the
analyst agent can create a branch, commit, and push as instructed (look for the
"permissions:" stanza and the "contents: read" entry).
---
Nitpick comments:
In @.github/workflows/bug-agent-analyst.yml:
- Around line 34-46: The workflow currently uses
anthropics/claude-code-action@main and injects github.event.issue.body and
steps.prompt.outputs.PROMPT directly into the `prompt`, creating
prompt-injection risk and an unpinned action; change the action reference from
`anthropics/claude-code-action@main` to a specific released tag (e.g., a stable
semver tag) and sanitize/escape or validate `github.event.issue.body` (and any
content fed into `steps.prompt.outputs.PROMPT`) before interpolation—implement a
step that strips or encodes dangerous characters and enforces the `bug` label
and `/bug-analyze` trigger as preconditions so only validated issue text reaches
the `prompt`.
In @.github/workflows/bug-agent-review.yml:
- Around line 31-44: The workflow currently interpolates untrusted
github.event.pull_request.body into the Claude prompt and pins
anthropics/claude-code-action to `@main`; to fix, sanitize or validate the PR body
before injection (e.g., strip/escape control tokens like AGENT_FIX_COMPLETE or
any instruction-like lines, or limit insertion to a safe summary from
steps.prompt.outputs.PROMPT) so the prompt cannot be manipulated by a malicious
PR, and pin the action reference (replace anthropics/claude-code-action@main) to
a specific release tag or commit SHA to avoid upstream changes; update the
workflow where steps.prompt.outputs.PROMPT and github.event.pull_request.body
are combined and the uses entry for anthropics/claude-code-action accordingly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ff9231ea-fcb5-47a6-8d61-e711a50d0601
📒 Files selected for processing (6)
.github/bug-agent-pipeline/analyst.md.github/bug-agent-pipeline/fix.md.github/bug-agent-pipeline/review.md.github/workflows/bug-agent-analyst.yml.github/workflows/bug-agent-fix.yml.github/workflows/bug-agent-review.yml
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/bug-agent-analyst.yml:
- Around line 14-16: The current job if condition only checks the comment text
and the 'bug' label and must be hardened to allow only trusted commenters;
update the job-level if to also require that
github.event.comment.author_association is one of OWNER, MEMBER, or COLLABORATOR
(e.g., add checks alongside contains(github.event.comment.body, '/bug-analyze')
and contains(github.event.issue.labels.*.name, 'bug') so the job runs only when
the commenter’s author_association equals 'OWNER' or 'MEMBER' or
'COLLABORATOR').
- Around line 6-8: Add a YAML document start marker and quote the problematic
key to satisfy yamllint: insert a leading `---` at the top of the workflow and
change the unquoted key on: to a quoted key "on": so the existing name: "Bug
analyst agent" and its following on/"on" block remain unchanged; this removes
the `document-start` and `truthy` warnings reported for name and on.
- Line 34: The workflow currently references the GitHub Action as
"anthropics/claude-code-action@main"; replace that mutable ref with an immutable
commit SHA (e.g., "anthropics/claude-code-action@<commit-sha>") to pin the
action; locate the uses: anthropics/claude-code-action@main entry in the
workflow, update the ref to a specific commit SHA, and verify the workflow runs
against that SHA (and optionally add a comment with the chosen tag/PR for future
updates).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9b90b72a-d633-4803-bd38-cec75bac3bb6
📒 Files selected for processing (1)
.github/workflows/bug-agent-analyst.yml
|
looks like it could be valuable |
Adds analyst, fix, and review agents as GitHub Actions workflows triggered by /bug-analyze issue comments. Each agent has a dedicated prompt file and workflow definition. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
@ajtmccarty A few minutes ago, I tested the workflow locally on this bug. #8670 / #8620 First feedbacks on my side:
|
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
polmichel
force-pushed
the
pmi-20260318-bug-analyst-agent
branch
from
d9e87fa to
4f0c631
Compare
github-actions
bot
added
the
group/frontend
polmichel
force-pushed
the
pmi-20260318-bug-analyst-agent
branch
from
4f0c631 to
1c5f98a
Compare
github-actions
bot
removed
the
group/frontend
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/bug-agent-pipeline/fixer.md (1)
15-17: Add an explicit AGENTS.md consultation stepPlease add a short instruction near the top to consult repository
AGENTS.mdguidance before implementation, so fixer behavior stays aligned with project standards.Based on learnings: Agents should use the AGENTS.md documentation for implementation guidelines and best practices.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/bug-agent-pipeline/fixer.md around lines 15 - 17, Insert a short step near the top of .github/bug-agent-pipeline/fixer.md instructing the fixer to consult the repository AGENTS.md before implementing fixes; specifically add a line immediately above or following the existing "Read the analysis carefully" step (the block that begins "1. Read the analysis carefully: pay special attention to the 'Notes for the fix agent' section.") that says something like "Consult AGENTS.md for repository-specific agent guidance and best practices before implementing fixes." Ensure the new instruction is brief, prominent, and references AGENTS.md so fixers follow project standards.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/bug-agent-pipeline/fixer.md:
- Around line 59-68: Renumber the procedural steps so they are sequential (the
current "7." and "8." should follow the previous "5."), e.g., change "7. Open a
**DRAFT Pull Request**..." to "6. Open a **DRAFT Pull Request**..." and change
the following "8. Post a comment..." to "7. Post a comment..." (and update any
other downstream step numbers if present), while keeping the PR-body
instructions and the hidden marker `<!-- AGENT_FIX_COMPLETE -->` unchanged.
- Around line 32-37: Replace direct directory changes that leak state by running
those commands in isolated subshells; specifically, change the invocation that
uses "cd frontend/app && npx biome check --write ." so the directory change is
scoped (use a subshell or pushd/popd) and does not affect subsequent phases, and
apply the same pattern to the other sequential commands mentioned around lines
39-50 so each phase runs with its own working directory isolation.
---
Nitpick comments:
In @.github/bug-agent-pipeline/fixer.md:
- Around line 15-17: Insert a short step near the top of
.github/bug-agent-pipeline/fixer.md instructing the fixer to consult the
repository AGENTS.md before implementing fixes; specifically add a line
immediately above or following the existing "Read the analysis carefully" step
(the block that begins "1. Read the analysis carefully: pay special attention to
the 'Notes for the fix agent' section.") that says something like "Consult
AGENTS.md for repository-specific agent guidance and best practices before
implementing fixes." Ensure the new instruction is brief, prominent, and
references AGENTS.md so fixers follow project standards.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 426d9f01-b407-449f-bfd1-19a811f52d2c
📒 Files selected for processing (6)
.github/bug-agent-pipeline/analyst.md.github/bug-agent-pipeline/fixer.md.github/bug-agent-pipeline/reviewer.md.github/workflows/bug-agent-analyst.yml.github/workflows/bug-agent-fix.yml.github/workflows/bug-agent-review.yml
✅ Files skipped from review due to trivial changes (4)
- .github/bug-agent-pipeline/reviewer.md
- .github/bug-agent-pipeline/analyst.md
- .github/workflows/bug-agent-review.yml
- .github/workflows/bug-agent-fix.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/bug-agent-analyst.yml
Calling such command seems overkill. This is fine from a user perspective as it is some sort of shortcut for us. However, it's litteraly about calling |
This may be overkill yes, however as the fix could include front-end code, schema modification, maybe |
All our linting commands should produce idempotent results (if they do not then we have bugs). So we should just call them one after the other not matter what was changed. I'm not in favour of having some agent depend on a claude command (because LLM do not operate in a idempotent way). But that's a personal take, I'll accept that I can be wrong if other people feel comfortable with using a claude slash command instead. |
But besides the Claude command, on which I agree with you, would we need commands other than just
|
|
Not sure if we should call
|
There was a problem hiding this comment.
🧹 Nitpick comments (3)
.github/bug-agent-pipeline/fixer.md (3)
48-48: Consider consistent error handling guidance for all Phase 2 commands.Line 48 provides explicit fallback instructions for
uv lock --check(runuv lockand commit), but other commands in Phase 2—especially the generate commands (lines 43-47)—lack similar guidance. Ifbackend.generateorschema.generate-graphqlschemafails or produces unexpected changes, the agent may be uncertain how to proceed.Adding a general note like "If any generate command fails, review the error, fix the root cause, and re-run" would improve consistency and agent confidence.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/bug-agent-pipeline/fixer.md at line 48, Add consistent fallback guidance for Phase 2 generate and lock steps: for commands like `backend.generate`, `schema.generate-graphqlschema`, and `uv lock --check` instruct the agent that if any generate command fails or yields unexpected changes it should inspect the error/output, fix the root cause (e.g., dependency, schema, or config issue), re-run the failing command, and if `uv lock --check` reports change, run `uv lock` and commit the updated lockfile; include a short general note covering these steps so all Phase 2 commands have the same error-handling guidance.
3-11: Consider referencing AGENTS.md for additional context.Based on learnings, agent-related documents should reference AGENTS.md for implementation guidelines and best practices. Adding a brief note directing the agent to consult AGENTS.md (if it exists in the repository) would align with project conventions and provide the agent with additional context.
Based on learnings: "Agents should use the AGENTS.md documentation for implementation guidelines and best practices."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/bug-agent-pipeline/fixer.md around lines 3 - 11, Add a short note under the "Before proceeding" or "Your role" section instructing the bug analyst agent to consult AGENTS.md (if present) for implementation guidelines and best practices; update the .github/bug-agent-pipeline/fixer.md content to include a single sentence like "See AGENTS.md for agent implementation guidelines and best practices (if present in the repository)" so agents know to reference that document when working on the branch referenced by the "Before proceeding" instructions.
39-52: Clarify parallel execution syntax for the AI agent.Line 39 instructs the agent to "run all in parallel," but the bullet-point list provides no explicit parallelization mechanism (e.g., bash
&withwait, GNUparallel, orxargs -P). An AI agent may interpret this as sequential execution, which would miss the intended performance benefit.Consider adding explicit parallel execution syntax or a note like: "Use background jobs (
&) andwait, or run withxargs -Pto execute concurrently."🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/bug-agent-pipeline/fixer.md around lines 39 - 52, The "Phase 2 — Regenerate & lint (run all in parallel)" step is ambiguous because the listed commands (e.g., `uv run invoke main.scan`, `uv run invoke main.lint`, `uv run invoke backend.lint`, `uv run invoke backend.generate`, `uv run invoke schema.generate-graphqlschema`, `uv run invoke schema.generate-jsonschema`, `uv run invoke docs.generate`, `uv run invoke docs.lint`, `uv lock --check`, `(cd frontend/app && npm run codegen:graphql)`, `(cd frontend/app && npm run codegen:openapi)`, `(cd frontend/app && npx betterer --update)`) lack explicit parallelization syntax; update the docs to show how to run them concurrently (for example instruct the agent to run each command as a background job with `&` and then `wait`, or provide an alternative using `xargs -P`/GNU `parallel`) and include a short example or note stating which method is preferred so the agent will not execute them sequentially.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/bug-agent-pipeline/fixer.md:
- Line 48: Add consistent fallback guidance for Phase 2 generate and lock steps:
for commands like `backend.generate`, `schema.generate-graphqlschema`, and `uv
lock --check` instruct the agent that if any generate command fails or yields
unexpected changes it should inspect the error/output, fix the root cause (e.g.,
dependency, schema, or config issue), re-run the failing command, and if `uv
lock --check` reports change, run `uv lock` and commit the updated lockfile;
include a short general note covering these steps so all Phase 2 commands have
the same error-handling guidance.
- Around line 3-11: Add a short note under the "Before proceeding" or "Your
role" section instructing the bug analyst agent to consult AGENTS.md (if
present) for implementation guidelines and best practices; update the
.github/bug-agent-pipeline/fixer.md content to include a single sentence like
"See AGENTS.md for agent implementation guidelines and best practices (if
present in the repository)" so agents know to reference that document when
working on the branch referenced by the "Before proceeding" instructions.
- Around line 39-52: The "Phase 2 — Regenerate & lint (run all in parallel)"
step is ambiguous because the listed commands (e.g., `uv run invoke main.scan`,
`uv run invoke main.lint`, `uv run invoke backend.lint`, `uv run invoke
backend.generate`, `uv run invoke schema.generate-graphqlschema`, `uv run invoke
schema.generate-jsonschema`, `uv run invoke docs.generate`, `uv run invoke
docs.lint`, `uv lock --check`, `(cd frontend/app && npm run codegen:graphql)`,
`(cd frontend/app && npm run codegen:openapi)`, `(cd frontend/app && npx
betterer --update)`) lack explicit parallelization syntax; update the docs to
show how to run them concurrently (for example instruct the agent to run each
command as a background job with `&` and then `wait`, or provide an alternative
using `xargs -P`/GNU `parallel`) and include a short example or note stating
which method is preferred so the agent will not execute them sequentially.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 33e1618b-e2b6-4fe7-8b27-ead0b0180dd2
📒 Files selected for processing (1)
.github/bug-agent-pipeline/fixer.md
I will close this PR and do another test with the new feedbacks incorporated |
root cause analysis review
suggested fix review
conclusionseems like the LLM did a fine job of reviewing and trying to fix the bug. I think that this approach would work, but we shouldn't do it. it might give a better result on a different bug |
| github.event.comment.author_association == 'MEMBER' || | ||
| github.event.comment.author_association == 'COLLABORATOR' | ||
| ) | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
I think we should run these actions on our bigger runners to allow the agent to run the tests
| contents: write | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 |
| cat .github/bug-agent-pipeline/analyst.md >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
|
|
||
| - uses: anthropics/claude-code-action@6062f3709600659be5e47fcddf2cf76993c235c2 # main 2026-03-23 |
There was a problem hiding this comment.
Prefer the release version instead
| pull-requests: write | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 |
| cat .github/bug-agent-pipeline/fix.md >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
| - uses: anthropics/claude-code-action@main |
| pull-requests: write | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 |
| cat .github/bug-agent-pipeline/review.md >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
|
|
||
| - uses: anthropics/claude-code-action@main |
|
|
||
| "on": | ||
| pull_request: | ||
| types: [opened, ready_for_review] |
There was a problem hiding this comment.
That's not a draft PR based on those types.
Also, we could limit the target branch as well, to avoid starting it for nothing until the job gets cancelled due to missing the AGENT_FIX_COMPLETE
| id: prompt | ||
| run: | | ||
| echo "PROMPT<<EOF" >> $GITHUB_OUTPUT | ||
| cat .github/bug-agent-pipeline/fix.md >> $GITHUB_OUTPUT |
There was a problem hiding this comment.
Workflow files reference fix.md / review.md but files were renamed to fixer.md / reviewer.md — fixer and reviewer will fail at runtime
| id: prompt | ||
| run: | | ||
| echo "PROMPT<<EOF" >> $GITHUB_OUTPUT | ||
| cat .github/bug-agent-pipeline/review.md >> $GITHUB_OUTPUT |
There was a problem hiding this comment.
Workflow files reference fix.md / review.md but files were renamed to fixer.md / reviewer.md — fixer and reviewer will fail at runtime
| - `uv run invoke docs.generate` | ||
| - `uv run invoke docs.lint` | ||
| - `uv lock --check` (if it fails, run `uv lock` and commit the updated lockfile) | ||
| - `(cd frontend/app && npm run codegen:graphql)` |
| fix: | ||
| if: | | ||
| contains(github.event.comment.body, 'AGENT_ANALYSIS_COMPLETE') && | ||
| contains(github.event.issue.labels.*.name, 'bug') |
There was a problem hiding this comment.
Fixer workflow has no author_association guard
|
|
||
| jobs: | ||
| review: | ||
| if: contains(github.event.pull_request.body, 'AGENT_FIX_COMPLETE') |
There was a problem hiding this comment.
Review workflow triggers on any PR containing the marker — no author check
5 participants
Why
We need an automated bug triage pipeline using Claude Code agents (analyst → fix → review), but it should not trigger automatically on every bug issue to avoid flooding the project with agent activity.
What changed
.github/bug-agent-pipeline/:analyst.md,fix.md,review.md— structured instructions for each stage of the pipeline.github/workflows/:bug-agent-analyst.yml,bug-agent-fix.yml,bug-agent-review.yml/bug-analyzeas a comment on abug-labeled issue (not on issue open/label)AGENT_ANALYSIS_COMPLETEmarker, and the review agent triggers on theAGENT_FIX_COMPLETEmarker in a PR bodyHow to review
ifconditions are tight enoughHow to test
buglabel/bug-analyzeANTHROPIC_API_KEYsecret configured)Impact & rollout
ANTHROPIC_API_KEYsecret in the repository/bug-analyzeSummary by CodeRabbit
/bug-analyzeon bug-labeled issues, auto-generate draft fixes when analysis completes, and perform automated reviews for agent-created PRs.