Push benchmark results to GitHub pages

[simonmarty]

Description

Why is this change being made?

1. Allows us to generate graphs of the agent's benchmarks.

What is changing?

1. Added a separate workflow with write permissions that runs on push to create benchmark charts.
2. Modified the workflow that runs on PRs so that it automatically fetches the latest bench baseline from GH pages.

We keep the two separate because they have different levels of permissions.

Related Links

• Issue #, if available:

---

Testing

Tested on fork https://simonmarty.github.io/aws-secretsmanager-agent/dev/bench/
Made a mock PR https://github.com/simonmarty/aws-secretsmanager-agent/actions/runs/24591788832

How was this tested?

1. Tested on fork https://simonmarty.github.io/aws-secretsmanager-agent/dev/bench/
2. Made a mock PR https://github.com/simonmarty/aws-secretsmanager-agent/actions/runs/24591788832

When testing locally, provide testing artifact(s):

---

Reviewee Checklist

Update the checklist after submitting the PR

• I have reviewed, tested and understand all changes
  If not, why:
• I have filled out the Description and Testing sections above
  If not, why:
• Build and Unit tests are passing
  If not, why:
• Unit test coverage check is passing
  If not, why:
• Integration tests pass locally
  If not, why:
• I have updated integration tests (if needed)
  If not, why: No change
• I have ensured no sensitive information is leaking (i.e., no logging of sensitive fields, or otherwise)
  If not, why:
• I have added explanatory comments for complex logic, new classes/methods and new tests
  If not, why:
• I have updated README/documentation (if needed)
  If not, why: No change
• I have clearly called out breaking changes (if any)
  If not, why: No breaking changes.

---

Reviewer Checklist

All reviewers please ensure the following are true before reviewing:

• Reviewee checklist has been accurately filled out
• Code changes align with stated purpose in description
• Test coverage adequately validates the changes

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.27%. Comparing base (e98c1ab) to head (677da78).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #193      +/-   ##
==========================================
+ Coverage   86.03%   86.27%   +0.23%     
==========================================
  Files          34       34              
  Lines        9331     9331              
  Branches     9331     9331              
==========================================
+ Hits         8028     8050      +22     
+ Misses       1170     1148      -22     
  Partials      133      133              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rstevens011]

for my edification: why is this being removed? Does benchmark-action/github-action-benchmark@v1 do this for you?

[simonmarty]

Sorry it took a minute to respond, I figured I could show you on my fork.

Here's a run from a push to main on the fork: https://github.com/simonmarty/aws-secretsmanager-agent/actions/runs/24808395936/job/72763846297

Here's a run against a no-op pull request I created on my fork, it correctly retrieves the test run results simonmarty#61 https://github.com/simonmarty/aws-secretsmanager-agent/actions/runs/24855062991/job/72765608319?pr=61

This does make me realize that there must be some variance in the performance of GitHub actions hosted runners. It's failing the regression check despite there not being any code changes. I'm going to set the workflow to be non-blocking and just post comments. Maybe up the tolerance threshold to the default of 200%.

[secrets-manager-ai-code-review]

The benchmark name here is 'Secrets Manager Agent Benchmarks' while the PR check workflow (benchmarks.yml) uses 'Secrets Manager Caching Benchmarks'. The github-action-benchmark stores and compares data keyed by this name. With mismatched names, the PR workflow will never find the baseline data pushed by this workflow, so PR comparisons will be against an empty/different dataset. These must match.

_{AI-generated, feedback may be incorrect}

[secrets-manager-ai-code-review]

After removing external-data-json-path (the cache) along with alert-threshold and fail-on-alert, this 'Compare benchmarks' step on PRs no longer has any baseline data to compare against locally, nor does it gate anything. For PRs (which don't typically have write access to push to the gh-pages branch), github-action-benchmark cannot fetch the baseline without external-data-json-path or save-data-file. Verify the PR comparison actually works—otherwise this step silently does nothing useful.

_{AI-generated, feedback may be incorrect}

[secrets-manager-ai-code-review]

There's no toolchain pinning/setup step (e.g. dtolnay/rust-toolchain or actions/setup). The baseline is built with whatever Rust ships on the ubuntu-latest image, which drifts over time. Benchmark baselines are sensitive to compiler version; unpinned toolchains can introduce phantom regressions in PR comparisons. Consider pinning the toolchain for reproducible baselines.

_{AI-generated, feedback may be incorrect}