Skip to content

test(deps): skip transient failures in github URL reachability test#344

Merged
dpup merged 1 commit into
mainfrom
fix/flaky-registry-url-test
Apr 27, 2026
Merged

test(deps): skip transient failures in github URL reachability test#344
dpup merged 1 commit into
mainfrom
fix/flaky-registry-url-test

Conversation

@dpup
Copy link
Copy Markdown
Collaborator

@dpup dpup commented Apr 27, 2026

Summary

TestRegistryGithubBinaryURLsExist HEADs every github-binary release URL in the registry to catch wrong asset names. Until now any non-2xx response was a hard failure — including transient 5xx blips from GitHub's CDN. That broke main in run 25020206997 when ripgrep's arm64 asset returned a 502, and it'll keep doing so on every GitHub hiccup.

This change keeps the validation but classifies the response:

  • 404 / 410 → fail (asset genuinely missing — what the test exists to catch)
  • other 4xx → fail (e.g. 401/403 would mean a real config problem)
  • 5xx → skip the subtest (transient server-side)
  • transport error / timeout → skip the subtest (transient network)

Also fixes the comment on the test, which claimed it was opt-in via a -urls flag — it only ever honored testing.Short.

Test plan

  • go vet ./internal/deps/... clean
  • golangci-lint run ./internal/deps/... clean
  • go test -short ./internal/deps/... passes (skips the URL test)
  • CI green on this branch

TestRegistryGithubBinaryURLsExist hits real GitHub release URLs to verify
asset names. Transient 5xx responses and network errors were being reported
as test failures, breaking CI on every GitHub blip (e.g. ripgrep arm64 502
on main, run 25020206997).

Only fail on 404/410 (asset genuinely missing) and other 4xx (real config
problems). Skip on 5xx and transport errors. Update the test comment to
match actual behavior — it claimed to be opt-in via -urls, but only
honored testing.Short.
@claude
Copy link
Copy Markdown

claude Bot commented Apr 27, 2026

Claude finished @dpup's task in 50s —— View job


Code Review

  • Read changed files
  • Analyze logic and error handling
  • Review against CLAUDE.md conventions
  • Post review feedback

Overall: LGTM. This is a clean, focused fix. The error classification is correct and the comment fix is accurate.

What's good:

  • The three-tier classification (404/410 → fail, 5xx → skip, transport error → skip) maps cleanly to "definitely broken", "server-side transient", and "network transient". That's the right mental model.
  • Fixing the comment that falsely claimed a -urls flag existed is a nice catch.
  • t.Skipf inside t.Run correctly skips only that subtest without poisoning the parent — the parallel structure handles this well.

One thing worth considering:

GitHub could return 429 Too Many Requests if the test hits rate limits on release asset URLs. The current case resp.StatusCode >= 400 branch would mark that as a hard failure rather than a skip. Release asset URLs are CDN-backed and less likely to rate-limit than the API, but if CI ever runs this test concurrently across many jobs it could surface. If it becomes an issue, resp.StatusCode == http.StatusTooManyRequests could be added alongside the 5xx skip case.

Minor nit (not worth a change): The client.Timeout: 10*time.Second and the per-request context.WithTimeout(..., 10*time.Second) are redundant — whichever fires first wins, and both are 10s. Not harmful, just slightly noisy.

Neither of these blocks merging.

@dpup dpup merged commit b8afc8e into main Apr 27, 2026
4 checks passed
@dpup dpup deleted the fix/flaky-registry-url-test branch April 27, 2026 22:02
@dpup dpup mentioned this pull request Apr 28, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant