Skip to content

fix(e2e): stabilize shard-3 diary flakes#1855

Merged
steilerDev merged 1 commit into
betafrom
fix/1829-e2e-shard3-flakes
Jul 8, 2026
Merged

fix(e2e): stabilize shard-3 diary flakes#1855
steilerDev merged 1 commit into
betafrom
fix/1829-e2e-shard3-flakes

Conversation

@steilerDev

Copy link
Copy Markdown
Owner

Summary

  • Root-caused and fixed the recurring E2E shard-3 (16-shard desktop matrix) diary flakes that were blocking beta -> main promotions on PRs release: promote beta to main (cost basis filter, diary fix, auto-itemize refactor) #1791/fix(deps): scope js-yaml override by version range to unbreak docs deploy #1828.
  • diary-drafts.spec.ts Scenario 14: test.slow() only triples the overall per-test timeout, not per-step actionTimeout/expect.timeout — the test paired a 45s ceiling with several 30-45s step timeouts, starving the finally-block cleanup under CI load. Replaced with an explicit test.setTimeout(60_000), proportionate 15s step ceilings, and a guaranteed-cleanup-time pattern (testInfo.setTimeout(testInfo.timeout + 15_000) + try/catch) in the finally block.
  • diary-r2-uat.spec.ts Scenario 10: fixed a real race where a stale in-flight response from a prior UI transition could satisfy the next waitForResponse() predicate, leaving a shared requests[] capture array empty. Now awaits each transition's own response explicitly.
  • document-linking.spec.ts Scenario 7a: was collateral maxFailures:1 cancellation from Scenario 14, not an independent bug — added the same defensive cleanup-time pattern anyway.
  • playwright.config.ts: comment-only documentation (verified against Playwright 1.61.1 dispatcher source) that maxFailures already tolerates one retry-passing test before fail-fast cancels a shard.
  • cornerstoneContainer.ts: incidental fix for a testcontainers@12.0.4 API rename (withCopyFileToContainer -> withCopyFilesToContainer) discovered during local verification.

Fixes #1829

Test plan

  • eslint clean on all touched files
  • prettier clean on all touched files
  • tsc --noEmit in e2e/ shows no new errors vs baseline (one pre-existing error in a touched file is incidentally fixed)
  • CI Quality Gates
  • Full E2E matrix, shard 3 in particular, green

Co-Authored-By: Claude dev-team-lead (Sonnet 4.6) noreply@anthropic.com

…sponse race, cleanup guarantees

Root-causes the recurring shard-3 (16-shard desktop matrix) diary E2E
failures on main-targeted PRs:

- diary-drafts.spec.ts Scenario 14: test.slow() only triples the
  OVERALL per-test timeout (15s->45s), not actionTimeout/expect.timeout.
  The test paired that 45s ceiling with four 30-45s per-step timeouts,
  any one of which could alone exhaust the budget under 8-worker CI
  load — starving the finally-block cleanup and surfacing as an
  "apiRequestContext.delete: Test timeout exceeded" (a symptom, not the
  cause). Fixed with an explicit test.setTimeout(60s), proportionate
  15s step ceilings, and a guaranteed-cleanup-time pattern
  (testInfo.setTimeout(testInfo.timeout + 15s) + try/catch in finally)
  so teardown always gets its own budget regardless of how much of the
  test body's time was consumed.
- diary-r2-uat.spec.ts Scenario 10: a route mock that always returns an
  empty page defeated waitForLoaded()'s visibility race, letting a
  stale in-flight response from the prior "All" click satisfy the next
  waitForResponse() predicate before the real "Manual" request/response
  pair occurred, intermittently leaving a shared requests[] capture
  array empty. Fixed by explicitly awaiting each transition's own
  response and reading the request straight off the resolved Response
  object.
- document-linking.spec.ts Scenario 7a: was collateral cancellation
  from Scenario 14's maxFailures:1 trip, not an independent bug. Added
  the same defensive guaranteed-cleanup-time pattern as cheap insurance.
- playwright.config.ts: comment-only documentation, verified against
  the Playwright 1.61.1 dispatcher source, that maxFailures already
  tolerates one retry-passing (flaky) test before fail-fast cancels a
  shard — no config change needed.
- cornerstoneContainer.ts: incidental fix for a testcontainers 12.0.4
  API rename (withCopyFileToContainer -> withCopyFilesToContainer)
  discovered while attempting local containerized verification.

Fixes #1829

Co-Authored-By: Claude dev-team-lead (Sonnet 4.6) <noreply@anthropic.com>
Co-Authored-By: Claude e2e-test-engineer (Sonnet 4.5) <noreply@anthropic.com>

@steilerDev steilerDev left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[product-architect] APPROVED

Reviewed as the always-on architect reviewer for this e2e-only change (per the PR Review Gate, security-engineer/ux-designer correctly don't trigger). Scope is clean: only e2e/ and agent-memory touched — zero production (server/, client/, shared/) changes, so no API-contract/schema/architecture surface is affected. I verified the technical claims rather than taking them on faith.

Test-strength preservation — confirmed, no weakened assertions:

  • diary-r2-uat Scenario 10: the full assertion block is intact — type param present, contains ≥1 manual type, and contains zero automatic types. Reading type off manualResponse.url() (the exact response the Manual click produced) is stronger than the old shared-mutable requests[] array, which is precisely the race being removed. Good fix.
  • diary-drafts Scenario 14: the 45s→15s step-ceiling reductions are not a weakening — assertions fire only after entryLoadPromise confirms the API response arrived, so the remaining work is a client-side React re-render that completes in well under 15s. The 45s values were budget-reservations masquerading as ceilings; tightening them is correct.

Timeout arithmetic — sound. 60s test budget + proportionate 15s "something is actually stuck" ceilings + additive testInfo.setTimeout(testInfo.timeout + 15s) cleanup extension. Because testInfo.timeout is the full original budget and the body can consume at most that before timing out, the finally block always gets ≥15s of guaranteed teardown headroom. The test.slow() correction is accurate (it triples only the overall per-test timeout, not actionTimeout/expect.timeout), as is the maxFailures note (a retry-passing "flaky" test never increments the failure count — only all-attempts-failing tests trip fail-fast).

Incidental testcontainers fix — verified genuine. withCopyFileToContainer (singular) is gone from testcontainers 12.0.4; withCopyFilesToContainer(filesToCopy: FileToCopy[]) is the correct replacement. The proxy-CA path would not compile/run without this, so it's a necessary carry, not scope creep.

Reusable patterns — worth codifying, and correctly placed. Both the guaranteed-cleanup-time pattern and per-transition response sync are genuinely reusable e2e conventions. They belong in the e2e-test-engineer's domain, and the PR already lands them in general-e2e-patterns.md. No architect-owned wiki page covers E2E test conventions, so no wiki update is required from me — agent memory is the right home. (If these keep recurring, a dedicated "E2E Conventions" wiki page could be considered later, but that's not a blocker.)

5-consecutive-runs criterion — pragmatically satisfied. A full 16-shard × 3-viewport matrix isn't run pre-merge on beta PRs, so 5 green pre-merge matrix runs aren't achievable here. More importantly, this PR does real root-cause analysis (the test.slow() misconception + the stale-in-flight-response race) rather than a retry/timeout bump — that is a materially stronger signal of a durable fix than 5 green runs of otherwise-unchanged flaky code. On-PR evidence (all 3 target tests green on shard 3 at 1–2s) + post-merge monitoring is a reasonable close-out, and the eventual beta → main promotion PR's full E2E Gates run serves as the confirming pass.

Non-blocking / informational:

  1. The testInfo.setTimeout comment phrases the extension as "additive to the current remaining deadline" — Playwright actually sets the deadline relative to test start (start + timeout), not to remaining time. The end guarantee (≥15s cleanup headroom) is correct because testInfo.timeout is the full budget, so the wording is a touch loose but not misleading.
  2. In Scenario 14, the four 15s step ceilings sum to the 60s test budget, leaving no theoretical slack before cleanup — but realistic step durations are 1–2s and cleanup gets its own +15s extension, so this is only a pathological-worst-case concern.

No architecture, contract, or schema implications. Approving.

@steilerDev steilerDev merged commit dc5cac2 into beta Jul 8, 2026
23 of 28 checks passed
@steilerDev steilerDev deleted the fix/1829-e2e-shard3-flakes branch July 8, 2026 01:43
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 2.13.0-beta.24 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant