fix(e2e): stabilize shard-3 diary flakes#1855
Conversation
…sponse race, cleanup guarantees Root-causes the recurring shard-3 (16-shard desktop matrix) diary E2E failures on main-targeted PRs: - diary-drafts.spec.ts Scenario 14: test.slow() only triples the OVERALL per-test timeout (15s->45s), not actionTimeout/expect.timeout. The test paired that 45s ceiling with four 30-45s per-step timeouts, any one of which could alone exhaust the budget under 8-worker CI load — starving the finally-block cleanup and surfacing as an "apiRequestContext.delete: Test timeout exceeded" (a symptom, not the cause). Fixed with an explicit test.setTimeout(60s), proportionate 15s step ceilings, and a guaranteed-cleanup-time pattern (testInfo.setTimeout(testInfo.timeout + 15s) + try/catch in finally) so teardown always gets its own budget regardless of how much of the test body's time was consumed. - diary-r2-uat.spec.ts Scenario 10: a route mock that always returns an empty page defeated waitForLoaded()'s visibility race, letting a stale in-flight response from the prior "All" click satisfy the next waitForResponse() predicate before the real "Manual" request/response pair occurred, intermittently leaving a shared requests[] capture array empty. Fixed by explicitly awaiting each transition's own response and reading the request straight off the resolved Response object. - document-linking.spec.ts Scenario 7a: was collateral cancellation from Scenario 14's maxFailures:1 trip, not an independent bug. Added the same defensive guaranteed-cleanup-time pattern as cheap insurance. - playwright.config.ts: comment-only documentation, verified against the Playwright 1.61.1 dispatcher source, that maxFailures already tolerates one retry-passing (flaky) test before fail-fast cancels a shard — no config change needed. - cornerstoneContainer.ts: incidental fix for a testcontainers 12.0.4 API rename (withCopyFileToContainer -> withCopyFilesToContainer) discovered while attempting local containerized verification. Fixes #1829 Co-Authored-By: Claude dev-team-lead (Sonnet 4.6) <noreply@anthropic.com> Co-Authored-By: Claude e2e-test-engineer (Sonnet 4.5) <noreply@anthropic.com>
steilerDev
left a comment
There was a problem hiding this comment.
[product-architect] APPROVED
Reviewed as the always-on architect reviewer for this e2e-only change (per the PR Review Gate, security-engineer/ux-designer correctly don't trigger). Scope is clean: only e2e/ and agent-memory touched — zero production (server/, client/, shared/) changes, so no API-contract/schema/architecture surface is affected. I verified the technical claims rather than taking them on faith.
Test-strength preservation — confirmed, no weakened assertions:
diary-r2-uatScenario 10: the full assertion block is intact — type param present, contains ≥1 manual type, and contains zero automatic types. ReadingtypeoffmanualResponse.url()(the exact response the Manual click produced) is stronger than the old shared-mutablerequests[]array, which is precisely the race being removed. Good fix.diary-draftsScenario 14: the 45s→15s step-ceiling reductions are not a weakening — assertions fire only afterentryLoadPromiseconfirms the API response arrived, so the remaining work is a client-side React re-render that completes in well under 15s. The 45s values were budget-reservations masquerading as ceilings; tightening them is correct.
Timeout arithmetic — sound. 60s test budget + proportionate 15s "something is actually stuck" ceilings + additive testInfo.setTimeout(testInfo.timeout + 15s) cleanup extension. Because testInfo.timeout is the full original budget and the body can consume at most that before timing out, the finally block always gets ≥15s of guaranteed teardown headroom. The test.slow() correction is accurate (it triples only the overall per-test timeout, not actionTimeout/expect.timeout), as is the maxFailures note (a retry-passing "flaky" test never increments the failure count — only all-attempts-failing tests trip fail-fast).
Incidental testcontainers fix — verified genuine. withCopyFileToContainer (singular) is gone from testcontainers 12.0.4; withCopyFilesToContainer(filesToCopy: FileToCopy[]) is the correct replacement. The proxy-CA path would not compile/run without this, so it's a necessary carry, not scope creep.
Reusable patterns — worth codifying, and correctly placed. Both the guaranteed-cleanup-time pattern and per-transition response sync are genuinely reusable e2e conventions. They belong in the e2e-test-engineer's domain, and the PR already lands them in general-e2e-patterns.md. No architect-owned wiki page covers E2E test conventions, so no wiki update is required from me — agent memory is the right home. (If these keep recurring, a dedicated "E2E Conventions" wiki page could be considered later, but that's not a blocker.)
5-consecutive-runs criterion — pragmatically satisfied. A full 16-shard × 3-viewport matrix isn't run pre-merge on beta PRs, so 5 green pre-merge matrix runs aren't achievable here. More importantly, this PR does real root-cause analysis (the test.slow() misconception + the stale-in-flight-response race) rather than a retry/timeout bump — that is a materially stronger signal of a durable fix than 5 green runs of otherwise-unchanged flaky code. On-PR evidence (all 3 target tests green on shard 3 at 1–2s) + post-merge monitoring is a reasonable close-out, and the eventual beta → main promotion PR's full E2E Gates run serves as the confirming pass.
Non-blocking / informational:
- The
testInfo.setTimeoutcomment phrases the extension as "additive to the current remaining deadline" — Playwright actually sets the deadline relative to test start (start + timeout), not to remaining time. The end guarantee (≥15s cleanup headroom) is correct becausetestInfo.timeoutis the full budget, so the wording is a touch loose but not misleading. - In Scenario 14, the four 15s step ceilings sum to the 60s test budget, leaving no theoretical slack before cleanup — but realistic step durations are 1–2s and cleanup gets its own +15s extension, so this is only a pathological-worst-case concern.
No architecture, contract, or schema implications. Approving.
|
🎉 This PR is included in version 2.13.0-beta.24 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
beta -> mainpromotions on PRs release: promote beta to main (cost basis filter, diary fix, auto-itemize refactor) #1791/fix(deps): scope js-yaml override by version range to unbreak docs deploy #1828.diary-drafts.spec.tsScenario 14:test.slow()only triples the overall per-test timeout, not per-stepactionTimeout/expect.timeout— the test paired a 45s ceiling with several 30-45s step timeouts, starving thefinally-block cleanup under CI load. Replaced with an explicittest.setTimeout(60_000), proportionate 15s step ceilings, and a guaranteed-cleanup-time pattern (testInfo.setTimeout(testInfo.timeout + 15_000)+try/catch) in thefinallyblock.diary-r2-uat.spec.tsScenario 10: fixed a real race where a stale in-flight response from a prior UI transition could satisfy the nextwaitForResponse()predicate, leaving a sharedrequests[]capture array empty. Now awaits each transition's own response explicitly.document-linking.spec.tsScenario 7a: was collateralmaxFailures:1cancellation from Scenario 14, not an independent bug — added the same defensive cleanup-time pattern anyway.playwright.config.ts: comment-only documentation (verified against Playwright 1.61.1 dispatcher source) thatmaxFailuresalready tolerates one retry-passing test before fail-fast cancels a shard.cornerstoneContainer.ts: incidental fix for atestcontainers@12.0.4API rename (withCopyFileToContainer->withCopyFilesToContainer) discovered during local verification.Fixes #1829
Test plan
tsc --noEmitine2e/shows no new errors vs baseline (one pre-existing error in a touched file is incidentally fixed)Co-Authored-By: Claude dev-team-lead (Sonnet 4.6) noreply@anthropic.com