fix: serialize platform groups + report-based device pass/fail#122
Merged
Conversation
Two independent failure modes seen running the full suite across 3 Android + iOS sim + physical iPhone: 1. iOS sim XCUITest driver bring-up (xcodebuild test-without-building on TEST_RUNNER_PORT) loses its port race under the host load of several concurrent Android Maestro processes, so every iOS flow fails with 'Unable to set permissions ...: Failed to connect to /127.0.0.1:<port>'. The iOS and Android groups previously ran via Promise.all. Add concurrentPlatforms (default false): run the groups sequentially, iOS first on an idle machine. In isolation the iOS suite runs green. 2. A device that ran every flow green but whose Maestro process exited non-zero in post-suite DebugLogStore.finalizeRun (NoSuchFileException zipping ~/Library/Logs/maestro, or the idle-kill watchdog firing during finalize) was counted as a failed device. Add devicePassed(): a device passes when exit==0 OR its JUnit report shows tests>0 and zero failures/errors. Used in the summary, headline tally, and live row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
xcodebuild test-without-buildingon the runner port) loses its port race under the host load of several concurrent Android Maestro processes → every iOS flow fails withFailed to connect to /127.0.0.1:<port>. AddconcurrentPlatforms(defaultfalse): run the iOS and Android groups sequentially (iOS first, on an idle machine) instead ofPromise.all. Opt in to overlap on a beefy machine.DebugLogStore.finalizeRun(NoSuchFileException zipping~/Library/Logs/maestro, or the idle-kill watchdog firing during finalize) was wrongly counted as failed. AdddevicePassed(): pass when exit==0 OR the JUnit report shows tests>0 and zero failures/errors. Used in summary, headline tally, and live row.Test plan
deno check+deno lintcleandeno testgreen — 70 passed (added 5devicePassedcases + 1concurrentPlatformsdefault)