Skip to content

Fix flakey tests where scan fails to find & label Test PR#146

Draft
rtyley wants to merge 3 commits into
mainfrom
fix-issue-145-flakey-tests-do-not-label-PR
Draft

Fix flakey tests where scan fails to find & label Test PR#146
rtyley wants to merge 3 commits into
mainfrom
fix-issue-145-flakey-tests-do-not-label-PR

Conversation

@rtyley

@rtyley rtyley commented Aug 14, 2025

Copy link
Copy Markdown
Member

Fixes #145 - the issue was caused by the slightly variable results returned by the GitHub API.

In all the tests in FunctionalSpec, we create several test fixtures with the GitHub API, each created to be used once and thrown away:

  • A brand-new GitHub repository, populated with whatever Git history we require
  • A brand new PR in the repository, which we also merge

Before continuing on to the next step, which would be having Prout scan the repo for recently-merged PRs, we perform several checks to ensure that the GitHub API confirms the status of the merged PR:

eventually {
whenReady(repo.pullRequests.get(pr.number)) {
_.mergeable.value mustBe true
}
}
eventually {
whenReady(pr.merge(MergePullRequest())) { _.merged must be(true) }
}
val mergedPR = eventually {
whenReady(repo.pullRequests.get(pr.number)) {
pr => pr.merged_by mustBe defined ;
logger.info("Created and merged PR: "+pr.html_url)
pr }
}

However, looking a the differing logs produced by the failure & success cases in #145, it became apparent that even though we've checked the GitHub API.

@rtyley rtyley force-pushed the fix-issue-145-flakey-tests-do-not-label-PR branch 10 times, most recently from abfe768 to 4f1f9a8 Compare August 18, 2025 08:31
# Conflicts:
#	build.sbt

# Conflicts:
#	build.sbt
@rtyley rtyley force-pushed the fix-issue-145-flakey-tests-do-not-label-PR branch from 1a02529 to 2e6649d Compare September 2, 2025 14:24
@rtyley rtyley changed the title Fix flakey tests which fail when scan does not find & label test PR Fix flakey tests where scan fails to find & label test PR Oct 10, 2025
@rtyley rtyley changed the title Fix flakey tests where scan fails to find & label test PR Fix flakey tests where scan fails to find & label Test PR Oct 10, 2025
@rtyley rtyley force-pushed the fix-issue-145-flakey-tests-do-not-label-PR branch from 7e1ddf1 to bfe4db0 Compare November 7, 2025 16:16
rtyley added a commit that referenced this pull request Apr 8, 2026
As detailed in this PR:

* rtyley/play-git-hub#18

...there are multiple issues affecting the flakiness of Prout's integration tests,
and fixing them all involved some pretty big changes to `play-git-hub`, released with
v10 of that library:

* https://github.com/rtyley/play-git-hub/releases/tag/v10.0.0

Unfortunately updating Prout to adapt to that new version of `play-git-hub` is a big job,
especially the migration from `Future` to `IO` - the PR to do that isn't yet completed:

* #146

...in the meantime, Prout's integration tests are failing frequently:

* #178 (comment)

This change disables Prout's integration tests - we'll just have
to double-check after deploy that Prout is still working!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flakey tests: merged PR does not get labelled Pending-on-PROD

1 participant