Many test failures are not catched in the CI #16350
Description
- This is a bug, not a question or an enhancement.
- I've searched for similar issues and didn't find a duplicate.
- I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
- I agree to follow Nextcloud's Code of Conduct.
The problem
See PR #16329, where I systematically provoked failures in automated tests I had recently added.
None of those were catched by the CI
- Screenshot tests failed, but they are broken on master anyways due to some permission issues. And these were no ScreenshotTests, so they were rightfully not included here.
- The Drone job also did not execute them.
This is very dangerous, as it conveys a false sense of security and confidence in the code quality.
It is also quite frustrating, having put effort into stuff developers usually don't like (automating tests), to prevent regressions, and none of it would actually matter.
The cause
Unclear right now - something seems to be wrong in the test selection setup, where we differentiate between
- Unit tests
- Screenshot tests
- Drone tests
Many tests seem to be picked up in neither category