Canonicalize tests around @safetestset (isolation, matches OrdinaryDiffEq)#158
Conversation
…ffEq) Wrap each independent Core test unit (Dense, Sparse, Symbols, Legacy, Miscellaneous, Public) in `@safetestset` so each runs in its own fresh module, giving cross-unit isolation and world-age safety, matching the canonical OrdinaryDiffEq structure. The shared `using` imports and the shared structs/helpers (CustomTag, the RuleConfig subtypes, FakeSparsityDetector, FakeColoringAlgorithm, every_ad, every_ad_with_options) that previously lived at runtests.jl top-level are moved into test/test_setup.jl, which each unit file now includes so every @safetestset module is self-contained. `using ADTypes` is kept in runtests.jl (Main): Julia qualifies a type's module in `show` based on visibility in `Base.active_module()` (Main), not the module the test runs in, so the printing assertions (`startswith(string(ad), "Auto")`) require ADTypes to be in scope in Main. Same tests, same assertions, same GROUP dispatch; only the unit wrappers and import plumbing changed. Adds SafeTestsets to the test deps. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@ChrisRackauckas what's the deal with the sequence of Claude-generated PRs changing tests and CI? Seems that's all we're getting in ADTypes lately |
|
It's the canonicalization roll out matching the new CI runners. Anant talked about it a bit ago: https://www.youtube.com/watch?v=CFzfJ7F-6Ks It started to become a major deal with the codecov v6/v7 issues, julia cache action v3 crashing runners (and effectively being a useless waste of time), etc. so most Julia repos were getting random test reds due to different interactions between runners and the random CI script changes. So I took the time to finish Anant's canonicalization, which is a pretty mechanical set of changes. But the canonicalization also found lots of little inconsistencies and issues with the current CI setup when rolled out across the mass of repos (which was the point of course), so the CI scripts needed a little bit in order to be hardened to all of the different possible behaviors. There will be another pass that happens hopefully later today that rips out the bespoke runtests.jl and instead uses the new SciMLTesting.jl repo in order to further reduce the repeated code. That will happen right on this PR though, no merging of this until that is done, and with the rollout we'll need to see if there's any edge case missed in SciMLTesting.jl. While this repo is relatively simple so most of the PRs passed right through it without hitting any troubles, it still will help a lot that it's canonicalized so that for example, the next time CodeCov decides to do a breaking v8, that gets fixed in 1 PR instead of 192, which was the main motivation to finally finish this project. |
|
superseded by the v1.2 folder conversion on sciml-testing-rollout (#159) |
3 participants
Summary
Canonicalizes the test suite to run each independent Core test unit in its own module via
@safetestset, matching the canonical OrdinaryDiffEq structure (cross-unit isolation + world-age safety). SafeTestsets expands@safetestset "name" begin ... endinto a fresh module, so each unit body must be self-contained.This is the test-isolation branch only. It does not touch the GROUP-dispatch ladder,
run_testsharness, or any assertion/logic (the harness migration is a separate branch).What changed
test/runtests.jl: each Core unit@testset "X" begin include("x.jl") end->@safetestset "X" begin include("x.jl") end(Dense, Sparse, Symbols, Legacy, Miscellaneous, Public). The outer@testset verbose = true "ADTypes.jl"wrapper and theGROUP == "All"/"Core"/"QA"dispatch are unchanged. QA is untouched.test/test_setup.jl(new): holds the sharedusingimports and the shared structs/helpers that previously lived atruntests.jltop-level —CustomTag,ForwardRuleConfig/ReverseRuleConfig/ForwardOrReverseRuleConfig,FakeSparsityDetector,FakeColoringAlgorithm,every_ad(),every_ad_with_options(). Each unit file nowinclude("test_setup.jl")so its@safetestsetmodule is self-contained.test/{dense,sparse,legacy,misc}.jl: addinclude("test_setup.jl")at the top (symbols.jl/public.jl already carried their ownusing ADTypes; using Test).Project.toml: addSafeTestsetsto[extras],[targets].test, and[compat]("0.1, 1").Subtlety worth flagging (load-bearing)
using ADTypesis kept atruntests.jltop level (Main) on purpose. Julia qualifies a type's module inshowbased on visibility inBase.active_module()(which isMain), not the module the test actually runs in. With the units now in fresh modules,Mainwould no longer have ADTypes in scope, sostring(AutoForwardDiff())becomes"ADTypes.AutoForwardDiff()"and thestartswith(string(ad), "Auto")assertions in the Printing block fail. Keepingusing ADTypesin Main restores the exact printing context the original tests relied on. (Caught by the local runtime gate — first run had 25 Printing failures, fixed by this.)Verification (local, Julia 1.11)
GROUP=Core:ADTypes.jl | 507 passed, 0 failed(same 507 tests as before: previously 482 pass + 25 Printing fail in an intermediate state, now all 507 pass).GROUPunset (All):507 passed.GROUP=QA:Quality Assurance | 11 passed(Aquadeps_compathappy with the new[compat] SafeTestsets).--check --diffclean on all changed/new files.Ignore until reviewed by @ChrisRackauckas.
🤖 Generated with Claude Code