Skip to content

Fix flaky zero-allocation tests on Julia 1.12+ via AllocCheck.check_allocs#66

Merged
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-alloc-tests-checkallocs
Jun 17, 2026
Merged

Fix flaky zero-allocation tests on Julia 1.12+ via AllocCheck.check_allocs#66
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-alloc-tests-checkallocs

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Problem

The Core test group is red on main across every platform (macOS, Ubuntu, Windows) and on both Julia 1 (1.12.6) and pre (1.13.0-rc1). The failures are all in test/alloc_tests.jl:

DisjointRange - Zero Allocations: Test Failed at test/alloc_tests.jl:21
  Evaluated: 739144 == 0
DisjointRange - Zero Allocations: Test Failed at test/alloc_tests.jl:45
  Evaluated: 704 == 0
getindex/setindex! - Zero Allocations: Test Failed at test/alloc_tests.jl:100
  Evaluated: 16 == 0

Root cause

These are @allocated measurement artifacts on Julia 1.12+, not real allocations in the package. @allocated measures the allocations of a single runtime call, which conflates first-call compilation and global-scope boxing with genuine heap allocation:

  • Line 21 (dr[8]) — the second-range branch of DisjointRange getindex is never warmed up before this @allocated (only dr[3], the first-range branch, runs at line 20), so it measures ~700 KB of first-call compilation.
  • Line 45 (iteration) — the accumulator s lives in the global scope of the @allocated begin ... end block, so it is boxed (704 bytes). That is an artifact of measuring outside a function, not of the iterate methods.
  • Line 100 (A[50, 50]) — even after one warmup call, the first @allocated captured residual specialization allocations (16 bytes).

The package targeted Julia 1.10/1.11 CI when these tests were written; they passed there and only began failing as CI moved to Julia 1.12/1.13.

Fix

Rewrite the tests to use AllocCheck.check_allocs (a test dependency that was already declared but unused). check_allocs statically proves the absence of any allocating code path for the given argument types — it is immune to compilation and global-scope boxing noise and is a strictly stronger guarantee than @allocated == 0. This is not a loosening: the same intended invariant (the operation never allocates) is verified more rigorously.

Local verification

check_allocs reports 0 allocation sites for every operation, confirming the package code is genuinely allocation-free; only the test methodology was flaky. Running test/alloc_tests.jl after the change:

Julia Result (before) Result (after)
1.10.11 16/16 pass 11/11 pass
1.12.6 13 pass, 3 fail 11/11 pass
1.13.0-rc1 13 pass, 3 fail 11/11 pass

Full Core group via the real SciMLTesting.run_tests() harness on 1.12.6:

Test Summary:       | Pass  Total   Time
Core/alloc_tests.jl |   11     11  23.0s
Test Summary:      | Pass  Broken  Total  Time
Core/core_tests.jl |   28       2     30  9.6s   (2 pre-existing @test_broken, unchanged)

Runic format check on the changed file passes (exit 0).


Please ignore until reviewed by @ChrisRackauckas.

🤖 Generated with Claude Code

The allocation tests in test/alloc_tests.jl used `@allocated`, which measures
allocations of a single runtime call. On Julia 1.12+ this surfaced false
positives that did not reflect any real allocation in the package:

- `dr[8]` (DisjointRange second-range branch) was never warmed up before its
  `@allocated`, so it measured ~700k bytes of first-call compilation.
- The iteration test accumulated into a global-scope variable inside the
  `@allocated begin ... end` block, so the accumulator was boxed (704 bytes) —
  an artifact of measuring in non-function scope, not of the iterate methods.
- The first `@allocated A[50, 50]` after a single warmup still captured residual
  specialization allocations (16 bytes).

`AllocCheck.check_allocs` (a test dependency already declared but unused)
statically proves the absence of any allocating code path for the given argument
types. It is immune to compilation and global-scope boxing noise and is a
strictly stronger guarantee than `@allocated == 0`. `check_allocs` reports zero
allocation sites for every operation on Julia 1.10, 1.12.6, and 1.13.0-rc1,
confirming the package code is genuinely allocation-free; only the test
methodology was flaky.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 17, 2026 10:38
@ChrisRackauckas ChrisRackauckas merged commit 9c2cce6 into SciML:main Jun 17, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants