WIP XML.jl v0.4: Rewrite of internals, streaming tokenizer, XPath support, and bug fixes

[joshday]

Summary of Changes

I revived an old rewrite I had halfway finished with the help of Claude Code. It produced some good results!

• Major rewrite of XML.jl's internals that addresses many open issues
• Self-contained src/XMLTokenizer.jl module for speedy tokenization
• Node{T} now parameterized by the string storage type, enabling quick reads via SubString or StringViews.jl
• StringViews extension — XML.mmap("file.xml", LazyNode) for memory-mapped parsing of very large files
• XPath support — xpath(node, path) with a practical subset of XPath 1.0
• Greatly expanded test suite — 243 libxml2 test cases, pugixml and libexpat compatibility tests, W3C conformance tests

Downstream

@TimG1964 you are likely the most impacted with these changes. The Downstream.yml action does indicate a failure in XLSX.jl tests related to Raw no longer existing. I'd appreciate your review here! I'm happy to submit a PR for a fix in XLSX.jl so that its ready to go before this gets merged.

Addressed Issues

• Closes XML character references are not unescaped/escaped #17 — XML character references are now unescaped/escaped
• Closes XPath support #30 — XPath support
• Closes Inconsistent type for attributes where nodes have no attributes #33 — Inconsistent type for attributes where nodes have no attributes
• Closes Simple XML.write followed by XML.parse fails #35 — Simple XML.write followed by XML.parse no longer fails
• Closes get not defined to match getindex #50 — get defined to match getindex
• Closes Question: Why the choice not to escape & to & ? #52 — escape now unconditionally escapes '&'
• Closes Incorrect unescape result. #53 — Incorrect unescape result (double-unescaping)

Benchmarks: See benchmarks/compare.jl

Here (SS) refers to using SubString{String} as storage type.

julia --project=. benchmarks/compare.jl
============================================================
  XML.jl Benchmark Comparison
  Current (dev) vs v0.3.8
============================================================

Running dev benchmarks... done
Setting up v0.3.8 worktree... done
Running v0.3.8 benchmarks... done

------------------------------------------------------------

  Parse (small)
          v0.3.8      0.114 ms
             dev     0.0335 ms  (70.6% faster)

  Parse (small, SS)
          v0.3.8           n/a
             dev     0.0285 ms

  Parse (medium)
          v0.3.8   634.7153 ms
             dev   161.0888 ms  (74.6% faster)

  Parse (medium, SS)
          v0.3.8           n/a
             dev   151.3025 ms

  Write (small)
          v0.3.8     0.0227 ms
             dev     0.0176 ms  (22.4% faster)

  Write (medium)
          v0.3.8   118.1504 ms
             dev     77.619 ms  (34.3% faster)

  Read file (medium)
          v0.3.8   645.5785 ms
             dev   170.8398 ms  (73.5% faster)

  Collect tags (small)
          v0.3.8     0.0005 ms
             dev     0.0006 ms  (10.3% slower)

  Collect tags (medium)
          v0.3.8    21.0988 ms
             dev    11.1532 ms  (47.1% faster)

============================================================

[TimG1964]

Hey @joshday . I've only had a very superficial look so far but it looks great. Thanks!

In terms of impact on XLSX.jl, I think it looks significant. It isn't just Raw. Since @nhz2 first suggested using Raw, I've known it was internal and therefore subject to change. On first inspection, I think the rework involved should be manageable.

More of a challenge will be the removal of prev and next, which are currently exported functions. I rely on these for fundamental elements of XLSX.jl like the sheetrow and tablerow iterators, and for reading and writing the XML files from/to the zip archive .xlsx file.

These obviously aren't insuperable, but will likely need a bit of time while I get to grips with xpath and tokenizer. Optimistic me thinks the new functionality will simplify the code of XLSX.jl, but I usually find things are considerably harder than I first imagine! I'll feedback more when I've had a bit more of a go at getting XLSX.jl working.

Thanks,

Tim

[TimG1964]

I'm happy to submit a PR for a fix in XLSX.jl so that its ready to go before this gets merged.

Hi @joshday, I've been a bit distracted recently by transferring XLSX.jl to JuliaData and subsequently making a v0.11 release, but my attention will be back on this again after the Easter break. I have to say I'd welcome any PR you could make on XLSX.jl to help facilitate this upgrade.

Thanks!

[mathieu17g]

Hi @joshday — the v0.4 rewrite looks good; the tokenizer architecture reads faster on eager mode.

I'm currently evaluating the v0.4 upgrade on FastKML.jl following your comment on my PR #58. The eager-path improvements are substantial on real-world KML: ×2–2.6 speedup and 37–69% memory reduction across four reference files (5k to 163k Placemarks), versus v0.3.8 + #58 + #59 eager.

There is a trade-off I wanted to surface before #54 is merged: v0.4 removed the linear-traversal API on LazyNode — next / prev (and the next! / prev! zero-alloc variants from PR #59) — replaced by eachchildnode / children, which allocate per child. That leaves no API equivalent for the zero-alloc lazy walk class that PR #59 provided under v0.3.8.

On real-world KML files with non-trivial structure, the v0.4 lazy path regresses by ×1.4 to ×2.6 vs v0.3.8 + #58 + #59 lazy across the full 4-file reference set. Two concrete cases:

• USGS WRS-2 tiles — 28k Polygon Placemarks (each a 5-vertex LinearRing) in a single flat top-level layer. Regresses ×2.6.
• EPA Facility Registry — 163k Point Placemarks across 19k nested folders. Regresses ×2.3.

On those same files, the v0.3.8+PRs lazy path was actually faster than v0.4 eager too, so a strict migration loses the previously optimal path. Full per-file profile and methodology in the linked results doc below.

I've written up the decomposition (synthetic bench + FastKML real workloads + cost attribution + a SOTA-informed design space) as a separate design issue #61 rather than clutter this PR thread.

Full data: benchmark/results_eager_vs_lazy_3way_2026-05-11.md on the FastKML wip-xml-v0.4 branch.

Happy to refine the benchmark or prototype any direction if useful.

[mathieu17g]

Hi, @joshday. I've posted some results on #61 — a Cursor streaming primitive plus an isbits Token that takes the iterate tuple allocation-free. The cursor is additive, but making Token isbits touches this PR's core Token, so flagging it here too: I'd value your read on whether that change fits v0.4.

[joshday]

Hey all, sincere apologies for my lack of communication here. I need to step back from finishing this PR and maintaining XML in the foreseeable future. I wish I had wrapped up this PR before getting into a busy season of life and I'm more than happy to share my thoughts and general vision for v0.4, but I should no longer be in the critical path of development and/or decision making.

I'm self-employed and (for better or worse) have been successful enough that I no longer have time to allocate for things that aren't (1) paying the bills or (2) spending time with my family. I'd love to get back to this someday! There's a sick twisted part of me that genuinely likes working on awful XML edge cases 😅.

@mathieu17g @TimG1964 My recommendation would be to transfer XML.jl to JuliaData. This will need to be initiated by someone at JuliaHub, but I think they'll onboard with it.

[TimG1964]

Thanks for the update and very sorry to hear you are "moving on". I wish you success in your business.

Will you be facilitating the transfer to JuliaData? I really hope they are able to take XML.jl on and can find a worthy successor.

Before you go, would you have time to review and possibly merge any of the pending PRs into a final v0.3.9. I fear it may be a while before a v0.4 can be finalized following the transfer of ownership.

Thanks!

[mathieu17g]

+1 on @TimG1964's v0.3.9. The five open PRs that fit a 0.3.x patch are all CI-green on current main and mutually conflict-free. Suggested merge order, #64 last since it carries the version bump: #60 → #56 → #58 → #59 → #64, then register on the final merge commit.

Optionally, two small regression tests, verified against the PR branches and ready to fold in:

# #56 — prev must cross a CDATA section (the prev call itself crashes on v0.3.8)
doc = parse("<r><a>x</a><![CDATA[hello]]><b>y</b></r>", LazyNode)
b = children(children(doc)[1])[3]              # the <b> element
p = nothing
@test (p = XML.prev(b)) isa LazyNode           # asserts the call does not throw
@test XML.nodetype(p) == XML.CData
@test XML.value(p) == "hello"

# #60 — escape on a SubString (MethodError on v0.3.8)
@test XML.escape(SubString("a&b<c>", 1)) == "a&amp;b&lt;c&gt;"

[joshday]

I don't have merge permissions here