Free-threading support

[crusaderky]

This is a meta-issue for what Dask needs in order to declare compatibility with Python 3.14t:

• The handling of warnings is substantially different in free-threading interpreters.
  • dask: (Propagate contextvars to worker threads; catch warnings in 3.14t dask#12224)
  • distributed: WIP
• contextvars are propagated automatically when spawning threads. distributed could use a PR to simplify its handling.
• dask/dask CI: Pandas in 3.14t CI dask#12284
• dask/distributed CI: WIP 3.14t support distributed#9194 tests are failing and need investigating. I do not know at this point if they highlight any actual issue.
• dask/distributed CI: we could use some more aggressive multithreading tests for a few common uses cases:
  • different threads run in parallel in the client process, each with their own Client, sharing the same Scheduler
  • different threads run in parallel in the client process, sharing a single Client
  • different threads wait on the same Future (this is a variant of the previous point)
• pandas: Need Pandas >=3.0.1
• msgpack: it contains a C extension that does not release the GIL. The extension has been audited and it's been found not to contain any race conditions as long as you don't modify the input data while it's being serialized, which is fine given Dask's model. However, the maintainer has proven hostile to external contributions and so we are stuck waiting. There are two workarounds:
  1. run python with PYTHON_GIL=0. This has the disadvantage of silencing any other package with the same issue.
  2. manually install a pure-python variant of msgpack; this has the disadvantage of reducing performance:
     MSGPACK_PUREPYTHON=1 pip install msgpack --no-binary msgpack --no-cache --force-reinstall -v
• brotli: waiting for release >=1.2.1 (Support for free-threaded Python google/brotli#1241). Optional dependency of urllib3. It is pulled in by the distributed conda-forge recipe. Workarounds:
  1. use pip to install dask and distributed
  2. force-uninstall the brotli conda package
• cytoolz: The package was not properly audited AFAIK, but it should be ok thanks to its stateless nature, as long as you don't do obviously dangerous stuff like having multiple threads draw from the same generator.
• investigate if it's possible to tweak pip and conda-forge dependencies for free-threading only, and if so set much more stringent minimum versions.
• performance regression testing at scale vs. 3.14 with GIL
• documentation and user announcement

The above are the prerequisites to advertise a good user experience to final users - e.g. it will be at worst like the GIL-enabled version, minus some % slower due to known bottlenecks that are actively being ironed out upstream.

In addition, real world users will be blocked by key missing optional packages; notably:

• zarr (blocked by its dependency numcodecs: ENH: Free-Threading (No-GIL) Support for Python 3.13+ zarr-developers/numcodecs#749)
• h5py (for xarray's NetCDF engine="h5netcdf")
• s3fs (in conda, it pulls in brotli through urllib3; see above for workarounds)
• tiledb
• jupyterlab (you can run jupyterlab in GIL-enabled Python and then attach a 3.14t kernel, but it's fiddly)

After the above, there is ample margin for performance tweaking:

• pickling/unpickling runs on a special thread pool with a single thread. With free-threading, you could have as many threads as you have CPUs.
• network I/O runs on a single thread; it could be moved to one thread per peer.
• spilling is single-threaded, blocks the worker state machine and networking, and has always been very painful
• the only thing that must remain single-threaded is the state machine, but I expect it not to be a bottleneck in and by itself.
• more in general, there is a lot of profiling needed as it becomes possible to have workers with a huge number of threads - which was previously unadvisable even at the default chunk size (128 MiB). My latest benchmarks at scale (on Coiled, circa 2022) showed severe performance degradation already when moving from 4 threads per worker to 8, given the same number of total CPUs on the cluster.