Serialize engine config in new pdsh benchmark CLI

[TomAugspurger]

Description

This updates the cudf-polars benchmarks CLI to serialize the engine configuration. This will let us see what options were actually used.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[TomAugspurger]

This needs more work. StreamingExecutor.spill_to_pinned_memory doesn't actually have any effect on pinned memory with the new frontends. That's controlled through the rapidsmpf_options["pinned_memory"] option / --pinned-memory CLI.

We need to ensure that the config_options contains, in some way, the rapidsmpf_options.

[TomAugspurger]

I've updated this to serialize config_options differently for the new StreamingEngine frontends to account for rapidsmpf_options being a level above ConfigOptions`. Now, we'll have a dict

{
   "config_options": ...,  # the old `config_options`, i.e. `dataclasses.asdict(cudf_polars.utils.config.ConfigOptions)`
   "rapidsmpf_options": # rapidsmpf.config.Options.get_strings()
}

And here are some example values of .rapidsmpf_options:

# dask
{
  "num_streaming_threads": "4"
}
# ray
{
  "num_streaming_threads": "4"
}
# spmd
{
  "spill_device_limit": "",
  "allow_overbooking_by_default": "",
  "log": "",
  "periodic_spill_check": "",
  "pinned_memory": "",
  "num_streaming_threads": "",
  "statistics": "",
  "memory_reserve_timeout": "",
  "num_streams": "",
  "unbounded_file_read_cache": ""
}

Unfortunately, I don't think rapidsmpf currently has a way to view the actual values the options resolve to without explicitly setting or using them, which is why the dask/ray examples only have the num_streaming_threads while the spmd has more. And now that I look at it, I have no idea why those are set to empty strings. I'll need to investigate that...

[TomAugspurger]

@madsbk do you have any ideas about how to handle #22365 (comment)? Specifically,

1. A way for a rapidsmpf.config.Options() object to resolve to the actual defaults used in rapidsmpf
2. Why the values in rapidsmpf_options.get_strings() are all "" for the SPMD engine?

[madsbk]

1. A way for a rapidsmpf.config.Options() object to resolve to the actual defaults used in rapidsmpf

I don’t think it is possible with the current design, where each call site parses the string value and defines its own default.

2. Why the values in rapidsmpf_options.get_strings() are all "" for the SPMD engine?

An empty string means that no one has accessed and parsed that option yet.

[TomAugspurger]

OK, I think this might be all we can do on the cudf-polars side then. I've opened rapidsai/rapidsmpf#1014 to track what I think needs to happen on the rapidsmpf side, and then hopefully we can get back to a state where all of the actual runtime options are being recorded.

For now, I'll merge this so that the non-rapidsmpf options are being recored properly.