fix: remove Dask client.shutdown() in build_renewable_profiles#2131
Open
thogin wants to merge 1 commit intoPyPSA:masterfrom
Open
fix: remove Dask client.shutdown() in build_renewable_profiles#2131thogin wants to merge 1 commit intoPyPSA:masterfrom
thogin wants to merge 1 commit intoPyPSA:masterfrom
Conversation
The explicit client.shutdown() call can fail with a TimeoutError when Dask worker processes don't exit within the nanny timeout on shared filesystems (e.g. BeeGFS), converting a completed run into a false Snakemake failure. Both output files are written before the shutdown call, so the failure only affects cleanup. The other Dask-using scripts in the repo (build_hac_features, build_line_rating, build_daily_heat_demand) rely on process exit for cleanup and do not call client.shutdown().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Removes the explicit
client.shutdown()call at the end ofbuild_renewable_profiles.py. This call can fail with aTimeoutErrorwhen Dask worker processes don't exit within the nanny timeout on shared filesystems (e.g. BeeGFS), converting a completed run into a false Snakemake failure.Both output files (
.ncprofile and.geojsonclass regions) are written before the shutdown call. The other Dask-using scripts in the repo (build_hac_features,build_line_rating,build_daily_heat_demand) do not callclient.shutdown()and rely on process exit for cleanup — this change alignsbuild_renewable_profileswith that pattern.Observed on a SLURM/BeeGFS cluster: 3 false failures out of ~130 runs (2.3%) over 4 weeks, all during the Dask teardown phase after successful computation and output file writing.
Checklist
doc/release_notes.rst.