Refactor: Lazily load or eliminate heavy dependencies (scipy, numba, h5py, astropy, cosmopy) to fix import overhead#140
Merged
Conversation
added 3 commits
May 29, 2026 16:20
…ing. Made cosmopy lazily loaded to improve import time loading. Changed constatns to be hard coded to improve import loading time.
…ss it is being run in MPI based on environment variable check or if mpi4py has been imported elsewhere. Updated holodeck/utils.py so that it only imports scipy in the functions declared within utils that actually use scipy. Same for h5py. Then made a lazy loader for numba so that it is only imported when needed. This avoids a penalty for importing holodeck, which automatically imports utils. I need to run a few tests to make sure this has not broken anything.
astrolamb
approved these changes
Jun 1, 2026
Collaborator
There was a problem hiding this comment.
Hey @kayhangultekin this looks great to me. I'm also seeing a similar speed-up on import. There's just some clean-up required for this PR.
I think we can also drop the requirement for python 3.9, which is the only version currently failing tests
…up logger import statements.
Collaborator
|
@kayhangultekin it looks like we have an assertion issue with the ubuntu tests... macos is ok though |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR resolves the critical package initialization bottleneck by completely decoupling heavy scientific packages (
scipy,numba,h5py,astropy, andcosmopy) from the global top-level import sequence.Key Changes
cosmopyby encapsulating it in a lazy caching function. This completely eliminates the significant overhead of setting up the default cosmology framework on package boot.holodeck/constants.pyto use hardcoded physical constant values rather than pulling fromastropy.constants. Becauseconstants.pyis loaded directly in__init__.py, this completely immunizes the base package initialization from the astropy constants registry overhead.scipy.stats.norm.cdf(1)call with its exactfloat64constant (0.8413447460685429), eliminating a globalscipyload inutils.py.scipy.interpolate(forPchipInterpolator),scipy.special(forjn), andh5pyimports directly into the specific function scopes where they are executed._LazyNJIT) around_gw_ecc_func. This completely bypasses loading the heavy LLVM compiler machinery at boot time, delaying the import until the function is actually called during execution.rto the docstring on line 1073 to resolve a Python 3.12SyntaxWarningregarding an invalid escape sequence (\l).Performance Impact
Using
time python -c "import holodeck", package startup overhead dropped by over 95% on subsequent runs (2600% if Trump is doing the math), dropping total wall-clock time from ~2.6 seconds down to around 0.1 seconds.Side-by-Side Comparison: