Defer heavy imports for faster package load times by johnjasa · Pull Request #762 · NatLabRockies/H2Integrate

johnjasa · 2026-05-19T19:13:58Z

Defer heavy imports for faster package load times

Dramatically reduces import h2integrate and H2IntegrateModel import times by deferring heavy transitive dependencies (OpenMDAO, HOPP, PySAM, pyomo, scipy, matplotlib, jsonschema, etc.) to the point of first use rather than loading them at module scope.

Old timing:

===========================================================================
H2Integrate Import Timing (existing timing)
===========================================================================
  import h2integrate                                        11.36s  [OK]
  from h2integrate import H2IntegrateModel                  10.33s  [OK]
  from h2integrate import load_yaml                         10.14s  [OK]
  Load supported_models + resolve 1 model                   12.63s  [OK]
  Load supported_models + resolve ALL models                11.46s  [OK]

New timing:

===========================================================================
H2Integrate Import Timing (with just-in-time loading)
===========================================================================
  import h2integrate                                         1.77s  [OK]
  from h2integrate import H2IntegrateModel                   1.53s  [OK]
  from h2integrate import load_yaml                          1.46s  [OK]
  Load supported_models + resolve 1 model                    6.23s  [OK]
  Load supported_models + resolve ALL models                11.68s  [OK]

No behavioral changes - all deferred imports are resolved on first use and cached by Python's import system.

Section 1: Type of Contribution

Section 2: Draft PR Checklist

Open draft PR
Describe the feature that will be added
Fill out TODO list steps
Describe requested feedback from reviewers on draft PR
Complete Section 7: New Model Checklist (if applicable)

TODO:

Defer heavy imports in h2integrate_model.py
Add ruff per-file E501 ignore for supported_models.py
Verify all examples still run correctly

Type of Reviewer Feedback Requested (on Draft PR)

Structural feedback: Is the _ModelRegistry dict subclass approach acceptable, or would you prefer a different loading pattern?

Implementation feedback: Are there any methods in h2integrate_model.py where the local import openmdao.api as om pattern feels problematic?

Other feedback: N/A

Section 3: General PR Checklist

Section 4: Related Issues

N/A

Section 5: Impacted Areas of the Software

Section 5.1: New Files

N/A

Section 5.2: Modified Files

h2integrate/__init__.py
- Replaced imports with __getattr__ lazy loader to avoid triggering the full dependency tree on import h2integrate.
h2integrate/core/supported_models.py
- Introduced _ModelRegistry; values don't include the h2integrate. prefix (prepended automatically in _resolve).
h2integrate/core/h2integrate_model.py
- Moved import openmdao.api as om and several other heavy imports from module-level to local method-level imports.
pyproject.toml
- Added */supported_models.py to [tool.ruff.lint.per-file-ignores] for E501.

Section 6: Additional Supporting Information

The primary bottleneck was supported_models.py, which imported all ~120 model classes and their transitive dependencies (HOPP, PySAM, pyomo, scipy, matplotlib, etc.) at module scope. The _ModelRegistry subclass stores import paths as strings and resolves them on first dict access, so only the models actually used in a given run are imported. Python's import caching means each module is only loaded once regardless of how many methods contain local import statements.

elenya-grant

Thanks for working on this! I think these changes are good!

elenya-grant · 2026-05-20T16:19:53Z

        the same for each technology. This includes site information, project parameters,
        control strategy, and finance parameters.
        """
+        import openmdao.api as om


does the om import really slow stuff down? If its used in multiple methods then I kind of think it should be left at the top of the file instead of in each method.

It doesn't really; I was trying out a few different things and this was leftover. I removed all of these and put the import at the top, thank you!

elenya-grant · 2026-05-20T16:23:05Z

+        return new
+
+
+supported_models = _ModelRegistry(


I think this is slick! I'll be bummed that we won't be able to right click on a package object and have it open the file in VSCode anymore - but I will survive! If this reduces the import time then I'm down with it!

I agree this obfuscates things a little bit, but I do think the reduced import time is worthwhile. This might also point us towards having more standardized naming conventions for files so you can easily find them in VSCode, e.g. TidalResource could be defined in tidal_resource.py instead of the (as-is) tidal.py, and maintain this same naming throughout H2I.

Do you think that would help quickly find what you're looking for? What other solutions are possible to help here?

RHammond2 · 2026-05-20T21:26:27Z

+        return default
+
+    def values(self):
+        return [self[k] for k in super().keys()]


I'm not sure I understand why we wouldn't return list(super().values()) or not even override values in the first place. Could you provide a bit more context on this decision making?

I'm so glad we have PR reviews because this is firmly not needed, good catch! I've removed the values and items overriding.

Glad to be of service!

RHammond2 · 2026-05-20T21:27:09Z

+        return [self[k] for k in super().keys()]
+
+    def items(self):
+        return [(k, self[k]) for k in super().keys()]


Same general comment as the one I left for values.

RHammond2

I like the simplicity of this approach, and the massive improvements in import speedup. This should actually help a bit with #742 because of the sheer number of tests that import the model, which is a really nice secondary impact.

I have a couple of specific questions before I hit the approve button, and a couple of non-blocking general que

RHammond2

I really like the simplicity of this approach and the all the added speedups. This should actually help a reasonable amount with #742 because of how many times the model is imported throughout the tests.

There are a couple spots in _ModelRegistry where some docstrings would be appreciated, a couple questions about the reimplementation of dict methods, and a non-blocking general question about import structures.

johnjasa · 2026-05-21T19:22:11Z

I really like the simplicity of this approach and the all the added speedups. This should actually help a reasonable amount with #742 because of how many times the model is imported throughout the tests.

There are a couple spots in _ModelRegistry where some docstrings would be appreciated, a couple questions about the reimplementation of dict methods, and a non-blocking general question about import structures.

Thanks, Rob! This is ready for re-review, I've implemented some good changes based on your comments.

Co-authored-by: Rob Hammond <13874373+RHammond2@users.noreply.github.com>

RHammond2

Thanks for the updates, @johnjasa!

johnjasa added 2 commits May 19, 2026 12:37

Working on just-in-time importing

08d387e

Updated changelog

0e755b8

johnjasa changed the title ~~Working on just-in-time importing~~ Defer heavy imports for faster package load times May 19, 2026

elenya-grant approved these changes May 20, 2026

View reviewed changes

RHammond2 reviewed May 20, 2026

View reviewed changes

RHammond2 requested changes May 20, 2026

View reviewed changes

updating based on PR feedback

795284f

johnjasa marked this pull request as ready for review May 21, 2026 18:30

RHammond2 mentioned this pull request May 21, 2026

Feature request: Simplify import pathways for primary models and functionality #767

Open

Updates based on PR feedback

0125e34

johnjasa requested a review from RHammond2 May 21, 2026 19:21

RHammond2 reviewed May 21, 2026

View reviewed changes

Comment thread h2integrate/core/supported_models.py Outdated

Update h2integrate/core/supported_models.py

7f7de10

Co-authored-by: Rob Hammond <13874373+RHammond2@users.noreply.github.com>

RHammond2 approved these changes May 21, 2026

View reviewed changes

Merge branch 'develop' into faster_importing

4f19f35

johnjasa enabled auto-merge (squash) May 21, 2026 20:03

johnjasa merged commit 6334faa into NatLabRockies:develop May 21, 2026
12 checks passed

Conversation

johnjasa commented May 19, 2026

Defer heavy imports for faster package load times

Section 1: Type of Contribution

Section 2: Draft PR Checklist

TODO:

Type of Reviewer Feedback Requested (on Draft PR)

Section 3: General PR Checklist

Section 4: Related Issues

Section 5: Impacted Areas of the Software

Section 5.1: New Files

Section 5.2: Modified Files

Section 6: Additional Supporting Information

Uh oh!

elenya-grant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RHammond2 left a comment

Choose a reason for hiding this comment

Uh oh!

RHammond2 left a comment

Choose a reason for hiding this comment

Uh oh!

johnjasa commented May 21, 2026

Uh oh!

Uh oh!

RHammond2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants