perf: load brain plugins lazily, only when their target module is imported#3069
Draft
Pierre-Sassoulas wants to merge 4 commits into
Draft
perf: load brain plugins lazily, only when their target module is imported#3069Pierre-Sassoulas wants to merge 4 commits into
Pierre-Sassoulas wants to merge 4 commits into
Conversation
Replace the hand-maintained ``register_all_brains`` -- a 50-line import block followed by 50 ``.register()`` calls -- with two data structures: an ``_EAGER_BRAINS`` tuple and a ``_LAZY_BRAINS`` mapping from a module-name trigger to the brain modules it needs. A brain is loaded via ``importlib`` and registered by ``_load_brain``, tracked in ``_loaded_brain_names`` so it registers at most once per process. ``register_all_brains`` keeps its behaviour -- it still eagerly loads every brain -- so this commit changes nothing observable. It adds ``register_brains`` (eager brains only) and ``load_brains_for_modname``, which the next commit wires up for lazy loading.
``astroid.astroid_manager`` eagerly loaded every brain plugin (~49 modules) so their transforms were registered up front. Most of that work is wasted: ``brain_numpy_*`` does nothing for code that never imports ``numpy``, ``brain_pytest`` nothing without ``pytest``, etc. Using the brain registry, register only the two universal brains (``brain_builtin_inference`` and ``brain_type``, which target import-less builtins) at startup, and load the rest on demand. ``AstroidBuilder._post_build`` scans a freshly built module's ``Import`` / ``ImportFrom`` nodes and ``AstroidManager.ast_from_module_name`` keys on the module being resolved; each calls ``load_brains_for_modname`` so a brain is registered before the transforms that need it run. ``test_recursion_error_trapped`` depended on the brain inference predicates running on every ``Call`` node and exhausting the stack via their internal ``safe_infer`` calls. With lazy loading those predicates are not registered, so the recursion no longer fires. The test now force-loads every brain to keep verifying the trap; the trap behaviour itself is unchanged. ``import astroid`` now loads 4 brain-related modules (the package, ``helpers``, and the 2 eager brains) instead of 53.
1 task
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3069 +/- ##
==========================================
- Coverage 93.52% 93.42% -0.11%
==========================================
Files 92 92
Lines 11329 11313 -16
==========================================
- Hits 10596 10569 -27
- Misses 733 744 +11
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
``_load_brain`` early-returns for a brain that is already registered, which keeps a brain reachable through several module triggers from registering its transforms more than once. Add a direct test for that guard so the path is exercised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
astroid_managereagerly loaded every brain plugin (~49 modules) atimport astroidso their transforms were registered up front. Most ofthat work is wasted —
brain_numpy_*does nothing for code that neverimports
numpy,brain_pytestnothing withoutpytest, and so on.refactor): replace the hand-maintained
register_all_brains— a50-line import block plus 50
.register()calls — with an_EAGER_BRAINStuple and a_LAZY_BRAINSmapping from a module-nametrigger to the brain modules it needs. Brains load via
importliband register at most once per process.
brains (
brain_builtin_inferenceandbrain_type, which targetimport-less builtins) at startup, and load the rest on demand.
AstroidBuilder._post_buildscans a freshly built module'sImport/
ImportFromnodes andAstroidManager.ast_from_module_namekeyson the module being resolved; each calls
load_brains_for_modnameso a brain is registered before the transforms that need it run.
import astroidgoes from 165 to 109 loaded modules (brain modules51 → 2). Combined with #3062, 165 → 100.
test_recursion_error_trappedrelied on the brain inferencepredicates running on every
Callnode and exhausting the stack; withlazy loading those predicates aren't registered, so it now force-loads
every brain to keep verifying the trap (trap behaviour unchanged).
Test plan