[mypyc] Speed up native-to-native imports within the same group#21101
Open
[mypyc] Speed up native-to-native imports within the same group#21101
Conversation
This comment has been minimized.
This comment has been minimized.
Member
|
@JukkaL the test failures on Windows look real. |
Collaborator
Author
|
Yeah, I'm investigating the Windows failures. |
p-sawicki
reviewed
Mar 25, 2026
mypyc/lib-rt/misc_ops.c
Outdated
Comment on lines
+1229
to
+1238
| PyObject *file = PyObject_GetAttrString(modobj, "__file__"); | ||
| if (file != NULL) { | ||
| // __file__ already set, nothing to do. | ||
| Py_DECREF(file); | ||
| return 0; | ||
| } | ||
| if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { | ||
| return -1; | ||
| } | ||
| PyErr_Clear(); |
Collaborator
There was a problem hiding this comment.
could use PyObject_GetOptionalAttrString to not have to deal with the exception here and in other new functions.
Comment on lines
+235
to
+239
| def func() -> int: | ||
| return 42 | ||
| [file driver.py] | ||
| import native | ||
| assert native.f() == 42 |
Collaborator
There was a problem hiding this comment.
test if _file is as expected?
This comment has been minimized.
This comment has been minimized.
1 similar comment
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When compiling multiple modules, mypyc generally creates one big shared library with all the code, and also tiny shim shared libraries for each compiled module so that Python import machinery can find the modules. This is inefficient at least on macOS, since each shared library that is loaded into the process seems to have a non-trivial cost, including each shim. On the first run, this cost is much higher, and the first mypy run after
pip installcan take 30s or more on macOS.This PR addresses the slow imports on macOS by adding a custom implementation of native-to-native imports within the same compilation group that avoids using the shim. We directly construct the module object, populate
sys.modules, and set an attribute in the parent package, without using Python import machinery.This speeds up a minimal mypy run (
mypy -c 'import os') on macOS by up to 10x (first cold run after installation), but even small warm runs are significantly faster. The measurements were all over the place, but at least in one measurement the minimal warm run was over 1.5x faster with these changes. Impact on Linux should be small (an earlier version of this PR was slightly faster on Linux, but didn't measure the current one). I haven't measured the impact on Windows.Some notes about the implementation:
from <...> importand try to generate a single call to import multiple names to avoid verbose IR.__init__.pyfiles when compiling mypy as a micro-optimization.I used Claude Code and Codex to implement much of the code in small increments (based on a manually written core implementation). I also iterated on the code quite significantly after the basic implementation was done.