[i386/i686] Fatal Python error: Segmentation fault while running the tests

[mr-c]

Crash Report

Repeated crashes while running the mypy 1.19.1 test suite on the Debian "i386" ci nodes using Python 3.14.3 (but not Python 3.13.12). This is causing us a problem promoting a newer build of the mypy package for the next stable release of Debian. orjson is not installed (and the problem persists even if it is).

Traceback

Current thread 0xf7ba3700 [python3.14] (most recent call first):                                                                                                                                                                       
  File "/usr/lib/python3/dist-packages/mypyc/test/testutil.py", line 123 in build_ir_for_single_file2
  File "/usr/lib/python3/dist-packages/mypyc/test/testutil.py", line 98 in build_ir_for_single_file
  File "/usr/lib/python3/dist-packages/mypyc/test/test_irbuild.py", line 83 in run_case
  File "/usr/lib/python3/dist-packages/mypy/test/data.py", line 323 in runtest
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 245 in <lambda>
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 353 in from_call
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 244 in call_and_report
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
  File "/usr/lib/python3/dist-packages/xdist/remote.py", line 227 in run_one_test
  File "/usr/lib/python3/dist-packages/xdist/remote.py", line 206 in pytest_runtestloop
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 372 in _main
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 318 in wrap_session
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
  File "/usr/lib/python3/dist-packages/xdist/remote.py", line 427 in <module>
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 1291 in executetask
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 341 in run
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 1273 in serve
  File "/usr/lib/python3/dist-packages/execnet/gateway_base.py", line 1806 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

Current thread's C stack trace (most recent call first):
  Binary file "/usr/bin/python3.14", at _Py_DumpStack+0x36 [0x80adcf4]
  Binary file "/usr/bin/python3.14" [0x81262f9]
  Binary file "linux-gate.so.1", at __kernel_sigreturn+0x0 [0xf7f5f5a0]
  Binary file "/usr/bin/python3.14" [0x81a8042]
  Binary file "/usr/bin/python3.14", at PyErr_Format+0xaa [0x81a6b6a]
  Binary file "/usr/bin/python3.14", at _PyNumber_Index+0x5c [0x81aefbc]
  Binary file "/usr/bin/python3.14", at PyLong_AsInt+0x45 [0x81d5e65]
  Binary file "/usr/bin/python3.14" [0x82b6dd7]
  Binary file "/usr/bin/python3.14", at _PyObject_MakeTpCall+0x23d [0x814f19d]
  Binary file "/usr/bin/python3.14", at PyObject_CallFunction+0x145 [0x814a7d5]
  Binary file "/usr/bin/python3.14" [0x82b6299]
  Binary file "/usr/bin/python3.14", at PyObject_Vectorcall+0x33 [0x81547d3]
  Binary file "/usr/lib/python3/dist-packages/4c842c94c09923bae9e4__mypyc.cpython-314-i386-linux-gnu.so", at CPyDef_mypy___build___add_catch_all_gitignore+0x138 [0xf3c946e8]
  Binary file "/usr/lib/python3/dist-packages/4c842c94c09923bae9e4__mypyc.cpython-314-i386-linux-gnu.so", at CPyDef_mypy___build____build+0x110e [0xf3cd87ce]
  Binary file "/usr/lib/python3/dist-packages/4c842c94c09923bae9e4__mypyc.cpython-314-i386-linux-gnu.so", at CPyDef_mypy___build___build+0x316 [0xf3cda096]
  Binary file "/usr/lib/python3/dist-packages/4c842c94c09923bae9e4__mypyc.cpython-314-i386-linux-gnu.so", at CPyPy_mypy___build___build+0x191 [0xf3cdace1]
  Binary file "/usr/bin/python3.14", at PyObject_Vectorcall+0x33 [0x81547d3]
  Binary file "/usr/bin/python3.14", at _PyEval_EvalFrameDefault+0x6891 [0x8170131]
  Binary file "/usr/bin/python3.14" [0x81a2449]
  Binary file "/usr/bin/python3.14" [0x81559d2]
  Binary file "/usr/bin/python3.14" [0x82352a9]
  Binary file "/usr/bin/python3.14" [0x81b9164]
  Binary file "/usr/bin/python3.14", at _PyEval_EvalFrameDefault+0x4c52 [0x816e4f2]
  Binary file "/usr/bin/python3.14" [0x81a2449]
  Binary file "/usr/bin/python3.14" [0x81559d2]
  Binary file "/usr/bin/python3.14" [0x82352a9]
  Binary file "/usr/bin/python3.14", at _PyObject_MakeTpCall+0x313 [0x814f273]
  Binary file "/usr/bin/python3.14", at PyObject_Vectorcall+0x5b [0x81547fb]
  Binary file "/usr/bin/python3.14", at _PyEval_EvalFrameDefault+0x6891 [0x8170131]
  Binary file "/usr/bin/python3.14" [0x81f9c8e]
  Binary file "/usr/bin/python3.14" [0x81b9127]
  Binary file "/usr/bin/python3.14", at _PyEval_EvalFrameDefault+0x4c52 [0x816e4f2]
  <truncated rest of calls>

To Reproduce

# on a i386/i686 system or virtual machine or docker image or chroot
apt-get install -y mypy pybuild-plugin-autopkgtest; apt-get build-dep -y mypy
git clone https://salsa.debian.org/python-team/packages/mypy.git  # to get a copy of the tests
cd mypy
export DEB_BUILD_OPTIONS=parallel=2
pybuild-autopkgtest

Your Environment

• Mypy version used: 1.19.1
• Python version used: 3.14.3 (no problems with 3.13)
• librt version: python3-librt i386 0.8.1-1
• Operating system and version: Debian "sid" (unstable)

In another error log there is a mention of test_irbuild.py::TestGenOps::irbuild-weakref.test::testWeakrefProxy
or in this log: mypyc/test/test_irbuild.py::TestGenOps::irbuild-classes.test::testGetAttr; see also this log for 'mypyc/test/test_irbuild.py::TestGenOps::irbuild-classes.test::testMypycAttrNativeClassErrors but I'm unsure if these are the real culprits

[mr-c]

I installed the dbg symbol packages (libc6-dbg python3-mypy-dbgsym python3.14-dbg python3.13-dbg) but the tracebacks didn't improve. Here's a sample capture of another failure and the full junit file

/usr/lib/python3/dist-packages/mypyc/test/test_irbuild.py:83: in run_case
    ir = build_ir_for_single_file(testcase.input, options)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3/dist-packages/mypyc/test/testutil.py:98: in build_ir_for_single_file
    return build_ir_for_single_file2(input_lines, compiler_options)[0].functions
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3/dist-packages/mypyc/test/testutil.py:123: in build_ir_for_single_file2
    result = build.build(sources=[source], options=options, alt_lib_path=test_temp_dir)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mypy/build.py:222: in build
    ???
mypy/build.py:301: in _build
    ???
mypy/build.py:2952: in dispatch
    ???
mypy/build.py:3347: in process_graph
    ???
mypy/build.py:925: in wait_for_done
    ???
mypy/build.py:3516: in process_stale_scc
    ???
mypy/build.py:2508: in write_cache
    ???
mypy/build.py:1609: in write_cache
    ???
mypy/util.py:946: in json_dumps
    ???
/usr/lib/python3.14/json/__init__.py:242: in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
/usr/lib/python3.14/json/encoder.py:202: in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/json/encoder.py:263: in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
E   MemoryError
E   when serializing dict item '.class'
E   when serializing dict item 'type'
E   when serializing dict item 'node'
E   when serializing dict item 'sort'
E   when serializing dict item 'names'
E   when serializing dict item 'node'
E   when serializing dict item 'list'
E   when serializing dict item 'names

pytest-junit.xml

[mr-c]

On Debian, mypy is compiled using gcc version 15.2.0-15, not llvm. Maybe we'll get a more stable binary if we build with MYPYC_OPT_LEVEL=2 instead of the default of 3, so I'll try that next.

[mr-c]

Compiling with MYPYC_OPT_LEVEL=2 for i386 resulted in many tests failing with a MemoryError, culminating in a seg fault and pytest worker node crash.

Sample failure

2362s ___________________________ testAsyncWithMixedReturn ___________________________
2362s [gw0] linux -- Python 3.14.3 /usr/bin/python3.14
2362s data: /tmp/autopkgtest-lxc.rdgnz86s/downtmp/build.nKT/src/mypyc/test-data/run-async.test:751:
2362s /usr/lib/python3/dist-packages/mypyc/test/test_run.py:163: in run_case
2362s     self.run_case_inner(testcase)
2362s /usr/lib/python3/dist-packages/mypyc/test/test_run.py:179: in run_case_inner
2362s     self.run_case_step(testcase, step)
2362s /usr/lib/python3/dist-packages/mypyc/test/test_run.py:269: in run_case_step
2362s     ir, cfiles, _ = emitmodule.compile_modules_to_c(
2362s mypyc/codegen/emitmodule.py:458: in compile_modules_to_c
2362s     ???
2362s mypyc/codegen/emitmodule.py:298: in compile_modules_to_ir
2362s     ???
2362s mypyc/codegen/emitmodule.py:253: in compile_scc_to_ir
2362s     ???
2362s mypyc/transform/refcount.py:72: in insert_ref_count_opcodes
2362s     ???
2362s mypyc/analysis/dataflow.py:369: in analyze_must_defined_regs
2362s     ???
2362s mypyc/analysis/dataflow.py:571: in run_analysis
2362s     ???
2362s E   MemoryError

Full log https://ci.debian.net/packages/m/mypy/testing/i386/69627923/

Copy for posterity: mypy-69627923-log.gz

Next I'm going to turn off compiling mypy with mypyc on i386 for the Debian package

[ilevkivskyi]

Yeah, I think the problem is that mypy tests eat a lot of memory (so that they don't fit into 4GB anymore, which is maximum for 32 bit). TBH I was bothered by this for a long time, if I look at the memory graph while running tests, it constantly grows.

One possible trick to fix this may be to force gc.collect() after say every 100 tests. @JukkaL what do you think? Long term however we should try to make mypy less memory hungry (which will also help with the parallel checking).

[ilevkivskyi]

Oh yeah, and obviously we should not segfault on a memory error, but this is a separate problem.

[JukkaL]

Yeah we should try to reduce memory usage if possible.

[ilevkivskyi]

@JukkaL I assume this means you are OK with the gc.collect() hack for tests. If yes, I will try to play with this soon to see if it helps.

[JukkaL]

Yeah, we can try that.

[ilevkivskyi]

Some more context: Python 3.14 has a new GC. It is (allegedly) faster but worse (i.e. keeps more garbage for longer). It looks like it causes so many issues that it is going to be reverted in 3.14.5, see https://discuss.python.org/t/reverting-the-incremental-gc-in-python-3-14-and-3-15/107014/27