Python should be built with "gcc -fprofile-update=atomic" for PGO to fix random GCC crashes #145801
Description
On Fedora Rawhide, we started to get GCC crashes ("internal compiler error: Segmentation fault") in January when building Python 3.15 on AArch64 with PGO. The crash occurs randomly in the second PGO step, when building Python the second time using the profile data (-fprofile-use uses generated .gcda files).
Example on crash:
gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -mbranch-protection=standard -fexceptions -mbranch-protection=standard -fexceptions -mbranch-protection=standard -fexceptions -O3 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -D_Py_TIER2=3 -D_Py_JIT -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv -O3 -fprofile-use -fprofile-correction -I/builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Include/internal -I/builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I/builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Include -fPIC -DPy_BUILD_CORE_BUILTIN -c /builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Modules/_opcode.c -o Modules/_opcode.o
*** WARNING *** there are active plugins, do not report this as a bug unless you can reproduce it without enabling any plugins.
Event | Plugins
PLUGIN_FINISH_UNIT | annobin: Generate final annotations
PLUGIN_START_UNIT | annobin: Generate global annotations
PLUGIN_ALL_PASSES_START | annobin: Generate per-function annotations
PLUGIN_ALL_PASSES_END | annobin: Register per-function end symbols
during IPA pass: profile
/builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Modules/_opcode.c: In function ‘_opcode_has_arg’:
/builddir/build/BUILD/python3.15-3.15.0_a7-build/Python-3.15.0a7/Modules/_opcode.c:449:1: internal compiler error: Segmentation fault
According to Jakub Jelinek who works on GCC, one explanation of this crash can be that the .gcda file is corrupted and building Python with -fprofile-update=atomic should fix that. -fprofile-update=atomic uses atomic operations to update profile information:
If ‘atomic’ is selected, then the profile information is updated using atomic operations on a best-effort basis. Ideally, the profile information is updated through atomic operations in hardware. If the target platform does not support the required atomic operations in hardware, however, libatomic is available, then the profile information is updated through calls to libatomic. If the target platform neither supports the required atomic operations in hardware nor libatomic, then the profile information is not atomically updated and a warning is issued. In this case, the obtained profiling information may be corrupt for multi-threaded applications.
For performance reasons, if 64-bit counters are used for the profiling information and the target platform only supports 32-bit atomic operations in hardware, then the performance critical profiling updates are done using two 32-bit atomic operations for each counter update. If a signal interrupts these two operations updating a counter, then the profiling information may be in an inconsistent state.
When Miro Hrončok rebuilt Python 3.15 with -fprofile-update=atomic 10 times, he got a success rate of 100% (10/10), whereas there is a failure rate of 30% without this option.
GCC uses -fprofile-update=prefer-atomic when -pthread option is used, but (at least on Fedora) -pthread is not used. So GCC uses -fprofile-update=prefer-single by default.