i686-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libatomic.so.1

[cimandef]

Let me jot down what I've found so far, for building OpenSSL v3.4¹ into Cargo with rust-lang CI.

• Similar to sfackler/rust-openssl#2163 and openssl/openssl#23376, the dist-i686-linux build job from rust-lang/rust CI produces the cargo executable that dynamically links to libatomic.so.1. The ldd output looks like

  ldd obj/dist-i686-linux/build/i686-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-gate.so.1 (0xf7fb6000)
        libdl.so.2 => /lib/libdl.so.2 (0xf583d000)
        libatomic.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf5822000)
        librt.so.1 => /lib/librt.so.1 (0xf5819000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf57fb000)
        libm.so.6 => /lib/libm.so.6 (0xf573e000)
        libc.so.6 => /lib/libc.so.6 (0xf55b0000)
        /lib/ld-linux.so.2 (0xf7fb8000)
  

  On some distributions there is no libatomic available, so it failed to
• Setting -DBROKEN_CLANG_ATOMICS in openssl-src build didn't help.
• Removing -latomic (cargo:rustc-link-lib=atomic) from rust-openssl didn't help.
• In contrast, cargo from the dist-x86_64-linux job doesn't dynamically link to libatomic at all. The ldd output is like

  ldd obj/dist-x86_64-linux/build/x86_64-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-vdso.so.1 (0x00007ffe9d249000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f250d26d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f250d057000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f250ce4f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f250cc31000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f250c8f1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f250c544000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f250f507000)
  

• Switching CC to gcc eliminates the dynamic links. The ldd output looks like

  ldd obj/dist-i686-linux/build/i686-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-gate.so.1 (0xf7f79000)
        libdl.so.2 => /lib/libdl.so.2 (0xf574f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf5734000)
        librt.so.1 => /lib/librt.so.1 (0xf572b000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf570d000)
        libm.so.6 => /lib/libm.so.6 (0xf5650000)
        libc.so.6 => /lib/libc.so.6 (0xf54c2000)
        /lib/ld-linux.so.2 (0xf7f7b000)
  

The potential reason is that we set CC in the dist-i686-linux job image, so it builds a cargo linking to libatomic.so instead of statically linkage. And clang doesn't provide built-in atomic lib, according to its doc. It has also be discussed in llvm/llvm-project#73361 and mstorsjo/llvm-mingw#384. I've tried adding -DCOMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF² and -DCOMPILER_RT_USE_ATOMIC_LIBRARY=ON when bootstrapping the first clang in the dist-i686-linudex job. Nothing changed.

It has been more than one year since OpenSSL v1.1.1 EOL (2023-09-11), we should take action on this, either try harder upgrading to OpenSSL v3, or find an alternative like rustls or aws-lc. See also how rustup struggles to upgrade OpenSSL rust-lang/rustup#3790.

Originally posted by @weihanglo in #13546

Footnotes

1. openssl@0.10.68, openssl-sys@0.9.104, and openssl-src@300.4.0+3.4.0 respectively ↩

2. requires gcc10 to fix preprocessor errors and -DCOMPILER_RT_HAS_WBUILTIN_DECLARATION_MISMATCH_FLAG=OFF to surpress some errors. ↩