Detect a seccomp-blocked set_thread_area so wibo runs under AWS Lambda

[JackPriceBurns]

Picks up the thread from #113 — running wibo on AWS Lambda, the GC/2.0 mwcceppc dies with SIGSYS during thread setup. It's setThreadArea64: it makes its set_thread_area call via int 0x80, and Lambda's seccomp filter blocks the compat syscall path. The signal lands before the existing modify_ldt fallback can run, since that only triggers when set_thread_area returns an error.

I started from the SIGSYS-handler idea in the #113 review (catch the trap, rewrite the return to -ENOSYS, let the normal fallback take over). It's neat, but it doesn't actually work on Lambda: the filter there uses a KILL action, not SECCOMP_RET_TRAP, so the SIGSYS is uncatchable and a handler never runs. I confirmed it with a small probe — install an SA_SIGINFO SIGSYS handler, fire int 0x80/set_thread_area, and the process is killed outright (the handler's si_code/si_arch checks never get a chance).

So rather than catch the signal, this detects whether set_thread_area is usable by trying it once in a forked child. If the child gets killed by the filter (or the syscall fails), the parent treats it as unavailable and uses modify_ldt. Forking isolates the trap: a KILL action only takes down the child's process group, so the parent survives either way, and it works for both KILL and TRAP filters.

Tested on Lambda (x86_64): mwcceppc -version and real compiles work and produce correct PPC objects, no env vars or config. The setup log shows the fall-through:

setup_linux: set_thread_area blocked by seccomp, using LDT
setup_linux: Using FS selector 0x17

On a normal machine it's one extra fork at startup (cached, so at most once) and otherwise unchanged. If you'd rather not fork on the common path, I can gate it behind a /proc/self/status Seccomp check so it only probes when a filter is actually installed — happy to add that.

Scope note: this only covers the x86_64 compat path (the int 0x80 in setThreadArea64); the 32-bit build's native set_thread_area is untouched.

[encounter]

I suggested a different approach here: #113 (comment)
But the original author never updated their PR with my suggestions. Feel free to try it

[encounter]

Thanks, I like the fork approach. Could you measure how much impact it has on startup (if any?)

[simonlindholm]

If fork overhead turns out to be measurable, then I wonder how vfork compares