Skip to content

Cope with smaller-than-default stack size rlimit

Simon McVittie requested to merge wip/smcv/stack-size into master

  • !59 (merged) (necessary to make tests pass on my system)

  • utils: Add an equivalent of g_new0

  • utils: Allocate ld_libs on the heap

    The ld_libs data structure is pre-allocated at maximum size instead of using dynamic memory allocation, in an effort to avoid use of malloc() during critical-path interactions with basic shared library infrastructure. This makes it about 1 MiB, so if users have reduced RLIMIT_STACK from its post-1995 default of 8 MiB for whatever reason, it doesn't take many levels of recursion to exceed that limit.

    The symptom for a stack overflow is particularly un-diagnosable (instead of exiting unsuccessfully with an error message, the program will just segfault), so allocate this data structure on the heap instead, which gives us the opportunity to log "libcapsule: out of memory".

    The equivalent data structures within libcapsule itself are still allocated on the stack, to avoid introducing new memory allocation inside a critical section.

    Resolves: https://github.com/ValveSoftware/steam-runtime/issues/653

  • utils: Mark error messages to be freed automatically

Successfully tested with prlimit -s2048000 make -C ${builddir} check. Previously, a less restrictive limit prlimit -s4096000 failed with a segfault in capsule-capture-libs.

Additionally, setting a pathologically low memory limit now fails with:

$ prlimit -v8192000 ${builddir}/capsule-capture-libs --dest=$HOME/tmp libc.so.6
libcapsule: out of memory
[1]    906813 IOT instruction (core dumped)

which is a less-bad symptom than a segfault.

/cc @denittis @vivek

Merge request reports