libcapsule merge requestshttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests2020-09-08T17:45:01Zhttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/39Add public/private knowledge about symbol versions and symbols2020-09-08T17:45:01ZLudovico de NittisAdd public/private knowledge about symbol versions and symbols* utils: Add public/private knowledge about symbol versions and symbols
We are now able to list the symbol versions and/or symbols that are
known to be public/private.
This is especially useful for comparing libraries that...* utils: Add public/private knowledge about symbol versions and symbols
We are now able to list the symbol versions and/or symbols that are
known to be public/private.
This is especially useful for comparing libraries that removed symbol
versions, or symbols, from one release to another. As this already
happened multiple times in the past, like for example with
`libdrm_nouveau.so.2` or `libedit.so.2`.
* tests: Add a verdef and symbols public/private knowledge test
/cc @smcv @vivekhttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/38tests: Work around a binutils behaviour change2020-07-07T16:44:32ZSimon McVittietests: Work around a binutils behaviour changeSee https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964457
/cc @denittis @vivekSee https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964457
/cc @denittis @vivekhttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/37Add CI with AddressSanitizer and UndefinedBehaviourSanitizer2020-07-07T15:21:36ZSimon McVittieAdd CI with AddressSanitizer and UndefinedBehaviourSanitizerIn addition to being helpful for QA, this codifies what we do and don't expect to work on developer systems.
* build: Add SANITIZE_CFLAGS build variable
When building with things like AddressSanitizer, we want to apply
inst...In addition to being helpful for QA, this codifies what we do and don't expect to work on developer systems.
* build: Add SANITIZE_CFLAGS build variable
When building with things like AddressSanitizer, we want to apply
instrumentation to the libcapsule library itself, the libcapsule
tools, and the unit tests. However, we do not want to instrument the
tiny libraries that are used as test fixtures, because giving them a
dependency on libasan.so.5 would confuse libcapsule when testing them.
* ci: Run CI with UndefinedBehaviourSanitizer and AddressSanitizer
/cc @denittis @vivekhttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/36Add support for comparing by ELF verdefs, symbols2020-07-08T12:49:56ZSimon McVittieAdd support for comparing by ELF verdefs, symbolsThis is an alternative to !23, which compares libraries by their ELF verdefs (versioned-symbol versions) and/or by their dynamic symbols. Includes !37 and !38, so please review those first (although it could be rebased to not).
While ...This is an alternative to !23, which compares libraries by their ELF verdefs (versioned-symbol versions) and/or by their dynamic symbols. Includes !37 and !38, so please review those first (although it could be rebased to not).
While testing !23, we realised it was too disruptive: it is common for libraries to have private symbols that are exported, either for use by utilities from the same source package, for use by unit tests, or (particularly in older libraries) accidentally. libcapsule cannot tell the difference between these private symbols and the public symbols that it is meant to be looking at, which means it might sort libraries incorrectly. In particularly, when a library's maintainers clean up its ABI to drop private symbols, if they do not introduce new public symbols at the same time, that library will appear to be strictly older than a version that is actually newer, which could result in us using an older version that does not implement the newer semantics required by a dependent library.
Less commonly, but similarly, it is possible for a library to have private verdefs that change release-by-release, to discourage third-party code from referencing private symbols (for example, libdbus does this).
The best solution to both of these seems to be to have library-specific knowledge: which comparator to use for which library, and eventually, which symbols and which verdefs are considered private. This branch makes the comparator configurable, but does not yet do the same for public/private verdefs/symbols.
Probably best reviewed commit-by-commit.
/cc @denittis @vivek
---
* utils: Add fallback definitions for alignof, offsetof, static_assert
This assumes either a C11 compiler, or as a fallback, a tolerably new
version of gcc.
* utils: Add and test ptr_list_free_to_array
Inspired by g_ptr_array_free(., FALSE) and g_bytes_unref_to_array(),
this lets us convert a ptr_list into a raw array suitable for use
with qsort() and bsearch().
* utils: Move library_cmp_by_name to its own translation unit
This will let us unit-test it more easily. While I'm moving it,
re-indent it in libcapsule's coding style (4 rather than 2 space
indents).
* tests: Add a unit test for library_cmp_by_name
* utils: Add functions to compare symbols and versions with libelf
When two libraries have the same numeric tail we were not able to reliably
determine which one was the newer. In particular, many distributions
install libgcc_s.so.1 as a regular file, rather than a symlink to a
versioned name, so library_cmp_by_name() can't work.
These functions let us also check the library's version-definitions and
the individual symbols, to make a more nuanced decision.
Implementation originally by Ludovico de Nittis, adapted by Simon
McVittie to fit the same signature as library_cmp_by_name() so that
we can call the comparison functions via function pointers, to set up
different comparison weights for each library if necessary. This version
also includes Simon's changes to ignore uninteresting symbols for the
purposes of library comparison, with a list of uninteresting symbols
that are part of various architectures' ABIs, taken from dpkg-gensymbols.
* tests: Test comparing by symbols, versions
* capture-libs: Expand flags into an options struct
We can pass other state through this, such as the comparators to be
used to compare libraries.
* utils: Allow selecting library comparators from a string
This is a piece of necessary infrastructure for exposing this as a
command-line option, or even as per-library metadata. We were originally
going to do the equivalent of "versions,name,symbols" unconditionally,
but it looks as though that could break more than it fixes, so let's
be a bit more cautious.
* tests: Test various library comparator strings
* capture-libs: Make library comparison configurable on the command-line
This is primarily useful for testing and experimenting. Using versions
by default in preference to filenames (--compare-by="versions,name") looks
like it might be viable, but is a destabilising change that we should
test more before considering a change of defaults.
Meanwhile, counting symbols as a fallback (--compare-by="...,symbols")
does not look as safe as we had hoped, because if a library maintainer
has cleaned up their ABI by hiding private symbols without adding any
new symbols, we will sort libraries in exactly the wrong order - and
in reality, that seems to be what has happened in several libraries,
for example libX11.so.6 and libXfixes.so.3.
* tests: Test configurable library comparison in capture-libs
Instead of asserting that the default behaviour is as desired, we now
assert that --compare-by=versions,name,symbols behaves the way it ought
to, and that the default behaviour is unchanged. It looks as though
applying --compare-by=versions,name,symbols indiscriminately could break
more than it fixes.
* utils: Make individual library comparison functions private
* utils: Move strstarts() here
* utils: Add library knowledge, loaded from a .desktop-style file
This will let us select the comparator to use for individual libraries.
* tests: Test loading library-specific knowledge
* capture-libs: Add --library-knowledge option
This gives capsule-capture-libs a source of library-specific knowledge.
For example, if we know that:
* libgcc_s.so.1 is installed with an unhelpful name, but it uses
versioned symbols the way you'd hope;
* libdbus-1.so.3 is installed with a helpful libtool-style name,
but has private symbols that defeat our current simplistic
comparisons
then we can express that as:
[Library libgcc_s.so.1]
CompareBy=versions;symbols;
[Library libdbus-1.so.3]
CompareBy=name;
A runtime that contains a known set of libraries would be a good place
to put library-specific knowledge about those libraries. For example,
the maintainers of the Steam Runtime know what libraries it contains,
and are well-placed to compare those libraries with their counterparts
in mainstream distributions.
* tests: Assert that capsule-capture-libs --library-knowledge workshttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/35ld-libs: Always clear ldlibs->needed entry if ld_lib_open() fails2020-06-12T15:33:04ZSimon McVittield-libs: Always clear ldlibs->needed entry if ld_lib_open() failsThis is what was documented to happen.
Previously, we did not clear the entry if we failed to open the
library fd, or if we succeeded but the library was "unacceptable"
(wrong ELF class or machine tag). Normally this results in a minor
...This is what was documented to happen.
Previously, we did not clear the entry if we failed to open the
library fd, or if we succeeded but the library was "unacceptable"
(wrong ELF class or machine tag). Normally this results in a minor
memory leak, and a fd leak if the library is "unacceptable".
However, when called from search_ldcache_cb(), it's particularly
important that we do this, because search_ldcache() uses the state
of the fd field - valid fd or not - to check whether ld_lib_open()
succeeded.
One practical symptom is that if your container has an x86_64
libfoo.so.0 that compares newer than the provider's libfoo.so.0,
and does not have an i386 libfoo.so.0, then capsule-capture-libs
would unexpectedly not capture the i386 libfoo.so.0 from the provider
either.https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/34capture-libs: Allow patterns to be read from a file2020-06-08T15:45:36ZSimon McVittiecapture-libs: Allow patterns to be read from a fileThe pressure-vessel tool used in the Steam Runtime has an increasingly
long list of graphics libraries which might be better as a file
than on the command-line.
---
Based on !33, please review that first.
/cc @vivek @denittisThe pressure-vessel tool used in the Steam Runtime has an increasingly
long list of graphics libraries which might be better as a file
than on the command-line.
---
Based on !33, please review that first.
/cc @vivek @denittishttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/33tests: Cope with newer versions of GNU make2020-06-08T15:43:38ZSimon McVittietests: Cope with newer versions of GNU makeThe file descriptor pointing to the jobserver is not passed through
the test script, so passing through a MAKEFLAGS environment variable
that refers to it is just going to confuse us.
/cc @vivek @denittisThe file descriptor pointing to the jobserver is not passed through
the test script, so passing through a MAKEFLAGS environment variable
that refers to it is just going to confuse us.
/cc @vivek @denittishttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/32tests: Make older linkers happy with one of our test libraries2020-03-13T10:29:10ZSimon McVittietests: Make older linkers happy with one of our test librariesThe linker used in Steam Runtime 1 'scout', which is from several years
ago, doesn't seem to like a symbol being made an alias for a versioned
symbol of the same name. Rename the symbol so that only the versioned
aliases get the same bas...The linker used in Steam Runtime 1 'scout', which is from several years
ago, doesn't seem to like a symbol being made an alias for a versioned
symbol of the same name. Rename the symbol so that only the versioned
aliases get the same basename.
---
/cc @denittis @vivekSimon McVittieSimon McVittiehttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/30CI: Codify how to build on Arch Linux2020-03-12T18:32:48ZSimon McVittieCI: Codify how to build on Arch LinuxSome libcapsule users and contributors are using Arch Linux or Manjaro
rather than a Debian derivative.
Many of the tests will be skipped on Gitlab-CI because they need a
working bubblewrap, which isn't allowed inside unprivileged Docke...Some libcapsule users and contributors are using Arch Linux or Manjaro
rather than a Debian derivative.
Many of the tests will be skipped on Gitlab-CI because they need a
working bubblewrap, which isn't allowed inside unprivileged Docker;
but this provides "executable documentation" for how to do a build
and test.
/cc @denittisSimon McVittieSimon McVittiehttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/29Test library choice2020-03-10T17:01:26ZSimon McVittieTest library choice* Add (failing) test coverage for !23
This demonstrates the behaviour we want from !23.
---
Also includes !27 and !28. Configurations tested:
* Debian unstable, 32- and 64-bit
* Debian unstable, `--disable-shared`, 32- and 64...* Add (failing) test coverage for !23
This demonstrates the behaviour we want from !23.
---
Also includes !27 and !28. Configurations tested:
* Debian unstable, 32- and 64-bit
* Debian unstable, `--disable-shared`, 32- and 64-bit
* Debian unstable, `--disable-library`, 32- and 64-bit
* Debian unstable, clang
* Debian unstable, Undefined Behaviour Sanitizer
* Debian unstable, Address Sanitizer (with `--disable-library` because it doesn't understand `dlmopen()`, and ignoring leaks if any with `LSAN_OPTIONS=exitcode=0`)
* Arch
* Arch, `--disable-shared`
* Arch, `--disable-library`
... which I think should include something sufficiently similar to basically everyone's development environment, except for Steam Runtime v1 'scout', which I'm currently testing.https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/28build/tests: Miscellaneous fixes2020-01-16T19:48:22ZSimon McVittiebuild/tests: Miscellaneous fixes* !27
* tests/capture-libs: Use run_ok instead of system to run rm -fr
Arch Linux's perl complains that system() is an undefined function.
I'm not sure why, because it's a standard library facility (and
unlike C system...* !27
* tests/capture-libs: Use run_ok instead of system to run rm -fr
Arch Linux's perl complains that system() is an undefined function.
I'm not sure why, because it's a standard library facility (and
unlike C system(), it takes a list of arguments rather than using
the shell), but in any case it's easy to avoid.
* Distinguish between enabling the library and enabling shared libraries
Some tests for capsule-capture-libs will need to compile shared
libraries, even if libcapsule itself is not built.https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/27capsule-malloc: Remove duplicate typedef for mstate2020-01-16T15:24:06ZSimon McVittiecapsule-malloc: Remove duplicate typedef for mstateI occasionally compile libcapsule with clang for better diagnostics.
Redefining typedefs is a C11 feature, which clang 8 won't allow in
C99 mode.
/cc @denittisI occasionally compile libcapsule with clang for better diagnostics.
Redefining typedefs is a C11 feature, which clang 8 won't allow in
C99 mode.
/cc @denittishttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/26tests/capture-libs: Tolerate path to ld.so not being fully resolved2020-01-15T18:28:04ZSimon McVittietests/capture-libs: Tolerate path to ld.so not being fully resolved--resolve-ld.so would ideally be the equivalent of realpath or
readlink -f, but in fact it only has special handling for symlinks in
the last position in the pathname (like AT_SYMLINK_NOFOLLOW and
O_NOFOLLOW), leading to different res...--resolve-ld.so would ideally be the equivalent of realpath or
readlink -f, but in fact it only has special handling for symlinks in
the last position in the pathname (like AT_SYMLINK_NOFOLLOW and
O_NOFOLLOW), leading to different results on at least Arch Linux.
The pressure-vessel utility for which this option was added does not
actually use it any more, so fixing this is not a high priority right now.https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/25tests/version: Fix Debianisms2020-01-15T17:33:15ZSimon McVittietests/version: Fix DebianismsThis test asserted that libz.so.1 was in `/lib/*-linux-gnu*` or
`/usr/lib/*-linux-gnu*`, but on non-Debian operating systems it could
equally well be in `/usr/lib32`, `/usr/lib`, `/usr/lib64` or even `/usr/libx32`.
---
Should fix...This test asserted that libz.so.1 was in `/lib/*-linux-gnu*` or
`/usr/lib/*-linux-gnu*`, but on non-Debian operating systems it could
equally well be in `/usr/lib32`, `/usr/lib`, `/usr/lib64` or even `/usr/libx32`.
---
Should fix a test failure reported by @denittis on Arch Linux. @denittis, please check?https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/22ci: Use smaller CI images2019-11-14T19:04:41ZSimon McVittieci: Use smaller CI imagesThis is a small efficiency improvement, but the real purpose is to trigger CI so we can make sure it passes :-)This is a small efficiency improvement, but the real purpose is to trigger CI so we can make sure it passes :-)Simon McVittieSimon McVittiehttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/21Add more ld.so implementations2019-11-11T19:54:35ZSimon McVittieAdd more ld.so implementationslibcapsule only supports x86, but for completeness we have a list of non-x86 ld.so implementations, in order to be one step closer to portability. This MR expands and corrects that list a bit.
* capture-libs: Fix path to hppa, m68k, pow...libcapsule only supports x86, but for completeness we have a list of non-x86 ld.so implementations, in order to be one step closer to portability. This MR expands and corrects that list a bit.
* capture-libs: Fix path to hppa, m68k, powerpc, s390 ld.so.1
This was probably copy/paste damage from some other architecture.
According to https://sourceware.org/glibc/wiki/ABIList these
architectures are 32-bit (or technically 31-bit for s390) and install
their ld.so in /lib.
* capture-libs: Add ld.so for aarch64_be-linux-gnu
We have no plans to support big-endian 64-bit ARM, but since we have a
reasonably comprehensive list here, we might as well include it.
Reference: <https://wiki.debian.org/Multiarch/Tuples>,
<https://sourceware.org/glibc/wiki/ABIList#aarch64>
* capture-libs: Add commented-out ld.so for some more obscure architectures
I don't know the right predefined macros to detect these, so they're
just comments for possible future reference.
/cc @vivek @denittisVivek Das MohapatraVivek Das Mohapatrahttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/20build: Check that the compiler and --host are consistent2019-10-09T11:51:13ZSimon McVittiebuild: Check that the compiler and --host are consistentlibcapsule's use with biarch containers like the Steam Runtime will
frequently make it necessary to compile it for both x86_64 and i386.
On recent Debian-derived OSs this is OK, because the toolchain is
provided as a complete set of cro...libcapsule's use with biarch containers like the Steam Runtime will
frequently make it necessary to compile it for both x86_64 and i386.
On recent Debian-derived OSs this is OK, because the toolchain is
provided as a complete set of cross-compiler-style prefixed tools like
i686-linux-gnu-gcc; but some OSs, like Arch Linux and very old versions
of Debian, rely on 'gcc -m32' for their biarch support. This makes it
very easy to do
./configure --build=x86_64-linux-gnu --host=i686-linux-gnu
and accidentally produce x86_64 binaries, because there is no
i686-linux-gnu-gcc. Give the user a hint towards the correct invocation
in this case, which is:
./configure --build=x86_64-linux-gnu --host=i686-linux-gnu CC='gcc -m32'
I've implemented this as a reusable macro, in case we want to add it to
other projects that are likely to be cross-compiled by inexperienced
cross-compiler users.
Signed-off-by: Simon McVittie <smcv@collabora.com>
---
I think this would have prevented some weird behaviour that @denittis experienced yesterday, which we tracked down to his `i386-linux-gnu-capsule-capture-libs` mistakenly being an x86_64 binary.Vivek Das MohapatraVivek Das Mohapatrahttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/19capture-libs: Compare symbols and versions to determine order2019-11-20T11:58:12ZLudovico de Nittiscapture-libs: Compare symbols and versions to determine orderWhen two libraries have the same numeric tail we were not able to
reliably determine which one was the newer.
Now we also check the libraries definitions and the symbols to make a
more weighted decision.
The order of comparison i...When two libraries have the same numeric tail we were not able to
reliably determine which one was the newer.
Now we also check the libraries definitions and the symbols to make a
more weighted decision.
The order of comparison is currently:
compare version definitions > compare numeric tail > one has strictly
more symbols
If even after these three checks we are still not sure which one to
choose, we pick the provider as the default.
~~This is still a WIP. There are a few issue to resolve and do a general cleanup and testing.~~
~~I decided to excluded all the symbols that started with an underscore. They should be just for internal use and it seems to be common for them to disappear/change from version to version.~~https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/18capture-libs: Treat unversioned libraries as in indeterminate order2019-09-26T15:22:50ZSimon McVittiecapture-libs: Treat unversioned libraries as in indeterminate order* capture-libs: Clarify by introducing more temporaries
* capture-libs: Factor out library_cmp_by_name()
* capture-libs: Treat unversioned libraries as in indeterminate order
On at least Debian, Ubuntu and Manjaro, libgc...* capture-libs: Clarify by introducing more temporaries
* capture-libs: Factor out library_cmp_by_name()
* capture-libs: Treat unversioned libraries as in indeterminate order
On at least Debian, Ubuntu and Manjaro, libgcc_s.so.1 is a regular
file, not a symlink to a versioned name (libgcc_s.so.1.2.3) like most
shared libraries.
However, on Fedora 30 it is a symlink to a versioned name like
libgcc_s-9-20190827.so.1. The name used in Fedora happens to be less
than libgcc_s.so.1 in strverscmp() order, causing capsule-capture-libs
to prefer the Debian/Manjaro libgcc_s.so.1, even if the container is
older. This can cause problems if an old Debian-derived container like the
Steam Runtime is used on a newer Fedora host, with host graphics drivers
imported by using capsule-capture-libs: the container's libgcc_s.so.1
does not satisfy the versioned symbol requirements of the graphics driver,
causing loading to fail.
Treat a regular file libgcc_s.so.1 as indeterminate order (or "equal"),
so that we fall back to the default behaviour, which is currently to
take the host version of the library in the case of a tie.
This is a stopgap solution: ideally we would consider the versioned
symbols exported by both libraries, and take whichever one has a superset
of the version definitions exported by the other, if there is a strict
superset in either direction (in the case of libgcc, in fact there is).Simon McVittieSimon McVittiehttps://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/17utils: Add --help, --version and GNU argument parsing to all tools2019-07-25T13:33:44ZSimon McVittieutils: Add --help, --version and GNU argument parsing to all toolsThey now all take named options (including the -- pseudo-option) for
future expansion, and fail with an error if given too many positional
parameters.
When libcapsule tools are bundled with a larger project, this should
make it easier t...They now all take named options (including the -- pseudo-option) for
future expansion, and fail with an error if given too many positional
parameters.
When libcapsule tools are bundled with a larger project, this should
make it easier to check which ones they are.Vivek Das MohapatraVivek Das Mohapatra