Skip to content

Add support for comparing by ELF verdefs, symbols

Simon McVittie requested to merge smcv/libcapsule:cmp-by-elf into master

This is an alternative to !23 (closed), which compares libraries by their ELF verdefs (versioned-symbol versions) and/or by their dynamic symbols. Includes !37 (merged) and !38 (merged), so please review those first (although it could be rebased to not).

While testing !23 (closed), we realised it was too disruptive: it is common for libraries to have private symbols that are exported, either for use by utilities from the same source package, for use by unit tests, or (particularly in older libraries) accidentally. libcapsule cannot tell the difference between these private symbols and the public symbols that it is meant to be looking at, which means it might sort libraries incorrectly. In particularly, when a library's maintainers clean up its ABI to drop private symbols, if they do not introduce new public symbols at the same time, that library will appear to be strictly older than a version that is actually newer, which could result in us using an older version that does not implement the newer semantics required by a dependent library.

Less commonly, but similarly, it is possible for a library to have private verdefs that change release-by-release, to discourage third-party code from referencing private symbols (for example, libdbus does this).

The best solution to both of these seems to be to have library-specific knowledge: which comparator to use for which library, and eventually, which symbols and which verdefs are considered private. This branch makes the comparator configurable, but does not yet do the same for public/private verdefs/symbols.

Probably best reviewed commit-by-commit.

/cc @denittis @vivek


  • utils: Add fallback definitions for alignof, offsetof, static_assert

    This assumes either a C11 compiler, or as a fallback, a tolerably new version of gcc.

  • utils: Add and test ptr_list_free_to_array

    Inspired by g_ptr_array_free(., FALSE) and g_bytes_unref_to_array(), this lets us convert a ptr_list into a raw array suitable for use with qsort() and bsearch().

  • utils: Move library_cmp_by_name to its own translation unit

    This will let us unit-test it more easily. While I'm moving it, re-indent it in libcapsule's coding style (4 rather than 2 space indents).

  • tests: Add a unit test for library_cmp_by_name

  • utils: Add functions to compare symbols and versions with libelf

    When two libraries have the same numeric tail we were not able to reliably determine which one was the newer. In particular, many distributions install libgcc_s.so.1 as a regular file, rather than a symlink to a versioned name, so library_cmp_by_name() can't work.

    These functions let us also check the library's version-definitions and the individual symbols, to make a more nuanced decision.

    Implementation originally by Ludovico de Nittis, adapted by Simon McVittie to fit the same signature as library_cmp_by_name() so that we can call the comparison functions via function pointers, to set up different comparison weights for each library if necessary. This version also includes Simon's changes to ignore uninteresting symbols for the purposes of library comparison, with a list of uninteresting symbols that are part of various architectures' ABIs, taken from dpkg-gensymbols.

  • tests: Test comparing by symbols, versions

  • capture-libs: Expand flags into an options struct

    We can pass other state through this, such as the comparators to be used to compare libraries.

  • utils: Allow selecting library comparators from a string

    This is a piece of necessary infrastructure for exposing this as a command-line option, or even as per-library metadata. We were originally going to do the equivalent of "versions,name,symbols" unconditionally, but it looks as though that could break more than it fixes, so let's be a bit more cautious.

  • tests: Test various library comparator strings

  • capture-libs: Make library comparison configurable on the command-line

    This is primarily useful for testing and experimenting. Using versions by default in preference to filenames (--compare-by="versions,name") looks like it might be viable, but is a destabilising change that we should test more before considering a change of defaults.

    Meanwhile, counting symbols as a fallback (--compare-by="...,symbols") does not look as safe as we had hoped, because if a library maintainer has cleaned up their ABI by hiding private symbols without adding any new symbols, we will sort libraries in exactly the wrong order - and in reality, that seems to be what has happened in several libraries, for example libX11.so.6 and libXfixes.so.3.

  • tests: Test configurable library comparison in capture-libs

    Instead of asserting that the default behaviour is as desired, we now assert that --compare-by=versions,name,symbols behaves the way it ought to, and that the default behaviour is unchanged. It looks as though applying --compare-by=versions,name,symbols indiscriminately could break more than it fixes.

  • utils: Make individual library comparison functions private

  • utils: Move strstarts() here

  • utils: Add library knowledge, loaded from a .desktop-style file

    This will let us select the comparator to use for individual libraries.

  • tests: Test loading library-specific knowledge

  • capture-libs: Add --library-knowledge option

    This gives capsule-capture-libs a source of library-specific knowledge. For example, if we know that:

    * libgcc_s.so.1 is installed with an unhelpful name, but it uses
      versioned symbols the way you'd hope;
    * libdbus-1.so.3 is installed with a helpful libtool-style name,
      but has private symbols that defeat our current simplistic
      comparisons

    then we can express that as:

      [Library libgcc_s.so.1]
      CompareBy=versions;symbols;
    
      [Library libdbus-1.so.3]
      CompareBy=name;

    A runtime that contains a known set of libraries would be a good place to put library-specific knowledge about those libraries. For example, the maintainers of the Steam Runtime know what libraries it contains, and are well-placed to compare those libraries with their counterparts in mainstream distributions.

  • tests: Assert that capsule-capture-libs --library-knowledge works

Edited by Simon McVittie

Merge request reports