Skip to content
  • Joel Brobecker's avatar
    gmp-utils: protect gdb_mpz exports against out-of-range values · 63c457b9
    Joel Brobecker authored
    The gdb_mpz class currently provides a couple of methods which
    essentially export an mpz_t value into either a buffer, or an integral
    type. The export is based on using the mpz_export function which
    we discovered can be a bit treacherous if used without caution.
    
    In particular, the initial motivation for this patch was to catch
    situations where the mpz_t value was so large that it would not fit
    in the destination area. mpz_export does not know the size of
    the buffer, and therefore can happily write past the end of our buffer.
    
    While designing a solution to the above problem, I also discovered
    that we also needed to be careful when exporting signed numbers.
    In particular, numbers which are larger than the maximum value
    for a given signed type size, but no so large as to fit in the
    *unsigned* version with the same size, would end up being exported
    incorrectly. This is related to the fact that mpz_export ignores
    the sign of the value being exportd, and assumes an unsigned export.
    Thus, for such large values, the appears as if mpz_export is able
    to fit our value into our buffer, but in fact, it does not.
    
    Also, I noticed that gdb_mpz::write wasn't taking its unsigned_p
    parameter, which was a hole.
    
    For all these reasons, a new low-level private method called
    "safe_export" has been added to class gdb_mpz, whose goal is
    to perform all necessary checks and manipulations for a safe
    and correct export. As a bonus, this method allows us to factorize
    the handling of negative value exports.
    
    The gdb_mpz::as_integer and gdb_mpz::write methods are then simplified
    to take advantage of this new safe_export method.
    
    gdb/ChangeLog:
    
            * gmp-utils.h (gdb_mpz::safe_export): New private method.
            (gdb_mpz::as_integer): Reimplement using gdb_mpz::safe_export.
            * gmp-utils.c (gdb_mpz::write): Rewrite using gdb_mpz::safe_export.
            (gdb_mpz::safe_export): New method.
            * unittests/gmp-utils-selftests .c (gdb_mpz_as_integer):
            Update function description.
            (check_as_integer_raises_out_of_range_error): New function.
            (gdb_mpz_as_integer_out_of_range): New function.
            (_initialize_gmp_utils_selftests): Register
            gdb_mpz_as_integer_out_of_range as a selftest.
    63c457b9
To find the state of this project's repository at the time of any of these versions, check out the tags.