Skip to content
  • Arnd Bergmann's avatar
    crypto: serpent - improve __serpent_setkey with UBSAN · c871c10e
    Arnd Bergmann authored
    When UBSAN is enabled, we get a very large stack frame for
    __serpent_setkey, when the register allocator ends up using more registers
    than it has, and has to spill temporary values to the stack. The code
    was originally optimized for in-order x86-32 CPU implementations using
    older compilers, but it now runs into a highly suboptimal case on all
    CPU architectures, as seen by this warning:
    
    crypto/serpent_generic.c: In function '__serpent_setkey':
    crypto/serpent_generic.c:436:1: error: the frame size of 2720 bytes is larger than 2048 bytes [-Werror=frame-larger-than=]
    
    Disabling -fsanitize=alignment would avoid that warning, presumably the
    option turns off a optimization step that is required for getting the
    register allocation right, but there is no easy way to do that on gcc-7
    (gcc-8 introduces a function attribute for this).
    
    I tried to figure out a way to modify the source code instead, and noticed
    that the two stages of the setkey() function (keyiter ...
    c871c10e