Description of changes:

This patch adds AVX-512 support for RSA 2k, 3k and 4k signing. It is built around the use of AVX512_IFMA within the (Almost) Montgomery Multiplication implementation that comprises the modular exponentiation part of the RSA algorithm. It is ported from the OpenSSL patch.

When running the provided speed tests, the following contains the results with and without this patch:

There is currently not support for 8k, so no change there. However, this could be a follow on if there is interest in that.

Call-outs:

This patch is primarily additive modulo a small logic change that occurs here, where, previously, the calls to mod_montgomery and BN_mod_exp_mont_consttime were interleaved. The intermediate value of r1 is needed for the first exponentiation call; in order to make this possible when doing parallel exponentiations, we create a new BIGNUM on the context (r2).

Testing:

I added coverage for the fuzzer and borrowed a couple of test cases from the existing mod_exp tests to hit the new BN_mod_exp_mont_consttime_x2 function. I'm more than happy to pull out more cases from those tests, or whatever else is suggested here, just let me know!

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.