The call RSA_private_decrypt((int)n, p, p, rsa, RSA_NO_PADDING); in ssl/s3_srvr.c which first appeared with OpenSSL 1.0.2m does not work in conjunction with the CryptoAPI ENGINE (capi). The effectively called function capi_rsa_priv_dec in engines/e_capi.c requires the padding parameter being set to RSA_PKCS1_PADDING. It is now no longer possible to implement a SSL/TLS service (well, unless we provide access to the server certificate's private key through other means than CryptoAPI, but that's sometimes not desirable. The "capi" engine's whole purpose is (or was?) to resolve the demand for making Windows' crypto-infrastructure available to OpenSSL, isn't/wasn't it ?

The affected commit that broke the feature is c63a5ea and also references
5b8fa43 . It appears as if this patch was quite important in terms of security. Simply reverting it would probably not be a good option....