Make the RAND code available from within the FIPS module#9035
Make the RAND code available from within the FIPS module#9035mattcaswell wants to merge 10 commits intoopenssl:masterfrom
Conversation
There was a problem hiding this comment.
These start/stop names look different
There was a problem hiding this comment.
Not sure I understand your point. There isn't a "start" equivalent of ossl_ctx_thread_stop
There was a problem hiding this comment.
What about ossl_init_thread_start?
There was a problem hiding this comment.
The stop equivalent of ossl_init_thread_start is ossl_init_thread_stop. There is no "start" equivalent of ossl_ctx_thread_stop.
crypto/initthread.c
Outdated
crypto/initthread.c
Outdated
There was a problem hiding this comment.
Doesnt this *hands get returned?
There was a problem hiding this comment.
I refactored this to make things a bit clearer.
crypto/initthread.c
Outdated
crypto/rand/drbg_lib.c
Outdated
There was a problem hiding this comment.
I assume all these goto's mean that CRYPTO_THREAD_cleanup_local() doesnt handle passing NULLS - looks messy
There was a problem hiding this comment.
Well, dgbl->private_drbg is not NULL if there's an error, just uninitialised - so we shouldn't attempt to use it in anyway.
There was a problem hiding this comment.
Since dgbl is allocated via zalloc, and since NULL is all-bits-zero (via the sanity test), it is initialized.
There was a problem hiding this comment.
It's the private_drbg field we are specifically concerned with there. And yes it will get set to 0 on create. However, when I said it was unitialised, I meant it has not been initialised via a call to CRYPTO_THREAD_init_local. We should not be calling CRYPTO_THREAD_cleanup_local on keys that have not been initialised in that way. We cannot know how CRYPTO_THREAD_cleanup_local would treat a key that is set to 0. It most likely would be fine but we cannot say for sure. For example in a pthreads implementation CRYPTO_THREAD_cleanup_local calls pthread_key_delete which has this description: "The pthread_key_delete() function shall delete a thread-specific data key previously returned by pthread_key_create()". It says nothing about what happens if you pass it zero data that was not created by pthread_key_create - so at best this behaviour is undefined.
|
I havent gone through the whole thing yet - but it seems that a large percentage of this PR is related to threads. Maybe the rand changes should be in a seperate PR? |
|
travis errors are valid (except for the timeout ones) |
|
I've attempt to respond to the comments so far and have also tried to fix the various travis issues. I've also rebased and force pushed. I'm going to take a look to see if I can easily split this up into smaller PRs to make it easier to review. |
|
Needs #9186 before this can progress. |
|
Please rebase then ping, I'll try to review quickly. |
370df5a to
825514f
Compare
|
I've rebased this PR. Note the feedback comments above were for commits no longer in this PR. Note this PR now also includes the commits from #9186, so that needs to go in first before I can take this out of WIP. |
In some circumstances the global data held in the "global" variable can be NULL, so we should error out in the circumstance instead of crashing.
Insert a dummy call to RAND_DRBG_bytes from inside the FIPS provider to demonstrate that it is possible to use the RAND code from inside the module. This is temporary and will be removed once real uses of the RAND code are available inside the module.
825514f to
d4c81c6
Compare
|
Finally all the dependent PRs are in and I am able to take this out of WIP! I've rebased on the latest master. Please review! |
|
Fixup commit pushed addressing the feedback, and also fixing a windows linking issue. I've not addressed the doc-nits issue because that is (presumably) also a problem on master so probably is better done in a different PR. |
|
doc-nits issue fixed in #9192 |
|
Looks good to me. I would like to know what @mspncp has to say, though. |
I started taking a look, but it might take a while to complete it, since I'm currently on vacation. |
|
Oh, sorry, forgot about that... |
crypto/hmac/hm_meth.c
Outdated
| if (!value) | ||
| return 0; | ||
| #ifndef FIPS_MODE | ||
| /* Not supported in FIPS mode due to the implict fetch of the md */ |
There was a problem hiding this comment.
I'm not sure I understand the explanation "...due to the implict fetch of the md" means. Actually, I believe this comment is incomprehensible for anybody who is not already deeply familiar with all the current replumbing and FIPS refactoring. Could this comment be a little bit more elaborated to make it more comprehensible?
Also, it contains a typo: 'implict' -> 'implicit'
There was a problem hiding this comment.
"Explicit" fetch occurs when we look up an algorithm using one of the "fetch" functions such as EVP_MD_fetch(). This finds the algorithm and associates it with an implementation from a provider. Conversely "Implicit" fetch is when we don't do that, and use a function such as "EVP_sha256()" which has no provider implementation associated with it. With an algorithm obtained in that way the "fetch" happens automatically when you try to use it - hence "implicit fetch". An important difference between these two types of EVP_MD objects is that explicitly fetched versions are dynamically allocated and must be "freed", whereas implicitly fetched ones do not. That fact makes this code more complicated if we want to convert it to use explicit fetch. Since we are unlikely to need this ctrl in FIPS_MODE I didn't bother.
I replaced this comment with a longer one to try and explain this better.
crypto/rand/drbg_hash.c
Outdated
| #ifndef FIPS_MODE | ||
| /* Any approved digest is allowed */ | ||
| md = EVP_get_digestbynid(drbg->type); | ||
| md = EVP_MD_meth_dup(EVP_get_digestbynid(drbg->type)); |
There was a problem hiding this comment.
I don't think the digest algorithm of the HASH-DRBG should be selectable via generic EVP lookup accross multiple providers. It is an integral part of the HASH-DRBG algorithm and not a replaceable part. If you take a look at the CTR-DRBG implementation, there is no such distinction between fips and generic provider: The aes-ctr algorithm is always fetched unconditionally from the provider context.
openssl/crypto/rand/drbg_ctr.c
Lines 375 to 391 in 56ce84b
The application can only choose from a fixed number of builtin implementations. The digest selection for the HASH-DRBG should work exactly the same way. In other words, drop the #if part and keep only the #else part.
openssl/crypto/rand/drbg_hash.c
Lines 315 to 321 in 56ce84b
The only difference between fips and non-fips might be that for the former the switch contains less cases.
There was a problem hiding this comment.
I don't think the digest algorithm of the HASH-DRBG should be selectable via generic EVP lookup accross multiple providers
...
The aes-ctr algorithm is always fetched unconditionally from the provider context.
I think there is some confusion here. Ignoring the FIPS provider for a moment (which is a special case), the DRBG code exists in libcrypto and is not associated with a particular provider (yet). We probably will want to make it possible to "fetch" a DRBG implementation from a provider and use that...but we're not there yet. For now its just in libcrypto. The look up across "multiple providers" will happen anyway when used, even with aes-ctr. It just looks for an implementation of (for example) "AES-128-ECB" and uses it. The default provider (which might not even be available for all we know) may or may not be used.
Now, as I said, the FIPS provider is a special case. It does not link against libcrypto and so can't use the built-in DRBG implementation. Therefore, for that provider only, we are additionally making the DRBG code available inside the FIPS provider. For that provider only, when we do a "fetch" it will only ever find algorithms provided by itself.
Anyway, aside from that confusion, your suggestion is fine and I have refactored things accordingly.
There was a problem hiding this comment.
I'd lean towards making the digest selection completely open. It provides a degree of flexibility in case something untoward happens to one of the digests. However, I'm happy enough with the current switch statement to not be too bothered (there are plenty of choices available and it avoids an explicit XOF check).
One could argue that the CTR-DRBG should have this flexibility too (ARIA and Camellia should both drop in instead of AES). Still, for some point in the future rather than now.
There was a problem hiding this comment.
I don't mind, as long as the implementation is consistent for the three DRBG implementations. Anyway, if this change is desired it would be a matter for a separate pull request.
Providers that link against libcrypto can just use OBJ_nid2sn() to look up the name of an algorithm given a NID. However that doesn't work for the FIPS provider because OBJ_nid2sn() is not available there (due to the reliance of the code on ASN.1 types). Therefore we provider a new function to do this mapping. For providers linking against libcrypto the new function just wraps OBJ_nid2sn(). For the FIPS provider it has a look up for all the NIDs known there.
…ames We use the new function ossl_prov_util_nid_to_name() to look up the algorithm and unify the FIPS_MODE and non-FIPS_MODE handling.
As per the previous commit we make the same change for DRBG HMAC and more closely align the FIPS_MODE and non FIPS_MODE implementations.
|
New commits addressing the feedback pushed. |
mspncp
left a comment
There was a problem hiding this comment.
Looks good, but some questions remain. (You can ignore the ones for @paulidale and @slontis).
BTW: Is it actually possible by now to run and check the DRBG code of the FIPS provider in the debugger (on linux)? If yes, what do I need to set it up?
crypto/hmac/hm_meth.c
Outdated
| /* | ||
| * We don't have EVP_get_digestbyname() in FIPS_MODE. That function returns | ||
| * an EVP_MD without an associated provider implementation (i.e. it is | ||
| * using "implict fetch"). We could replace it with an "explicit" fetch |
There was a problem hiding this comment.
typo: 'implict' -> 'implicit'
|
|
||
| #include "internal/providercommon.h" | ||
|
|
||
| const char *ossl_prov_util_nid_to_name(int nid); |
There was a problem hiding this comment.
I'm surprised that an internal function gets documented in the manual pages (which doesn't mean I'm against documenting it). Is it intended to be used by third party providers, too?
There was a problem hiding this comment.
@levitte has been championing the documentation of the "internal" API, i.e. stuff that ends up in our internal header files. There's quite a few man pages there already. This particular function is purely internal. There is no intention for it to be used by third party providers.
| #ifndef FIPS_MODE | ||
| case NID_blake2b512: | ||
| case NID_blake2s256: | ||
| case NID_sm3: |
There was a problem hiding this comment.
This is more a question to @paulidale and @slontis than a review comment: NIST SP 800-90Ar1 does not mention SHA3, BLAKE and SM3. Is it 'legal' nevertheless to use them?
There was a problem hiding this comment.
Section 10.1 opens by saying that the hash DRBG mechanisms have been designed to use any approved hash function. Based on this SHA3 would be legal.
BLAKE and SM3 cannot be FIPS validated.
| #ifndef FIPS_MODE | ||
| case NID_blake2b512: | ||
| case NID_blake2s256: | ||
| case NID_sm3: |
There was a problem hiding this comment.
I think it has the same answer. 10.1.2 again mentions an approved hash function.
| }; | ||
|
|
||
| int rand_crngt_get_entropy_cb(OPENSSL_CTX *ctx, | ||
| RAND_POOL *pool, |
There was a problem hiding this comment.
What is the rationale behind the changes in rand_crng_test.c?
There was a problem hiding this comment.
Is it a fixup for 4e297b7 which you made along the way?
There was a problem hiding this comment.
Effectively yes. I encountered a crash related to this code. The change is related to the crngt_get_entropy callback. That callback gets called from rand_crng_ossl_ctx_new. That function is called when we are initialising an OPENSSL_CTX with CRNG data for the first time. Unfortunately, in the original implementation, the default callback was itself trying to get CRNG data from the OPENSSL_CTX. But since we're still initialising it it went round in an an infinite recursive loop trying to initialise the OPENSSL_CTX until eventually it blew the stack. The solution to the problem was to change the callback to take the RAND_POOL to be used as a parameter instead of getting it from the OPENSSL_CTX.
crypto/rand/rand_win.c
Outdated
| } | ||
|
|
||
| # if !OPENSSL_API_1_1_0 | ||
| # if !OPENSSL_API_1_1_0 || defined(FIPS_MODE) |
There was a problem hiding this comment.
Are you sure to enable it in FIPS mode? Or did you mean
# if !OPENSSL_API_1_1_0 && !defined(FIPS_MODE)
There was a problem hiding this comment.
Yes, I did mean that!
There was a problem hiding this comment.
Hmmm... but why? The code is marked deprecated-in-1.1.0 and not called anywhere.
msp@msppc:~/src/openssl$ git grep -E -nH 'RAND_(event|screen)' pr-9035
pr-9035:crypto/rand/rand_win.c:166:int RAND_event(UINT iMsg, WPARAM wParam, LPARAM lParam)
pr-9035:crypto/rand/rand_win.c:172:void RAND_screen(void)
pr-9035:doc/man3/RAND_add.pod:5:RAND_add, RAND_poll, RAND_seed, RAND_status, RAND_event, RAND_screen,
pr-9035:doc/man3/RAND_add.pod:25: int RAND_event(UINT iMsg, WPARAM wParam, LPARAM lParam);
pr-9035:doc/man3/RAND_add.pod:26: void RAND_screen(void);
pr-9035:doc/man3/RAND_add.pod:74:RAND_event() and RAND_screen() are equivalent to RAND_poll() and exist
pr-9035:doc/man3/RAND_add.pod:84:RAND_event() returns RAND_status().
pr-9035:doc/man3/RAND_add.pod:98:RAND_event() and RAND_screen() were deprecated in OpenSSL 1.1.0 and should
pr-9035:include/openssl/rand.h:68:DEPRECATEDIN_1_1_0(void RAND_screen(void))
pr-9035:include/openssl/rand.h:69:DEPRECATEDIN_1_1_0(int RAND_event(UINT, WPARAM, LPARAM))
pr-9035:util/libcrypto.num:1331:RAND_event 1318 3_0_0 EXIST:_WIN32:FUNCTION:DEPRECATEDIN_1_1_0
pr-9035:util/libcrypto.num:1816:RAND_screen 1804 3_0_0 EXIST:_WIN32:FUNCTION:DEPRECATEDIN_1_1_0
There was a problem hiding this comment.
Is it because it is still referenced in libcrypto.num?
pr-9035:util/libcrypto.num:1331:RAND_event 1318 3_0_0 EXIST:_WIN32:FUNCTION:DEPRECATEDIN_1_1_0
pr-9035:util/libcrypto.num:1816:RAND_screen 1804 3_0_0 EXIST:_WIN32:FUNCTION:DEPRECATEDIN_1_1_0
There was a problem hiding this comment.
Sorry, I wasn't clear. What you wrote is what I meant! I've fixed it now.
There was a problem hiding this comment.
But the FIPS provider does not reuse libcrypto.num, right?
There was a problem hiding this comment.
Sorry, I wasn't clear. What you wrote is what I meant! I've fixed it now.
Ok, now the world is in order again :-)
There was a problem hiding this comment.
But the FIPS provider does not reuse libcrypto.num, right?
Right. It has its own num file which exports exactly one function:
https://github.com/openssl/openssl/blob/master/util/providers.num
With this PR, yes. Set a breakpoint on line 120 of |
|
Thanks for the instructions. I'll give it a try this evening... |
|
Fix up commit pushed addressing the feedback. |
Did you manage to do that? |
|
Yes I did, and I have some thoughts to share, which I did not find the time to write down yet. Also, I might have some questions for you. But it's nothing that would prevent this pull request from being merged, I haven't found any issues (yet). I'd still be more comfortable if @levitte would approve the provider related (non-drbg) parts of this pull request, but if you think my approval is sufficient, feel free to go ahead and merge it. |
I think @levitte is away at the moment with only occasional network access. I'll leave this open a little bit longer to see if he comments, but otherwise I will merge this later today. |
|
Pushed! Thanks! |
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
In some circumstances the global data held in the "global" variable can be NULL, so we should error out in the circumstance instead of crashing. Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
Insert a dummy call to RAND_DRBG_bytes from inside the FIPS provider to demonstrate that it is possible to use the RAND code from inside the module. This is temporary and will be removed once real uses of the RAND code are available inside the module. Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
Providers that link against libcrypto can just use OBJ_nid2sn() to look up the name of an algorithm given a NID. However that doesn't work for the FIPS provider because OBJ_nid2sn() is not available there (due to the reliance of the code on ASN.1 types). Therefore we provider a new function to do this mapping. For providers linking against libcrypto the new function just wraps OBJ_nid2sn(). For the FIPS provider it has a look up for all the NIDs known there. Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
…ames We use the new function ossl_prov_util_nid_to_name() to look up the algorithm and unify the FIPS_MODE and non-FIPS_MODE handling. Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
As per the previous commit we make the same change for DRBG HMAC and more closely align the FIPS_MODE and non FIPS_MODE implementations. Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from #9035)
This enables calls to RAND_*() functions from within the FIPS module. This is required in order to bring in support for BIGNUM, and the various asymmetric algorithms.
I've added a dummy call into the FIPS module just to demonstrate that it works. This will be removed in due course when we no longer need it.