Add basic EVP_KEYMGMT API and libcrypto <-> provider interface by levitte · Pull Request #9312 · openssl/openssl

levitte · 2019-07-04T22:49:01Z

The idea with the key management "operation" is to support the
following set of functionality:

Key domain parameter generation
Key domain parameter import
Key domain parameter export
Key generation
Key import
Key export
Key loading (HSM / hidden key support)

With that set of function, we can support handling domain parameters
on one provider, key handling on another, and key usage on a third,
with transparent export / import of applicable data. Of course, if a
provider doesn't offer export / import functionality, then all
operations surrounding a key must be performed with the same
provider.

This method also avoids having to do anything special with legacy
assignment of libcrypto key structures, i.e. EVP_PKEY_assign_RSA().
They will simply be used as keys to be exported from whenever they are
used with provider based operations.

This change only adds the EVP_KEYMGMT API and the libcrypto <->
provider interface. Further changes will integrate them into existing
libcrypto functionality.

levitte · 2019-07-04T22:50:34Z

Do note that this is a very early draft. The primary purpose for submitting this PR this early is to expose they ideas and get early feedback on the line of thoughts.

I hope that this will be useful for changes such as #9266

levitte · 2019-07-04T22:52:58Z

Key loading (HSM / hidden key support)

This is highly experimental for now, but does provide the necessary support for getting keys through provider OSSL_STORE backends. More on that in a separate PR.

paulidale · 2019-07-05T08:17:51Z

I really want to review this in more detail. I didn't get to it today and am too tired to think too hard this evening.

petrovr · 2019-07-08T18:13:38Z

Key Derivation?

I'm not sure about operations "Key loading" vs "Key import". Second is defined for private and public. What will load first one?

petrovr · 2019-07-08T18:45:31Z

Perhaps API could be based on basic(elementary/atomic) operations defined in "key management protocol". There is some standard protocols but I prefer experts to propose which one to use. Roumen

levitte · 2019-07-08T20:35:11Z

Key Derivation?

Different operation, see #9266... but yes, it has occured to @mattcaswell and me that key exchange result in a key as well, that wasn't lost on us. There will be some form of integration eventually, I'm sure, but not now and not in this PR. Let's take it a step at a time, yeah?

levitte · 2019-07-08T20:36:06Z

Perhaps API could be based on basic(elementary/atomic) operations defined in "key management protocol".

Not sure if that would make things better or not. Maybe...

levitte · 2019-07-08T20:39:55Z

I'm not sure about operations "Key loading" vs "Key import".

The key import function takes key material from the input OSSL_PARAM array and creates a key from that input (usually a bunch of big numbers).
The key loading function is aimed at HSMs, and gives the option to "load" an HSM protected key given some kind of identity. The identity isn't defined at all for the moment, and I'm thinking of it as highly provider specific, i.e. unstandardised. My current thinking is that it can work together with a provider OSSL_STORE implementation that returns such identities instead of EVP_PKEYs, and leaves it to libcrypto to actually "load" the key given such an identity. I've an unsubmitted branch where I currently experiment with this.

levitte · 2019-07-09T14:52:29Z

Documentation added

petrovr · 2019-07-09T15:36:19Z

Richard Levitte wrote:

> I'm not sure about operations "Key loading" vs "Key import". The key import function takes key material from the input `OSSL_PARAM` array and creates a key from that input (usually a bunch of big numbers).

Interesting. Working with different cryptographic libraries I note that common part is exchange of DER encoded data (key, cert). So instead ASN1 encoding you prefer data encoded as OSSL_PARAM, right?

The key loading function is aimed at HSMs, and gives the option to "load" an HSM protected key given some kind of identity. The identity isn't defined at all for the moment, and I'm thinking of it as highly provider specific, i.e. unstandardised.

This cannot be restricted to "hardware" only. Software "key-store" may not allow private part to be exported.

My current thinking is that it can work together with a provider OSSL_STORE implementation that returns such identities instead of EVP_PKEYs, and leaves it to libcrypto to actually "load" the key given such an identity. I've an unsubmitted branch where I currently experiment with this.

Right now I'm not sure what you mean by provider and what is difference to engine. Shall we consider engine as simple provider? Regards, Roumen

levitte · 2019-07-09T15:39:08Z

I think you need to read up in https://www.openssl.org/docs/OpenSSL300Design.html and https://www.openssl.org/docs/OpenSSLStrategicArchitecture.html. The former is what were working toward, and is the reason you're seeing so much moving around right now.

petrovr · 2019-07-09T15:59:37Z

Richard Levitte wrote:

The identity isn't defined at all for the moment, and I'm thinking of it as highly provider specific, i.e. unstandardised.

In some implementation "identity" is X.509 certificate(a) and private key. For other is pair public(b) and private key. Private key is accessed indirectly by (a) or by (b). May be you want to use another kind of keys . But this will require delegation of cryptographic operations to identity or "identity provider". Roumen

levitte · 2019-07-09T19:39:49Z

Working with different cryptographic libraries I note that common part is exchange of DER encoded data (key, cert). So instead ASN1 encoding you prefer data encoded as OSSL_PARAM, right?

We have chosen the OSSL_PARAM array as a data transfer medium between libcrypto and provider, yes. We wanted something that isn't tied to any particular form of serialization. I've realized after the fact that they have certain similarities with PKCS#11's attribute templates.

levitte · 2019-07-09T19:41:05Z

The key loading function is aimed at HSMs, and gives the option to "load" an HSM protected key given some kind of identity. The identity isn't defined at all for the moment, and I'm thinking of it as highly provider specific, i.e. unstandardised.

This cannot be restricted to "hardware" only. Software "key-store" may not allow private part to be exported.

Correct, I use "HSM" loosly, i.e. for something that keeps keys and other sensitive data for themselves.

levitte · 2019-07-11T11:11:12Z

This is not a draft any more

mattcaswell · 2019-07-12T14:41:15Z

"make doc-nits" is complaining.

mattcaswell

We should have some tests for this code.

crypto/evp/keymgmt_lib.c

crypto/evp/pmeth_lib.c

doc/internal/man3/evp_keymgmt_export_to_provider.pod

mattcaswell · 2019-07-12T15:37:14Z

doc/internal/man3/evp_keymgmt_freekey.pod

Should we be documenting what "importdomain" actually does? Similarly with the other lines below.

Yeah, we should. I also think we should document the dispatch functions, I just dunno how, or rather where. (I'm trying to document in the header file at the very least, something we generally suck at, I might add...)

Whether in this PR or another, we're going to need to document all of these provider functions in the man pages. A provider author should be able to find all the information they need in the man pages, without have to read the source.

Agreed. For the core_numbers.h stuff, that will be another PR. I have ideas on that...

doc/man3/EVP_KEYMGMT.pod

levitte · 2019-07-12T17:29:13Z

crypto/evp/keymgmt_lib.c

I just realised nothing is done about the case where all the slots of pk->pkeys are filled in and none of them matched the given keymgmt, i.e. we're not dropping any single entry from the cache... Needs to be fixed.

levitte · 2019-07-12T17:43:42Z

We should have some tests for this code.

Ah, in test/evp_extra_test.c? (I noticed that EVP_MD_fetch has a few tests there)

Sure, I will, if you add one for EVP_CIPHER_fetch 😉

levitte · 2019-07-12T19:35:14Z

"make doc-nits" is complaining.

Fixed

levitte · 2019-07-16T10:20:32Z

Looks like you need a rebase

Yup, I noticed. Especially with #9266 just being merged...

levitte · 2019-07-16T11:11:57Z

Rebased and squashed, to get a clean slate.

levitte · 2019-07-17T14:25:54Z

I think this is ready for final review. There was a failure in Travis, but that turned out to be an unrelated timeout. I've restarted that job, just for the hope of greenery 😉

mattcaswell

LGTM. A couple of minor comments which may or may not be addressed before committing. Approval counts either way.

mattcaswell · 2019-07-19T13:42:35Z

crypto/evp/keymgmt_lib.c

This sentence doesn't seem to quite make sense

mattcaswell · 2019-07-19T13:50:50Z

crypto/include/internal/asn1_int.h

Perhaps we should add a TODO here (or somewhere else appropriate) to remind us to go and implement these for the various legacy keys that we have?

Okie, added one here

The idea with the key management "operation" is to support the following set of functionality: - Key domain parameter generation - Key domain parameter import - Key domain parameter export - Key generation - Key import - Key export - Key loading (HSM / hidden key support) With that set of function, we can support handling domain parameters on one provider, key handling on another, and key usage on a third, with transparent export / import of applicable data. Of course, if a provider doesn't offer export / import functionality, then all operations surrounding a key must be performed with the same provider. This method also avoids having to do anything special with legacy assignment of libcrypto key structures, i.e. EVP_PKEY_assign_RSA(). They will simply be used as keys to be exported from whenever they are used with provider based operations. This change only adds the EVP_KEYMGMT API and the libcrypto <-> provider interface. Further changes will integrate them into existing libcrypto functionality.

This function is used to transport private key materia from whatever is already attached to the EVP_PKEY to the new provider, using key data export and import functionality. If a legacy lower level key has been assigned to the EVP_PKEY, we use its data to create a provider side key, and thereby have a bridge between old style public key types and the EVP_PKEY on providers. If successful, this function returns a reference to the appropriate provider side data for the key. This can be used by any operation that wants to use this key.

This affects all its callers: EVP_PKEY_CTX_new(), EVP_PKEY_CTX_new_id(). They are now possible to called with "zero" values, i.e.: EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new(NULL, NULL); or EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new_id(0, NULL); This is suitable for provider use, as the key functionality is tied with its keys, and the operation time is determined by the init functions the EVP_PKEY_CTX is used with.

This function clears the cache of provider key references, and is used in evp_keymgmt_export_to_provider() when the internal key is dirty, as well as by EVP_PKEY_free_it().

… providers

levitte · 2019-07-22T04:26:36Z

Merged.

a94a3e0 Add basic EVP_KEYMGMT API and libcrypto <-> provider interface
70a1f7b Add evp_keymgmt_export_to_provider(), for key transfer between providers
d0ea49a Adapt int_ctx_new() to use with providers
4cae07f Add evp_keymgmt_clear_pkey_cache() and use it
fa9faf0 Add an internal API to access the KEYMGMT provider functions

The idea with the key management "operation" is to support the following set of functionality: - Key domain parameter generation - Key domain parameter import - Key domain parameter export - Key generation - Key import - Key export - Key loading (HSM / hidden key support) With that set of function, we can support handling domain parameters on one provider, key handling on another, and key usage on a third, with transparent export / import of applicable data. Of course, if a provider doesn't offer export / import functionality, then all operations surrounding a key must be performed with the same provider. This method also avoids having to do anything special with legacy assignment of libcrypto key structures, i.e. EVP_PKEY_assign_RSA(). They will simply be used as keys to be exported from whenever they are used with provider based operations. This change only adds the EVP_KEYMGMT API and the libcrypto <-> provider interface. Further changes will integrate them into existing libcrypto functionality. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

This function is used to transport private key materia from whatever is already attached to the EVP_PKEY to the new provider, using key data export and import functionality. If a legacy lower level key has been assigned to the EVP_PKEY, we use its data to create a provider side key, and thereby have a bridge between old style public key types and the EVP_PKEY on providers. If successful, this function returns a reference to the appropriate provider side data for the key. This can be used by any operation that wants to use this key. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

This affects all its callers: EVP_PKEY_CTX_new(), EVP_PKEY_CTX_new_id(). They are now possible to called with "zero" values, i.e.: EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new(NULL, NULL); or EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new_id(0, NULL); This is suitable for provider use, as the key functionality is tied with its keys, and the operation time is determined by the init functions the EVP_PKEY_CTX is used with. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

This function clears the cache of provider key references, and is used in evp_keymgmt_export_to_provider() when the internal key is dirty, as well as by EVP_PKEY_free_it(). Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

levitte added the branch: master Applies to master branch label Jul 4, 2019

paulidale self-requested a review July 5, 2019 08:17

levitte mentioned this pull request Jul 7, 2019

Move PKCS#3 DH to the default provider #9266

Closed

levitte marked this pull request as ready for review July 11, 2019 11:10

levitte changed the title ~~WIP: Add basic EVP_KEYMGMT API and libcrypto <-> provider interface~~ Add basic EVP_KEYMGMT API and libcrypto <-> provider interface Jul 11, 2019

levitte force-pushed the evp_keymgmt branch from 7140f29 to ad9b55e Compare July 11, 2019 11:13

mattcaswell requested changes Jul 12, 2019

View reviewed changes

levitte commented Jul 12, 2019

View reviewed changes

levitte force-pushed the evp_keymgmt branch 2 times, most recently from 8134a5b to 4017ca5 Compare July 16, 2019 10:45

levitte mentioned this pull request Jul 16, 2019

Adapt DH to use with KEYMGMT #9394

Closed

levitte force-pushed the evp_keymgmt branch 2 times, most recently from 3c8faa8 to 5ad4970 Compare July 17, 2019 12:01

mattcaswell approved these changes Jul 19, 2019

View reviewed changes

levitte force-pushed the evp_keymgmt branch from 217d761 to af520b4 Compare July 20, 2019 06:05

levitte added 11 commits July 22, 2019 06:05

Add evp_keymgmt_clear_pkey_cache() and use it

d609f8b

This function clears the cache of provider key references, and is used in evp_keymgmt_export_to_provider() when the internal key is dirty, as well as by EVP_PKEY_free_it().

Add an internal API to access the KEYMGMT provider functions

f5870be

fixup! Add evp_keymgmt_export_to_provider(), for key transfer between…

6bfe915

… providers

fixup! Add basic EVP_KEYMGMT API and libcrypto <-> provider interface

968d3c1

fixup! Add evp_keymgmt_clear_pkey_cache() and use it

860de5d

fixup! Add basic EVP_KEYMGMT API and libcrypto <-> provider interface

84be00d

fixup! Add an internal API to access the KEYMGMT provider functions

c876bec

fixup! Add evp_keymgmt_export_to_provider(), for key transfer between…

4ac21cb

… providers

levitte force-pushed the evp_keymgmt branch from af520b4 to 4ac21cb Compare July 22, 2019 04:05

levitte closed this Jul 22, 2019

levitte added a commit that referenced this pull request Jul 22, 2019

Add an internal API to access the KEYMGMT provider functions

fa9faf0

Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from #9312)

levitte mentioned this pull request Aug 10, 2020

OSSL_STORE for providers, take 2 #12512

Closed

Uh oh!

Conversation

levitte commented Jul 4, 2019

Uh oh!

levitte commented Jul 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

levitte commented Jul 4, 2019

Uh oh!

paulidale commented Jul 5, 2019

Uh oh!

petrovr commented Jul 8, 2019

Uh oh!

petrovr commented Jul 8, 2019 via email

Uh oh!

levitte commented Jul 8, 2019

Uh oh!

levitte commented Jul 8, 2019

Uh oh!

levitte commented Jul 8, 2019

Uh oh!

levitte commented Jul 9, 2019

Uh oh!

petrovr commented Jul 9, 2019 via email

Uh oh!

levitte commented Jul 9, 2019

Uh oh!

petrovr commented Jul 9, 2019 via email

Uh oh!

levitte commented Jul 9, 2019

Uh oh!

levitte commented Jul 9, 2019

Uh oh!

levitte commented Jul 11, 2019

Uh oh!

mattcaswell commented Jul 12, 2019

Uh oh!

mattcaswell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

levitte commented Jul 12, 2019

Uh oh!

levitte commented Jul 12, 2019

Uh oh!

levitte commented Jul 16, 2019

Uh oh!

levitte commented Jul 16, 2019

Uh oh!

levitte commented Jul 17, 2019

Uh oh!

mattcaswell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

levitte commented Jul 4, 2019 •

edited

Loading