contrib: add QAT private key provider. by ipuustin · Pull Request #21984 · envoyproxy/envoy

ipuustin · 2022-07-01T12:10:49Z

Commit Message:

contrib: add QAT private key provider for RSA handshake acceleration.

Next generation Intel® QAT support with Intel® Xeon® Scalable processors will feature an Intel® QAT cryptography and compression acceleration engine.

QAT private key provider extension will use qatlib library (https://github.com/intel/qatlib) to accelerate RSA operations in handshakes. The extension will look a bit like the existing cryptomb private key provider. The use case is to move the expensive cryptographic operations away from the CPU to the accelerator device, leaving CPU cycles for other use.

Additional Description:

Support for Intel® QAT is already present in the mainline Linux kernel and in Kubernetes device plugins (to expose the device files to containers). There are previous generations of Intel QAT® hardware devices, but they are not supported by this extension.

Two patches to libqat are present, but they should be included into the next qatlib release.

Risk Level: medium (contrib extension, no HW available yet, crypto-related code)
Testing: unit tests
Docs Changes: Protobuf comments
Release Notes: N/A
Platform Specific Features: Requires a future Intel Xeon Scalable processor.
Fixes #21531 (partly, QATzip extension will be a later PR)

Next generation Intel® QAT support with Intel® Xeon® Scalable processors will feature an Intel® QAT cryptography and compression acceleration engine. QAT private key provider extension will use qatlib library (https://github.com/intel/qatlib) to accelerate RSA operations in handshakes. The extension will look a bit like the existing cryptomb private key provider. The use case is to move the expensive cryptographic operations away from the CPU to the accelerator device, leaving CPU cycles for other use. Support for Intel® QAT is already present in the mainline Linux kernel and in Kubernetes device plugins (to expose the device files to containers). There are previous generations of Intel QAT® hardware devices, but they are not supported by this extension. Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

repokitteh-read-only · 2022-07-01T12:10:58Z

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).
envoyproxy/dependency-shepherds assignee is @moderation

🐱

Caused by: #21984 was opened by ipuustin.

see: more, trace.

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

daixiang0 · 2022-07-04T07:50:22Z

api/contrib/envoy/extensions/private_key_providers/qat/v3alpha/qat.proto

+// The private key provider provides RSA sign and decrypt operation
+// hardware acceleration.
+
+// [#extension-category: envoy.tls.key_providers]


Please remove this line as we do need one envoy.tls.key_providers category.

Ping @daixiang0 if you are able to re-review and stamp

Looks like autoconf can't parse the -fuse-ld=lld flag, which leads to wrong linker being used, which in turn leads to unrecognized linker command line options. An alternative would be to set the LD env variable in the BUILD file for clang-based builds. Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

wrowe · 2022-07-08T18:52:15Z

/assign-from @envoyproxy/senior-maintainers

for new contrib component review

repokitteh-read-only · 2022-07-08T18:52:19Z

@envoyproxy/senior-maintainers assignee is @ggreenway

🐱

Caused by: a #21984 (comment) was created by @wrowe.

see: more, trace.

ggreenway · 2022-07-12T20:51:18Z

@moderation how does this look for dependencies?

moderation · 2022-07-12T21:33:40Z

Not great but this is contrib and follows prior Intel contributions. Lack of code review, contributors and maintenance noted.

scorecard --repo=https://github.com/intel/qatlib

RESULTS
-------
Aggregate score: 5.4 / 10

Check scores:
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
|  SCORE  |          NAME          |             REASON             |                             DOCUMENTATION/REMEDIATION                             |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Binary-Artifacts       | no binaries found in the repo  | https://github.com/ossf/scorecard/blob/main/docs/checks.md#binary-artifacts       |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 6 / 10  | Branch-Protection      | branch protection is not       | https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection      |
|         |                        | maximal on development and all |                                                                                   |
|         |                        | release branches               |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | CI-Tests               | 0 out of 2 merged PRs          | https://github.com/ossf/scorecard/blob/main/docs/checks.md#ci-tests               |
|         |                        | checked by a CI test -- score  |                                                                                   |
|         |                        | normalized to 0                |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | CII-Best-Practices     | no badge detected              | https://github.com/ossf/scorecard/blob/main/docs/checks.md#cii-best-practices     |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 1 / 10  | Code-Review            | GitHub code reviews found for  | https://github.com/ossf/scorecard/blob/main/docs/checks.md#code-review            |
|         |                        | 2 commits out of the last 17   |                                                                                   |
|         |                        | -- score normalized to 1       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Contributors           | 0 different organizations      | https://github.com/ossf/scorecard/blob/main/docs/checks.md#contributors           |
|         |                        | found -- score normalized to 0 |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Dangerous-Workflow     | no dangerous workflow patterns | https://github.com/ossf/scorecard/blob/main/docs/checks.md#dangerous-workflow     |
|         |                        | detected                       |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Dependency-Update-Tool | no update tool detected        | https://github.com/ossf/scorecard/blob/main/docs/checks.md#dependency-update-tool |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Fuzzing                | project is not fuzzed          | https://github.com/ossf/scorecard/blob/main/docs/checks.md#fuzzing                |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | License                | license file detected          | https://github.com/ossf/scorecard/blob/main/docs/checks.md#license                |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | Maintained             | 0 commit(s) out of 17 and 0    | https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained             |
|         |                        | issue activity out of 5 found  |                                                                                   |
|         |                        | in the last 90 days -- score   |                                                                                   |
|         |                        | normalized to 0                |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| ?       | Packaging              | no published package detected  | https://github.com/ossf/scorecard/blob/main/docs/checks.md#packaging              |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Pinned-Dependencies    | all dependencies are pinned    | https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies    |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 0 / 10  | SAST                   | SAST tool is not run on all    | https://github.com/ossf/scorecard/blob/main/docs/checks.md#sast                   |
|         |                        | commits -- score normalized to |                                                                                   |
|         |                        | 0                              |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Security-Policy        | security policy file detected  | https://github.com/ossf/scorecard/blob/main/docs/checks.md#security-policy        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| ?       | Signed-Releases        | no releases found              | https://github.com/ossf/scorecard/blob/main/docs/checks.md#signed-releases        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Token-Permissions      | tokens are read-only in GitHub | https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions      |
|         |                        | workflows                      |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| 10 / 10 | Vulnerabilities        | no vulnerabilities detected    | https://github.com/ossf/scorecard/blob/main/docs/checks.md#vulnerabilities        |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|
| ?       | Webhooks               | check is not supported for     | https://github.com/ossf/scorecard/blob/main/docs/checks.md#webhooks               |
|         |                        | this request: SCORECARD_V6     |                                                                                   |
|         |                        | is not set, not running the    |                                                                                   |
|         |                        | Webhook check                  |                                                                                   |
|---------|------------------------|--------------------------------|-----------------------------------------------------------------------------------|

moderation · 2022-07-12T21:33:47Z

/lgtm deps

phlax · 2022-07-18T08:34:15Z

@ggreenway would you mind looking over this one again

/wait-any

phlax · 2022-07-19T09:34:22Z

@ipuustin this seems to have introduced a flake in CI:

https://dev.azure.com/cncf/envoy/_build/results?buildId=113225&view=logs&j=8c169225-0ae8-53bd-947f-07cb81846cb5&t=6b0ace90-28b2-51d9-2951-b1fd35e67dec&l=2212

ipuustin · 2022-07-20T21:35:37Z

@ipuustin this seems to have introduced a flake in CI:

https://dev.azure.com/cncf/envoy/_build/results?buildId=113225&view=logs&j=8c169225-0ae8-53bd-947f-07cb81846cb5&t=6b0ace90-28b2-51d9-2951-b1fd35e67dec&l=2212

Thanks for the heads up. I'll take a look at ops_test to see what might go wrong there.

Next generation Intel® QAT support with Intel® Xeon® Scalable processors will feature an Intel® QAT cryptography and compression acceleration engine. QAT private key provider extension will use qatlib library (https://github.com/intel/qatlib) to accelerate RSA operations in handshakes. The extension will look a bit like the existing cryptomb private key provider. The use case is to move the expensive cryptographic operations away from the CPU to the accelerator device, leaving CPU cycles for other use. Support for Intel® QAT is already present in the mainline Linux kernel and in Kubernetes device plugins (to expose the device files to containers). There are previous generations of Intel QAT® hardware devices, but they are not supported by this extension. Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Jul 1, 2022

repokitteh-read-only bot assigned moderation Jul 1, 2022

ipuustin added 4 commits July 1, 2022 15:26

Fix typos.

5e438b6

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Add protobuf docs to documentation tree.

78ac1f0

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

More spelling fixes.

6fff04a

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Update patches to work around missing pkg-config.

6dbe3fa

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

daixiang0 reviewed Jul 4, 2022

View reviewed changes

ipuustin added 8 commits July 4, 2022 10:54

Fix clang-tidy errors.

275363f

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Remove QAT private key provider from ARM build.

847a337

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Try even harder to disable qatlib from ARM build.

c5b6e3a

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Fix format.

a462f9d

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

More formatting.

2a97ca9

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

More QAT disabling from ARM builds.

0cf3b5b

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

Remove envoy.tls.key_providers duplicate category.

861a5f9

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>

repokitteh-read-only bot assigned ggreenway Jul 8, 2022

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label Jul 12, 2022

repokitteh-read-only bot added the waiting:any label Jul 18, 2022

ggreenway approved these changes Jul 18, 2022

View reviewed changes

repokitteh-read-only bot removed the waiting:any label Jul 18, 2022

ggreenway merged commit 9447ff5 into envoyproxy:main Jul 18, 2022

ipuustin mentioned this pull request Jul 20, 2022

qat: remove flaking test. #22338

Merged

heyuanliu-intel mentioned this pull request Oct 9, 2022

SSLNTV-29 Add the ability to use a custom OpenSSL engine wildfly-security/wildfly-openssl-natives#16

Open

This was referenced Nov 15, 2022

Enable configuring Envoy QAT private key provider istio/api#2565

Merged

Enable QAT PrivateKeyProvider extension istio/proxy#4202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

contrib: add QAT private key provider.#21984

contrib: add QAT private key provider.#21984
ggreenway merged 13 commits intoenvoyproxy:mainfrom
ipuustin:qat-private-key-provider-pr

ipuustin commented Jul 1, 2022

Uh oh!

repokitteh-read-only bot commented Jul 1, 2022

Uh oh!

daixiang0 Jul 4, 2022

Uh oh!

ipuustin Jul 4, 2022

Uh oh!

wrowe Jul 7, 2022

Uh oh!

wrowe commented Jul 8, 2022

Uh oh!

repokitteh-read-only bot commented Jul 8, 2022

Uh oh!

ggreenway commented Jul 12, 2022

Uh oh!

moderation commented Jul 12, 2022

Uh oh!

moderation commented Jul 12, 2022

Uh oh!

phlax commented Jul 18, 2022

Uh oh!

phlax commented Jul 19, 2022

Uh oh!

ipuustin commented Jul 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ipuustin commented Jul 1, 2022

Uh oh!

repokitteh-read-only bot commented Jul 1, 2022

Uh oh!

daixiang0 Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

ipuustin Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

wrowe Jul 7, 2022

Choose a reason for hiding this comment

Uh oh!

wrowe commented Jul 8, 2022

Uh oh!

repokitteh-read-only bot commented Jul 8, 2022

Uh oh!

ggreenway commented Jul 12, 2022

Uh oh!

moderation commented Jul 12, 2022

Uh oh!

moderation commented Jul 12, 2022

Uh oh!

phlax commented Jul 18, 2022

Uh oh!

phlax commented Jul 19, 2022

Uh oh!

ipuustin commented Jul 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants