Skip to content

Commit b6854d7

Browse files
zaniebsalmonsd
zanieb
and
salmonsd
authored
Upgrade reqwest to 0.13 (#18550)
The following user-facing changes are included here: - `aws-lc` is used instead of `ring` for a cryptography backend - Expands our certificate signature algorithm support to include ECDSA_P256_SHA512, ECDSA_P384_SHA512, ECDSA_P521_SHA256, ECDSA_P521_SHA384, and ECDSA_P521_SHA512 - `--native-tls` is deprecated in favor of a new `--system-certs` flag, avoiding confusion with the TLS implementation used (we use `rustls` not `native-tls`, see prior confusion at #11595) - NASM is a new build requirement on Windows, it is required by `aws-lc` on x86-64 and i386 - `rustls-platform-verifier` is used instead of `rustls-native-certs` for system certificate verification - On macOS, certificate validation is now delegated to `Security.framework` (`SecTrust`). Performance when using `--system-certs` is improved by avoiding exporting and parsing all the certificates from the keychain at startup. - On Windows, certificate validation is now delegated to `CertGetCertificateChain` and `CertVerifyCertificateChainPolicy` - On Linux, certificate validation should be approximately unchanged - Some previously failing chains may succeed, and some previously accepted chains may fail; generally, this should result in behavior closer matching browsers and other native applications - macOS and Windows may now perform live OCSP fetches for early revocation, which could add latency to some requests - Empty `SSL_CERT_FILE` values are ignored (for consistency with `SSL_CERT_DIR`) The following internal changes are included here: - Certificate loading has been refactored to use a newtype with helper methods - The certificate tests have been rewritten - We use `webpki-root-certs` instead of `webpki-roots`, see #17543 (comment) - We request `identity` encoding for range requests, see astral-sh/async_http_range_reader#3 (comment) - Various dependencies (including forks) updates to versions which use reqwest 0.13+ This is a replacement of #17543 with an updated description. See that pull request for prior discussion. I've made the following changes from the initial approach there: - Previously, the `native-tls` TLS implementation was added which included an OpenSSL build. We don't currently use the `native-tls` implementation, but the `--native-tls` flag there was erroneously updated to enable it. - Previously, there was a `--tls-backend` flag to toggle between `native-tls` and `rustls`. Since we currently always use `rustls`, this is deferred to future work (if we need it at all). - Previously, there were unintentional breaking changes to `SSL_CERT_FILE` and `SSL_CERT_DIR` handling, including merging with the base certificates instead of replacing them, dropping support for OpenSSL hash-named certificate files, skipping deduplication of certificates. Here, we retain use of `rustls-native-certs` for loading certificates from the system as it handles these edge cases. Closes #17427 --------- Co-authored-by: salmonsd <22984014+salmonsd@users.noreply.github.com>
1 parent c43c0d0 commit b6854d7

43 files changed

Lines changed: 2167 additions & 739 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build-dev-binaries.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,7 +307,9 @@ jobs:
307307
echo "CC_aarch64_linux_android=${TOOLCHAIN}/bin/aarch64-linux-android24-clang" >> "$GITHUB_ENV"
308308
echo "CXX_aarch64_linux_android=${TOOLCHAIN}/bin/aarch64-linux-android24-clang++" >> "$GITHUB_ENV"
309309
echo "AR_aarch64_linux_android=${TOOLCHAIN}/bin/llvm-ar" >> "$GITHUB_ENV"
310+
echo "RANLIB_aarch64_linux_android=${TOOLCHAIN}/bin/llvm-ranlib" >> "$GITHUB_ENV"
310311
echo "CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=${TOOLCHAIN}/bin/aarch64-linux-android24-clang" >> "$GITHUB_ENV"
312+
echo "CARGO_TARGET_AARCH64_LINUX_ANDROID_RANLIB=${TOOLCHAIN}/bin/llvm-ranlib" >> "$GITHUB_ENV"
311313
312314
# NDK 23+ removed libgcc, provide a stub that redirects to libunwind
313315
LIBDIR=$(echo "${TOOLCHAIN}"/lib/clang/*/lib/linux/aarch64)

‎.github/workflows/build-release-binaries.yml‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -256,7 +256,14 @@ jobs:
256256
- name: "Install cargo extensions"
257257
shell: bash
258258
run: scripts/install-cargo-extensions.sh
259-
259+
- name: "Install NASM"
260+
# NASM is required for x86/x86-64 Windows targets by aws-lc-sys.
261+
# On aarch64-pc-windows-msvc, it uses clang-cl instead.
262+
# See: https://aws.github.io/aws-lc-rs/requirements/windows.html#build-requirements
263+
if: contains(matrix.platform.target, 'x86') || contains(matrix.platform.target, 'i686')
264+
run: |
265+
winget install NASM.NASM --accept-source-agreements --accept-package-agreements
266+
echo "C:\Program Files\NASM" | Out-File -FilePath $env:GITHUB_PATH -Append
260267
# uv
261268
- name: "Build wheels"
262269
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1.50.1
@@ -266,6 +273,8 @@ jobs:
266273
args: --release --locked --out dist --features self-update,windows-gui-bin --compatibility pypi
267274
env:
268275
CARGO: ${{ github.workspace }}/scripts/cargo.cmd
276+
# Disable prebuilt NASM objects so we always compile assembly from source.
277+
AWS_LC_SYS_PREBUILT_NASM: "0"
269278
- name: "Test wheel"
270279
shell: bash
271280
run: |
@@ -351,25 +360,25 @@ jobs:
351360
manylinux: 2_17
352361
docker-options: -e CARGO
353362
args: --release --locked --out dist --features self-update --compatibility pypi
354-
# See: https://github.com/sfackler/rust-openssl/issues/2036#issuecomment-1724324145
355363
before-script-linux: |
356364
# Install the 32-bit cross target on 64-bit (noop if we're already on 64-bit)
357365
rustup target add ${{ matrix.target }}
358366
# If we're running on rhel centos, install needed packages.
359367
if command -v yum &> /dev/null; then
360-
yum update -y && yum install -y perl-core openssl openssl-devel pkgconfig libatomic
368+
yum update -y && yum install -y pkgconfig libatomic
361369
362-
# If we're running on i686 we need to symlink libatomic
363-
# in order to build openssl with -latomic flag.
364-
if [[ ! -d "/usr/lib64" ]]; then
370+
# Install cross build requirements
371+
if [[ "${{ matrix.target }}" == "i686-unknown-linux-gnu" ]]; then
372+
yum install -y glibc-devel.i686 libstdc++-devel.i686 libatomic.i686
373+
fi
374+
375+
# Symlink libatomic so the linker can find it with -latomic.
376+
if [[ -f "/usr/lib/libatomic.so.1" && ! -f "/usr/lib/libatomic.so" ]]; then
365377
ln -s /usr/lib/libatomic.so.1 /usr/lib/libatomic.so
366-
else
367-
# Support cross-compiling from 64-bit to 32-bit
368-
yum install -y glibc-devel.i686 libstdc++-devel.i686
369378
fi
370379
else
371380
# If we're running on debian-based system.
372-
apt update -y && apt-get install -y libssl-dev openssl pkg-config
381+
apt update -y && apt-get install -y pkg-config
373382
fi
374383
# Install cargo extensions as a static musl binary so it runs in any container.
375384
scripts/install-cargo-extensions.sh
@@ -595,8 +604,12 @@ jobs:
595604
rust-toolchain: ${{ matrix.platform.toolchain || null }}
596605
before-script-linux: |
597606
scripts/install-cargo-extensions.sh
607+
# Install the s390x cross target on x86_64
608+
rustup target add ${{ matrix.platform.target }}
609+
apt-get update && apt-get install -y gcc-s390x-linux-gnu binutils-s390x-linux-gnu
598610
env:
599611
CARGO: ${{ github.workspace }}/scripts/cargo.sh
612+
600613
- uses: uraimo/run-on-arch-action@d94c13912ea685de38fccc1109385b83fd79427d # v3.0.1
601614
name: "Test wheel"
602615
with:

‎CONTRIBUTING.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,17 @@ On Fedora-based distributions, you can install a C compiler with:
5353
sudo dnf install gcc
5454
```
5555

56+
On Windows, [NASM](https://www.nasm.us/) is required for building the TLS backend (`aws-lc-sys`). If
57+
it is not present, a prebuilt blob provided by `aws-lc-sys` will be used instead. WinGet can be used
58+
to install NASM:
59+
60+
```shell
61+
winget install NASM.NASM
62+
```
63+
64+
After installation, add `C:\Program Files\NASM` to your `PATH`. While the prebuilt blob will not be
65+
used when NASM is found, you can guarantee this behavior by setting `AWS_LC_SYS_PREBUILT_NASM=0`.
66+
5667
## Testing
5768

5869
For running tests, we recommend [nextest](https://nexte.st/).

0 commit comments

Comments
 (0)