Skip to content

Graduate the QUIC codec from incubator to Netty Core#14979

Merged
chrisvest merged 901 commits intonetty:4.2from
chrisvest:4.2-quic
Apr 4, 2025
Merged

Graduate the QUIC codec from incubator to Netty Core#14979
chrisvest merged 901 commits intonetty:4.2from
chrisvest:4.2-quic

Conversation

@chrisvest
Copy link
Copy Markdown
Member

Motivation:
QUIC is the underlying transport protocol for HTTP/3.
The prior HTTP versions are all core protocols, so naturally HTTP/3 should be too.

Modification:
Copy over the source code from netty/netty-incubator-codec-quic@1a4b5dc.
Adjust the pom.xml files to fit into the project as a proper Netty submodule.

Result:
The QUIC module has graduated from incubator.

Fixes netty/netty-incubator-codec-quic#580

normanmaurer and others added 30 commits July 24, 2023 14:58
Motivation:

Let's use dependencies which have not known CVEs

Modifications:

Upgrade bouncycastle and logback

Result:

Cleanup
Motivation:

There are new releases of java.

Modifications:

Upgrade java versions to latest release

Result:

Use latest versions on the CI
Motivation:

A new version of netty was released

Modifications:

Upgrade to latest version

Result:

Use latest netty release
Motivation:

When client auth is optional we must not fail the handshake on the
client side when no keymanager was configured on the client side.

Modifications:

- Fix handling on the client-side
- Add testcase
- Fix test

Result:

Fixes netty/netty-incubator-codec-quic#566
Motivation:

We are a bit behind, lets update to latest quiche sha.

Modifications:

Update to 5ddba81a3ab03c0fd4c262d444bded4208013669

Result:

Use latest quiche code
Motivation:

Let's use the commit sha of quiche for the 0.18.0 release.

Modifications:

- Change quiche sha
- Skip building of apps as its not needed

Result:

Use sha that represent the latest quiche release
…etty#572)

Motivation:

When caching fails during workflow execution we should still continue
the build as its just an optimization. Also we should use the new fork
for docker layer caching

Modifications:

- Update action that does the docker layer caching to the new fork
- Ignore errors during caching

Result:

More stable builds
Motivation:

We need quite some diskspace due our ue of docker, let's ensure we have
enough by free up what we dont need on the runner.

Modifications:

Use action that frees up diskspace

Result:

More stable build
Motivation:

We should use weak references to hold global references to our own
classes as otherwise it will be not possible to unload the classloader.

Modifications:

Use weak references for the classes in JNI.

Result:

Related to netty#13480
Motivation:

There were updates to the chromium-stable branch of boringssl.

Modifications:

Update to latest sha

Result:

Use more recent boringssl version
Motivation:

Since version 0.0.42.Final the connection from an android client was not
working anymore, caused by a QuicClosedChannelException. After
inspecting the changes I found that
885a18a introduced the bug.

Modifications:

The call to Quiche.memoryAddress used the capacity of the wrong buffer.
The bug only appeared on Android because on most other platforms the
buf.hasMemoryAddress() is true, and the len parameter is not used
therefore.

Result:

Make netty-incubator-codec-quic work on Android again
Motivation:

The javaModuleNameClassifier was missing in the android profile and
therefore the `AndroidManifest.xml` contained
`package="io.netty.incubator.codec.quic.linux.x86_64"`

Modifications:

Added javaModuleNameClassifier property in pom.xml

Result:

Show the right package name in the `AndroidManifest.xml`
netty#579)

… and Datagram extension is used

Motivation:

We can't assume we can directly access the memoryAddress of the ByteBuf
as it might not be possible on all platforms or even explicit disabled
by the end user

Modifications:

Use static helper method that will always be able to return the correct
memory address

Result:

Code works as expected on all platforms
Motivation:

We recently had a bug which only showed up when the memoryAddress of a
direct buffer could not be accessed directly. We didnt catch this as the
memory address always was accessible by unsafe. Let's add a PR build job
which explicit disable unsafe.

Modifications:

- Add profile which disables unsafe during build
- Add job which builds with unsafe disabled

Result:

Better test coverage
…" (netty#582)

Motivation:

The used github action does not work anymore

Modifications:

This reverts commit 776c601.

Result:

Build pass again
Motivation:

Docker images can be quite large, lets try to cleanup things we not
need.

Modifications:

- Cleanup things during docker image creation
- Disable docker layer caching as it most of the times not work anyway

Result:

Docker image needs less disk-space
Motivation:

There was a new maven release

Modifications:

Upgrade to 3.9.4

Result:

Use latest maven release via maven wrapper
…s it is not needed anymore (netty#586)

Motivation:
Recently GraalVM added support for Apple M1.

Modifications:
- Remove the comment from resource-config.json file
- Update the version for native-maven-plugin

Result:
The comment in resource-config.json is not needed anymore.
netty#587)

…eference

Motivation:

We missed to delete the local reference in some cases due early returns.

Modifications:

Move the delete calls close to the creation calls and so ensure we not
miss to delete local references

Result:

More correct code
…ty#588)

Motivation:

The code introduced as part of
netty/netty-incubator-codec-quic#574 contained a
bug which could crash the JVM. This only manifested in JDK21+ as earlier
versions did not contain the related assert.

Modifications:

Remove left-over call that could crash the JVM

Result:

Correct code that also works on JDK21
Motivation:

Now that JDK21 is released we should also build with it on our CI

Modifications:

Add docker-compose config for JDK21 and use it in the PR workflow

Result:

Test with latest LTS release
Motivation:

We should run our testsuite with '-Xcheck:jni' to ensure we catch bugs
in our JNI code which could later cause issues like crashes

Modifications:

- Add -Xcheck:jni
- Add -ea

Result:

Testsuite will be able to catch more JNI bugs
Motivation:

A new netty release is out.

Modifications:

Upgrade to latest release

Result:

Use latest release
Motivation:

We should update all our used actions to the latest version

Modifications:

Update all actions

Result:

Cleanup
@chrisvest chrisvest requested a review from normanmaurer April 2, 2025 21:26
@chrisvest chrisvest marked this pull request as ready for review April 2, 2025 21:26
chrisvest and others added 14 commits April 2, 2025 16:18
…available

The '$JAVA_HOME/bin/gu' binary is no longer part of the graalce distribution.
Motivation:

We only link against the quiche package so we only need to build it with
cargo

Modifications:

Only build quiche package

Result:

Speedup build and cleanup
Motivation:

We should link against the latest quiche release

Modifications:

Update to 0.23.4

Result:

Use sha that matches latest quiche release
Motivation:

Let's ensure we can build with latest netty 4.2.0 release

Modifications:

Change version to latest release

Result:

Validate everything works with latest netty 4.2.0 release
# Conflicts:
#	docker/docker-compose.centos-7.yaml
…ing on Linux

Also remove some duplicate test dependencies
@chrisvest chrisvest merged commit dc1829a into netty:4.2 Apr 4, 2025
24 of 25 checks passed
@chrisvest chrisvest deleted the 4.2-quic branch April 4, 2025 21:50
@chrisvest chrisvest added this to the 4.2.1.Final milestone Apr 4, 2025
@phonty29
Copy link
Copy Markdown

phonty29 commented Apr 9, 2025

Will every change in QUIC codec be applied to incubator first ?

@chrisvest
Copy link
Copy Markdown
Member Author

Will every change in QUIC codec be applied to incubator first ?

No, I think it'll be supported at about the same level as the rest of 4.1. That is, only bug and compatibility fixes from now on.

dongjoon-hyun added a commit to apache/spark that referenced this pull request Dec 5, 2025
…` license

### What changes were proposed in this pull request?

This PR aims to update `NOTICE-binary` with `Netty` 4.2.7 license.
- https://github.com/netty/netty/blob/netty-4.2.7.Final/NOTICE.txt

### Why are the changes needed?

It seems that we updated `Netty Notice` at Apache Spark `3.0.0-preview` with `Netty 4.1.30.Final`.
- #25544

Since there are many changes like the following, we need to update it by simply copying and pasting.
- Netty 4.1.38.Final
  - netty/netty#9344
- Netty 4.1.44.Final
  - netty/netty#9161
- Netty 4.1.54.Final
  - netty/netty#10773
- Netty 4.1.66.Final
  - netty/netty#11256
  - netty/netty#11437
- Netty 4.1.108.Final
  - netty/netty#13864
- Netty 4.2.1.Final
  - netty/netty#14979
- Netty 4.2.7.Final
  - netty/netty#15658

Additionally, I also double-checked newly added transitive license through ASF [LEGAL-700](https://issues.apache.org/jira/browse/LEGAL-700). We are good to go.

- **Apple Public Source License 2.0**
  - https://spdx.org/licenses/APSL-2.0.html

### Does this PR introduce _any_ user-facing change?

No behavior change.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #53335 from dongjoon-hyun/SPARK-54602.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun added a commit to apache/spark that referenced this pull request Dec 5, 2025
…` license

### What changes were proposed in this pull request?

This PR aims to update `NOTICE-binary` with `Netty` 4.2.7 license.
- https://github.com/netty/netty/blob/netty-4.2.7.Final/NOTICE.txt

### Why are the changes needed?

It seems that we updated `Netty Notice` at Apache Spark `3.0.0-preview` with `Netty 4.1.30.Final`.
- #25544

Since there are many changes like the following, we need to update it by simply copying and pasting.
- Netty 4.1.38.Final
  - netty/netty#9344
- Netty 4.1.44.Final
  - netty/netty#9161
- Netty 4.1.54.Final
  - netty/netty#10773
- Netty 4.1.66.Final
  - netty/netty#11256
  - netty/netty#11437
- Netty 4.1.108.Final
  - netty/netty#13864
- Netty 4.2.1.Final
  - netty/netty#14979
- Netty 4.2.7.Final
  - netty/netty#15658

Additionally, I also double-checked newly added transitive license through ASF [LEGAL-700](https://issues.apache.org/jira/browse/LEGAL-700). We are good to go.

- **Apple Public Source License 2.0**
  - https://spdx.org/licenses/APSL-2.0.html

### Does this PR introduce _any_ user-facing change?

No behavior change.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #53335 from dongjoon-hyun/SPARK-54602.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 191ce4c)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move out of incubator