Skip to content

Modify OpenSSL native library loading to accommodate GraalVM#10395

Merged
normanmaurer merged 3 commits intonetty:4.1from
dzou:4.1
Jul 8, 2020
Merged

Modify OpenSSL native library loading to accommodate GraalVM#10395
normanmaurer merged 3 commits intonetty:4.1from
dzou:4.1

Conversation

@dzou
Copy link
Copy Markdown
Contributor

@dzou dzou commented Jul 7, 2020

Motivation:

We are interested in building Netty libraries with Ahead-of-time compilation with GraalVM. We saw there was prior work done on this. We want to introduce a change which will unblock GraalVM support for applications using netty and netty-tcnative.

This solves the error that some others encounter:

Exception in thread "main" java.lang.UnsatisfiedLinkError: io.grpc.netty.shaded.io.netty.internal.tcnative.NativeStaticallyReferencedJniMethods.sslOpCipherServerPreference()I [symbol: Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference or Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference__]

Modification:

The root cause of the issue is that in the tcnative libraries, the SSL.java class references a native call in the static initialization of the class - the method sslOpCipherServerPreference() on line 67 is used to initialize the static variable. But you see that OpenSsl also uses SSL.class to check if netty-tcnative is on the classpath before loading the native library.

So this is the problem because in ahead-of-time compilation, when it references the SSL class, it will try to load those static initializers and make the native library call, but it cannot do so because the native library has not been loaded yet since the SSL class is being referenced to check if the library should be loaded in the first place.

Solution: So the proposed solution here is to just choose a different class in the tcnative package which does not make a native library call during static initialization. I just chose SSLContext if this is OK.

This should have no negative effects other than unblocking the GraalVM use-case.

Result:

It fixes the unsatisfied link error. It will fix error for future users; it is a common error people encounter.

I also have a reproducible sample demonstrating this: https://github.com/dzou/graal-gcp-experiments

cc/ @meltsufin

@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

3 similar comments
@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@normanmaurer
Copy link
Copy Markdown
Member

@dzou sounds like a good solution ...

Can you sign the icla and let me know once done ?
https://netty.io/s/icla

@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

@dzou
Copy link
Copy Markdown
Contributor Author

dzou commented Jul 7, 2020

@normanmaurer - Thanks norman, I just finished signing it.

@normanmaurer normanmaurer merged commit e04803d into netty:4.1 Jul 8, 2020
@normanmaurer
Copy link
Copy Markdown
Member

@dzou thanks a lot

normanmaurer pushed a commit that referenced this pull request Jul 8, 2020
**Motivation:**

We are interested in building Netty libraries with Ahead-of-time compilation with GraalVM. We saw there was [prior work done on this](https://github.com/netty/netty/search?q=graalvm&unscoped_q=graalvm). We want to introduce a change which will unblock GraalVM support for applications using netty and `netty-tcnative`.

This solves the error [that some others encounter](https://github.com/oracle/graal/search?q=UnsatisfiedLinkError+sslOpCipherServerPreference&type=Issues):

```
Exception in thread "main" java.lang.UnsatisfiedLinkError: io.grpc.netty.shaded.io.netty.internal.tcnative.NativeStaticallyReferencedJniMethods.sslOpCipherServerPreference()I [symbol: Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference or Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference__]
```

**Modification:**

The root cause of the issue is that in the tcnative libraries, the [`SSL.java` class](https://github.com/netty/netty-tcnative/blob/783a8b6b69028ef947354cd3c54910dbf519c225/openssl-dynamic/src/main/java/io/netty/internal/tcnative/SSL.java#L67) references a native call in the static initialization of the class - the method `sslOpCipherServerPreference()` on line 67 is used to initialize the static variable. But you see that `OpenSsl` also uses[ `SSL.class` to check if `netty-tcnative` is on the classpath before loading the native library](https://github.com/netty/netty/blob/cbe238a95bab238455e3ff1849d450a9836f39ef/handler/src/main/java/io/netty/handler/ssl/OpenSsl.java#L123). 

So this is the problem because in ahead-of-time compilation, when it references the SSL class, it will try to load those static initializers and make the native library call, but it cannot do so because the native library has not been loaded yet since the `SSL` class is being referenced to check if the library should be loaded in the first place.

**Solution:** So the proposed solution here is to just choose a different class in the `tcnative` package which does not make a native library call during static initialization. I just chose `SSLContext` if this is OK.

This should have no negative effects other than unblocking the GraalVM use-case.

**Result:**

It fixes the unsatisfied link error. It will fix error for future users; it is a common error people encounter.
@normanmaurer normanmaurer added this to the 4.1.51.Final milestone Jul 8, 2020
Kvicii pushed a commit to Kvicii/netty that referenced this pull request Jul 10, 2020
* '4.1' of github.com:netty/netty:
  [maven-release-plugin] prepare for next development iteration
  [maven-release-plugin] prepare release netty-4.1.51.Final
  Correctly return NEED_WRAP if we produced some data even if we could not consume any during SSLEngine.wrap(...) (netty#10396)
  Modify OpenSSL native library loading to accommodate GraalVM (netty#10395)
Kvicii pushed a commit to Kvicii/netty that referenced this pull request Jul 20, 2020
* '4.1-read' of github.com:Kvicii/netty: (43 commits)
  Make the TLSv1.3 check more robust and not depend on the Java version… (netty#10409)
  Reduce the scope of synchronized block in PoolArena (netty#10410)
  Add IndexOutOfBoundsException error message (netty#10405)
  Add default handling for switch statement (netty#10408)
  Review PooledByteBufAllocator in respect of jemalloc 4.x changes and update allocate algorithm.(netty#10267)
  Support session cache for client and server when using native SSLEngine implementation (netty#10331)
  Simple fix typo (netty#10403)
  Eliminate a redundant method call in HpackDynamicTable.add(...) (netty#10399)
  jdk.tls.client.enableSessionTicketExtension must be respected by OPENSSL and OPENSSL_REFCNT SslProviders (netty#10401)
  重新编译4.1分支
  [maven-release-plugin] prepare for next development iteration
  [maven-release-plugin] prepare release netty-4.1.51.Final
  Correctly return NEED_WRAP if we produced some data even if we could not consume any during SSLEngine.wrap(...) (netty#10396)
  Modify OpenSSL native library loading to accommodate GraalVM (netty#10395)
  Update to netty-tcnative 2.0.31.Final and make SslErrorTest more robust (netty#10392)
  Add option to HttpObjectDecoder to allow duplicate Content-Lengths (netty#10349)
  Add detailed error message corresponding to the IndexOutOfBoundsException while calling getEntry(...) (netty#10386)
  Do not report ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify as blocking call (netty#10387)
  Add unit test for HpackDynamicTable. (netty#10389)
  netty用于测试的内嵌Channel | ChannelInboundHandler主要方法
  ...
ihanyong pushed a commit to ihanyong/netty that referenced this pull request Jul 31, 2020
…0395)

**Motivation:**

We are interested in building Netty libraries with Ahead-of-time compilation with GraalVM. We saw there was [prior work done on this](https://github.com/netty/netty/search?q=graalvm&unscoped_q=graalvm). We want to introduce a change which will unblock GraalVM support for applications using netty and `netty-tcnative`.

This solves the error [that some others encounter](https://github.com/oracle/graal/search?q=UnsatisfiedLinkError+sslOpCipherServerPreference&type=Issues):

```
Exception in thread "main" java.lang.UnsatisfiedLinkError: io.grpc.netty.shaded.io.netty.internal.tcnative.NativeStaticallyReferencedJniMethods.sslOpCipherServerPreference()I [symbol: Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference or Java_io_grpc_netty_shaded_io_netty_internal_tcnative_NativeStaticallyReferencedJniMethods_sslOpCipherServerPreference__]
```

**Modification:**

The root cause of the issue is that in the tcnative libraries, the [`SSL.java` class](https://github.com/netty/netty-tcnative/blob/783a8b6b69028ef947354cd3c54910dbf519c225/openssl-dynamic/src/main/java/io/netty/internal/tcnative/SSL.java#L67) references a native call in the static initialization of the class - the method `sslOpCipherServerPreference()` on line 67 is used to initialize the static variable. But you see that `OpenSsl` also uses[ `SSL.class` to check if `netty-tcnative` is on the classpath before loading the native library](https://github.com/netty/netty/blob/cbe238a95bab238455e3ff1849d450a9836f39ef/handler/src/main/java/io/netty/handler/ssl/OpenSsl.java#L123). 

So this is the problem because in ahead-of-time compilation, when it references the SSL class, it will try to load those static initializers and make the native library call, but it cannot do so because the native library has not been loaded yet since the `SSL` class is being referenced to check if the library should be loaded in the first place.

**Solution:** So the proposed solution here is to just choose a different class in the `tcnative` package which does not make a native library call during static initialization. I just chose `SSLContext` if this is OK.

This should have no negative effects other than unblocking the GraalVM use-case.

**Result:**

It fixes the unsatisfied link error. It will fix error for future users; it is a common error people encounter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants