[SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql` #34153

LuciferYang · 2021-09-30T04:09:38Z

What changes were proposed in this pull request?

In order to pass the UTs related to sql/core and dependent modules, this pr mainly does the following change:

Add a new property named extraJavaTestArgs to pom.xml, It includes all --add-opens configurations required for UTs with Java 17 and It also include -XX:+IgnoreUnrecognizedVMOptions to compatible with Java 8.
Add a new helper class named JavaModuleOptions to place constant and methods related to --add-opens
--add-opens options and -XX:+IgnoreUnrecognizedVMOptions is added by default when SparkSubmitCommandBuilder builds the submit command, this can help local-cluster related UTs pass and help users reduce the cost of using java 17
--add-opens options and -XX:+IgnoreUnrecognizedVMOptions is supplemented to spark.driver.extraJavaOptions and spark.executor.extraJavaOptions when init SparkContext, this also can help UTs pass and help users reduce the cost of using java 17

Why are the changes needed?

Pass Spark UTs with JDK 17

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the Jenkins or GitHub Action
Manual test mvn clean install -pl sql/core -am with Java 17, only select format_string('%0$s', 'Hello') in postgreSQL/text.sql failed due to the different behavior of Java 8 and Java 17

SparkQA · 2021-09-30T04:25:30Z

Test build #143745 has finished for PR 34153 at commit 09fcb3e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-30T04:30:17Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48256/

SparkQA · 2021-09-30T04:34:03Z

Test build #143746 has finished for PR 34153 at commit adc566d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-30T04:37:25Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48257/

SparkQA · 2021-09-30T04:40:29Z

Test build #143747 has finished for PR 34153 at commit 70fc3bf.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-09-30T04:43:39Z

core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala

+    "--add-opens java.base/java.net=ALL-UNNAMED",
+    "--add-opens java.base/java.io=ALL-UNNAMED",
+    "--add-opens java.base/java.util.concurrent=ALL-UNNAMED",
+    "--add-exports java.base/jdk.internal.util.random=ALL-UNNAMED")


Thank you, @LuciferYang !

BTW, I didn't realized that we need --add-exports. Could you explain a little more about this?

--add-exports is useless for the current pr , already remove it.

But It is needed in modules mllib-local and mllib, some code in test like mock[java.util.Random] need this --add-exports

pom.xml

dongjoon-hyun

Instead of adding jdk-17 profile, you can add like the following always. Last time, I learned that --add-opens= do the tricks on the JIRA you mentioned your idea.

-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.nio=ALL-UNNAMED ...

dongjoon-hyun · 2021-09-30T04:48:20Z

core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala

+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",


Let's use = instead of space like --add-opens=java.base/java.nio=ALL-UNNAMED.

SparkQA · 2021-09-30T04:51:23Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48258/

SparkQA · 2021-09-30T05:31:03Z

Test build #143748 has finished for PR 34153 at commit 2b60264.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-30T05:34:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48259/

SparkQA · 2021-09-30T06:32:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48262/

SparkQA · 2021-09-30T06:38:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48265/

LuciferYang · 2021-10-19T03:06:33Z

Do we need to wait on a resolution to the date difference here or treat it separately? would it mean we don't have to change the expected output?

I created a new Jira SPARK-37013 and #34153 try to fix the different string format behavior between Java 8 and Java 17, the relevant code has been removed from this pr.

At present, this pr only adds the default --add-opens options and -XX:+IgnoreUnrecognizedVMOptions where needed, and passed UTs except select format_string('%0$s', 'Hello') in postgreSQL/text.sql

LuciferYang · 2021-10-19T03:09:55Z

If this pr is merged, we can continue to verify other modules

SparkQA · 2021-10-19T03:26:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48857/

SparkQA · 2021-10-19T03:34:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48858/

SparkQA · 2021-10-19T04:19:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48858/

SparkQA · 2021-10-19T04:27:23Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48857/

SparkQA · 2021-10-19T05:19:35Z

Test build #144383 has finished for PR 34153 at commit e773fc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-19T05:27:34Z

Test build #144384 has finished for PR 34153 at commit 2dec62e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-19T06:50:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48871/

SparkQA · 2021-10-19T07:52:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48871/

SparkQA · 2021-10-19T08:57:11Z

Test build #144396 has finished for PR 34153 at commit c7265c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ExtractValue extends Expression with NullIntolerant

srowen · 2021-10-19T13:59:31Z

Merged to master

dongjoon-hyun · 2021-10-19T21:54:05Z

Thank you so much, @LuciferYang and @srowen and all!

LuciferYang · 2021-10-20T02:32:55Z

thanks all ~

…s` to drivers ### What changes were proposed in this pull request? This PR aims to make `StandaloneRestServer` add `JavaModuleOptions` to drivers by default. ### Why are the changes needed? Since Apache Spark 3.3.0 (SPARK-36796, #34153), `SparkContext` adds `JavaModuleOptions` by default. We had better add `JavaModuleOptions` when `StandaloneRestServer` receives submissions via REST API, too. Otherwise, it fails like the following if the users don't set it manually. **SUBMISSION** ```bash $ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh $ curl -s -k -XPOST http://yourserver:6066/v1/submissions/create \ --header "Content-Type:application/json;charset=UTF-8" \ --data '{ "appResource": "", "sparkProperties": { "spark.master": "local[2]", "spark.app.name": "Test 1", "spark.submit.deployMode": "cluster", "spark.jars": "/Users/dongjoon/APACHE/spark-release/spark-3.5.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.0.jar" }, "clientSparkVersion": "", "mainClass": "org.apache.spark.examples.SparkPi", "environmentVariables": {}, "action": "CreateSubmissionRequest", "appArgs": [] }' ``` **DRIVER `stderr` LOG** ``` Exception in thread "main" java.lang.reflect.InvocationTargetException ... at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module 0x6d7a93c9) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module 0x6d7a93c9 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42975 from dongjoon-hyun/SPARK-45197. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…s` to drivers ### What changes were proposed in this pull request? This PR aims to make `StandaloneRestServer` add `JavaModuleOptions` to drivers by default. ### Why are the changes needed? Since Apache Spark 3.3.0 (SPARK-36796, apache#34153), `SparkContext` adds `JavaModuleOptions` by default. We had better add `JavaModuleOptions` when `StandaloneRestServer` receives submissions via REST API, too. Otherwise, it fails like the following if the users don't set it manually. **SUBMISSION** ```bash $ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh $ curl -s -k -XPOST http://yourserver:6066/v1/submissions/create \ --header "Content-Type:application/json;charset=UTF-8" \ --data '{ "appResource": "", "sparkProperties": { "spark.master": "local[2]", "spark.app.name": "Test 1", "spark.submit.deployMode": "cluster", "spark.jars": "/Users/dongjoon/APACHE/spark-release/spark-3.5.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.0.jar" }, "clientSparkVersion": "", "mainClass": "org.apache.spark.examples.SparkPi", "environmentVariables": {}, "action": "CreateSubmissionRequest", "appArgs": [] }' ``` **DRIVER `stderr` LOG** ``` Exception in thread "main" java.lang.reflect.InvocationTargetException ... at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module 0x6d7a93c9) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module 0x6d7a93c9 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42975 from dongjoon-hyun/SPARK-45197. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

lachen-rms · 2024-09-10T07:51:15Z

Hi guys, is there today a spark version that fully supports java17, im trying to run locally a project that uses spark an java 17, scala version 2.12.15, spark 3.3.4 and is failing because trying to access internal java APIs.

It works when adding:

--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/sun.nio.cs=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens java.base/sun.security.action=ALL-UNNAMED --add-opens java.base/sun.util.calendar=ALL-UNNAMED --add-opens java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang.invoke=ALL-UNNAMED

srowen · 2024-09-10T12:25:24Z

(This isn't the place for general questions, but:)
Yes Java 17 has been supported for a while. You have to open access to internal packages.

lachen-rms · 2024-09-11T17:28:10Z

Hi @srowen thanks for replying, will stick into that. Have a good day.

Pass sql/core releated cases

09fcb3e

LuciferYang changed the title ~~[SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17~~ [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 Sep 30, 2021

github-actions bot added BUILD CORE SQL labels Sep 30, 2021

LuciferYang marked this pull request as draft September 30, 2021 04:09

SparkBuild.scala

adc566d

revert project/SparkBuild.scala

70fc3bf

LuciferYang changed the title ~~[WIP][SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17~~ [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 and Maven Sep 30, 2021

dongjoon-hyun reviewed Sep 30, 2021

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

dongjoon-hyun requested changes Sep 30, 2021

View reviewed changes

dongjoon-hyun reviewed Sep 30, 2021

View reviewed changes

add SparkBuild.sh

2b60264

LuciferYang added 3 commits September 30, 2021 13:36

fix comments

7e016b3

change SparkBuild

44c566f

fix utils

1761e74

LuciferYang added 2 commits September 30, 2021 14:54

add commnets

3b0f6ed

add commnets

0bb8df7

LuciferYang changed the title ~~[WIP][SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 and Maven~~ [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs use Maven with JDK 17 Sep 30, 2021

LuciferYang added 2 commits October 19, 2021 10:40

remove HashSet

e773fc4

revert change of SQLQueryTestSuite

2dec62e

LuciferYang changed the title ~~[SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs use Maven with JDK 17~~ [SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 Oct 19, 2021

LuciferYang changed the title ~~[SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17~~ [SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 except one case in postgreSQL/text.sql Oct 19, 2021

Merge branch 'upmaster' into SPARK-36796

c7265c9

srowen approved these changes Oct 19, 2021

View reviewed changes

srowen closed this in 3849340 Oct 19, 2021

xerial mentioned this pull request Jun 28, 2022

Support JDK17 msgpack/msgpack-java#660

Merged

edmondop mentioned this pull request Apr 24, 2023

Adding support for JDK 17 holdenk/spark-testing-base#380

Merged

dongjoon-hyun mentioned this pull request Sep 18, 2023

[SPARK-45197][CORE] Make StandaloneRestServer add JavaModuleOptions to drivers #42975

Closed

LuciferYang deleted the SPARK-36796 branch October 22, 2023 07:43


		object JavaModuleUtils {

		private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",

[SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 except one case in postgreSQL/text.sql #34153

[SPARK-36796][BUILD][CORE][SQL] Pass all sql/core and dependent modules UTs with JDK 17 except one case in postgreSQL/text.sql #34153

Uh oh!

Conversation

LuciferYang commented Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

dongjoon-hyun Sep 30, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Sep 30, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Sep 30, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Sep 30, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Sep 30, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

SparkQA commented Sep 30, 2021

Uh oh!

LuciferYang commented Oct 19, 2021

Uh oh!

LuciferYang commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

SparkQA commented Oct 19, 2021

Uh oh!

srowen commented Oct 19, 2021

Uh oh!

dongjoon-hyun commented Oct 19, 2021

Uh oh!

LuciferYang commented Oct 20, 2021

Uh oh!

lachen-rms commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

[SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql` #34153

[SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql` #34153

LuciferYang commented Sep 30, 2021 •

edited

Loading

lachen-rms commented Sep 10, 2024 •

edited

Loading