Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Sep 18, 2023

What changes were proposed in this pull request?

This PR aims to make StandaloneRestServer add JavaModuleOptions to drivers by default.

Why are the changes needed?

Since Apache Spark 3.3.0 (SPARK-36796, #34153), SparkContext adds JavaModuleOptions by default.

We had better add JavaModuleOptions when StandaloneRestServer receives submissions via REST API, too. Otherwise, it fails like the following if the users don't set it manually.

SUBMISSION

$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh
$ curl -s -k -XPOST http://yourserver:6066/v1/submissions/create \
    --header "Content-Type:application/json;charset=UTF-8" \
    --data '{
      "appResource": "",
      "sparkProperties": {
        "spark.master": "local[2]",
        "spark.app.name": "Test 1",
        "spark.submit.deployMode": "cluster",
        "spark.jars": "/Users/dongjoon/APACHE/spark-release/spark-3.5.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.0.jar"
      },
      "clientSparkVersion": "",
      "mainClass": "org.apache.spark.examples.SparkPi",
      "environmentVariables": {},
      "action": "CreateSubmissionRequest",
      "appArgs": []
    }'

DRIVER stderr LOG

Exception in thread "main" java.lang.reflect.InvocationTargetException
...
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module @0x6d7a93c9) cannot access class sun.nio.ch.DirectBuffer
(in module java.base) because module java.base does not export sun.nio.ch
to unnamed module @0x6d7a93c9

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs with the newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-45197][CORE] Make StandaloneRestServer add JavaModuleOptions to drivers [SPARK-45197][CORE] Make StandaloneRestServer add JavaModuleOptions to drivers Sep 18, 2023
@github-actions github-actions bot added the CORE label Sep 18, 2023
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This private method arguments are revised for better testing by taking advantage of dependency injection pattern.

@dongjoon-hyun
Copy link
Member Author

Thank you always, @yaooqinn and @LuciferYang !

@dongjoon-hyun
Copy link
Member Author

Merged to master.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-45197 branch September 18, 2023 11:10
viirya pushed a commit to viirya/spark-1 that referenced this pull request Oct 19, 2023
…s` to drivers

### What changes were proposed in this pull request?

This PR aims to make `StandaloneRestServer` add `JavaModuleOptions` to drivers by default.

### Why are the changes needed?

Since Apache Spark 3.3.0 (SPARK-36796, apache#34153), `SparkContext` adds `JavaModuleOptions` by default.

We had better add `JavaModuleOptions` when `StandaloneRestServer` receives submissions via REST API, too. Otherwise, it fails like the following if the users don't set it manually.

**SUBMISSION**
```bash
$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh
$ curl -s -k -XPOST http://yourserver:6066/v1/submissions/create \
    --header "Content-Type:application/json;charset=UTF-8" \
    --data '{
      "appResource": "",
      "sparkProperties": {
        "spark.master": "local[2]",
        "spark.app.name": "Test 1",
        "spark.submit.deployMode": "cluster",
        "spark.jars": "/Users/dongjoon/APACHE/spark-release/spark-3.5.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.0.jar"
      },
      "clientSparkVersion": "",
      "mainClass": "org.apache.spark.examples.SparkPi",
      "environmentVariables": {},
      "action": "CreateSubmissionRequest",
      "appArgs": []
    }'
```

**DRIVER `stderr` LOG**
```
Exception in thread "main" java.lang.reflect.InvocationTargetException
...
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module 0x6d7a93c9) cannot access class sun.nio.ch.DirectBuffer
(in module java.base) because module java.base does not export sun.nio.ch
to unnamed module 0x6d7a93c9
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs with the newly added test case.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42975 from dongjoon-hyun/SPARK-45197.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants