What needs to happen?

The Spark 2 has been deprecated for a while. Even more significant, it's actually in a badly broken state since long due to classpath issues when running on a Spark cluster (#23568).

Before Spark 2 support can be removed, all documentation and dependent components should be migrated to use Spark 3.

Notes:

• Test examples against Spark 3
• Java Quickstart on Spark 3
• Remove obsolete sparkRunner from Hadoop-format
• Use Spark 3 as default for portable Spark job-server.
• Validate Go & Python against Spark 3
• Bump metrics-core to match version used by Spark 3 (4.1.1)
• Nexmark, Tpcds, Load tests
• Jpms tests (cleanup build)

Issue Priority

Priority: 2

Issue Component

Component: runner-spark