Run Spark ValidatesRunner

Run Spark StructuredStreaming ValidatesRunner

Run Spark ValidatesRunner Java 11

Run Spark Runner Tpcds Tests

I'd also suggest to document this change in CHANGES.md. Particularly for anyone using the job-server, this is a breaking change as they have to upgrade their cluster version.

Codecov Report

Merging #23805 (b8d6681) into master (8f12469) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #23805      +/-   ##
==========================================
- Coverage   71.15%   71.15%   -0.01%     
==========================================
  Files         861      861              
  Lines      104568   104568              
==========================================
- Hits        74404    74401       -3     
- Misses      28615    28618       +3     
  Partials     1549     1549              

Flags with carried forward coverage won't be shown. Click here to find out more.

see 8 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Run Spotless PreCommit

Run Spark ValidatesRunner

Run Spark StructuredStreaming ValidatesRunner

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Run Spark ValidatesRunner

@mosche All tests are passing now

@aromanenko-dev a few things are still left

• Please update the Spark version mentioned in the Spark runner docs. Also this change should be mentioned in CHANGES.md https://github.com/apache/beam/blame/master/website/www/site/content/en/documentation/runners/spark.md#L70

• What about the Spark versions used for compatibility tests? It currently contains 3.2.1, if we bump to 3.2.0 we should keep that. But what about 3.1.1? Should we add it there to ensure we stay compatible?
  https://github.com/apache/beam/blob/master/runners/spark/3/build.gradle#L36-L39

• The hadoop version tests won't reliably work work anymore as both dependencies are pulled. So it's unclear what gets actually used.

  hadoopVersions.each { kv ->
    configurations."hadoopVersion$kv.key" {
      resolutionStrategy {
        force "org.apache.hadoop:hadoop-common:$kv.value"
      }
    }
  }
  

• Similarly, VR tests are pulling hadoop-format as dependency and it's not clear what version of hadoop is effectively used to run the tests. We have to make sure we run tests enforcing the version used by Spark.

Reminder, please take a look at this pr: @damccorm

stop reviewer notifications

Run Python_Integration PreCommit

Run Spark ValidatesRunner

Run Spark StructuredStreaming ValidatesRunner

@mosche ping on this
As we discussed privately, hadoop-related things are not so important in this case and could be addressed out of this change.

Run Spark ValidatesRunner

Run Spark StructuredStreaming ValidatesRunner

Run Spark ValidatesRunner

Run SQL PreCommit