Exemption Request: no changes in README or integration tests needed.

One of the authors of the new L2 here - We talked about this during RFC and implementation phases as a potential anti-pattern. Can you share why you need extra jars for a python job?

Hi Natalie, we need to use the spark-xml package in order to read XML files in Spark v3 (as you probably know, this package will be included in Spark v4). This package must be provided via the extraJars parameter, because AWS Glue does not accept installing packages via --conf spark.jars.packages=<maven coordinates>.

Thanks for the extra clarification. Let me get with the Glue service team; it sounds like this may be more of a Glue feature request than something we should work around in the L2 construct. Stay tuned.

+1 to this. Some libraries that provide additional spark capabilities require a jar, even if one is actually using spark via python (pyspark).

Here's a chatgpt-generated list of examples https://chatgpt.com/share/67a8e12d-ccd8-800e-a641-75e58db91d7b

We had some internal discussions and (in addition to the data here) decided this is a valid use case. But we should add them to all 3 PySpark job types.

@gontzalm Would you be able to add this change to all 3 pyspark job types?

not from CDK team, but with the discussion I started in #33356, I would suggest also supporting extraJarsFirst prop for setting --user-jars-first as both the --extra-jars and --user-jars-first tend to go together.

@humanzz I think that's a good point, but I don't think it's worth blocking to merge this PR. If anyone is interested in contributing extraJarsFirst feature, feel free to tag me for a review!

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.16%. Comparing base (6f1aa80) to head (2bbebe1).
Report is 1 commits behind head on main.

@@           Coverage Diff           @@
##             main   #33238   +/-   ##
=======================================
  Coverage   82.16%   82.16%           
=======================================
  Files         119      119           
  Lines        6857     6857           
  Branches     1157     1157           
=======================================
  Hits         5634     5634           
  Misses       1120     1120           
  Partials      103      103           

This pull request has been removed from the queue for the following reason: pull request branch update failed.

The pull request can't be updated

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

@mergify update

update

❌ Mergify doesn't have permission to update

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 2bbebe1
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.