[SPARK-37632][PYTHON] Drop code that targets Python < 3.7 by zero323 · Pull Request #34886 · apache/spark

zero323 · 2021-12-13T23:13:46Z

What changes were proposed in this pull request?

This PR drops parts of code that target Python < 3.7 including:

Components required for type hint compatibility.
Old style (Python 2) class declarations.
Dictionary-related code that was intended for drop after removal of Python 3.5 support.

Why are the changes needed?

With SPARK-36145 we remove support for Python 3.6 ‒ we should remove all dead code as well, to improve style and reduce maintenance overhead.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

zero323 · 2021-12-13T23:14:02Z

FYI @HyukjinKwon

SparkQA · 2021-12-13T23:48:57Z

Test build #146148 has finished for PR 34886 at commit e4db2f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-13T23:59:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50621/

HyukjinKwon · 2021-12-14T00:44:59Z

Looks pretty good otherwise.

SparkQA · 2021-12-14T01:00:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50621/

SparkQA · 2021-12-14T01:29:38Z

Test build #146152 has finished for PR 34886 at commit c2a78a8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-14T02:00:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50625/

SparkQA · 2021-12-14T03:00:25Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50625/

SparkQA · 2021-12-14T11:16:56Z

Test build #146181 has finished for PR 34886 at commit 6af93b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
(defaultdict(<class 'list'>,

SparkQA · 2021-12-14T11:19:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50654/

itholic · 2021-12-15T06:12:26Z

Otherwise, looks fine to me when I roughly scanned the whole changes.

HyukjinKwon

Looks good. cc @ueshin FYI

HyukjinKwon · 2021-12-15T10:35:17Z

Oh, let's also:

remove https://github.com/apache/spark/blob/master/python/setup.py#L279, and
fix https://github.com/apache/spark/blob/master/python/pyspark/context.py#L309-L312 together
and .. maybe this "Python 3.6 support was removed in Spark 3.3.0." too at https://github.com/apache/spark/blob/master/docs/rdd-programming-guide.md. Since this is documentation, we can do it separately though.
I think we can also safely remove: https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L375-L376
and also remove https://github.com/apache/spark/blob/master/python/pyspark/pandas/series.py#L376-L377
https://github.com/apache/spark/blob/master/python/pyspark/sql/tests/test_streaming.py#L376 "Those foreach tests are failed in Python 3.6 and macOS High Sierra by defined rules" -> "Those foreach tests are failed in macOS High Sierra by defined rules"

SparkQA · 2021-12-15T10:54:44Z

Test build #146228 has finished for PR 34886 at commit 084a1cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-15T11:11:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50702/

…mment

SparkQA · 2021-12-15T11:53:04Z

Test build #146230 has finished for PR 34886 at commit 33f1901.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-15T12:06:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50704/

SparkQA · 2021-12-15T12:09:11Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50702/

SparkQA · 2021-12-15T13:06:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50704/

ueshin · 2021-12-15T21:55:47Z

Additionally, we might want to reduce the usage of OrderedDict, mostly in pyspark.pandas module, in a separate PR. cc @HyukjinKwon @itholic @xinrong-databricks

ueshin

LGTM, pending tests.

SparkQA · 2021-12-15T21:57:25Z

Test build #146244 has finished for PR 34886 at commit f9f0cfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-15T22:18:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50718/

SparkQA · 2021-12-15T23:18:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50718/

HyukjinKwon · 2021-12-16T01:52:54Z

Thanks @zero323 for doing this!!!

Merged to master.

zero323 added 6 commits December 13, 2021 22:23

Drop Python 3.5 conditional

0144a5a

Switch to new style classes

1af9bcc

Drop pre 3.7 code from pandas.typedef.typehints

7b0a091

Drop pre 3.7 code from sql.pandas.typehints

0ff703d

Drop pre 3.7 code from pandas.frame

88e4ef3

Drop pre 3.7 code from pandas.groupby

e4db2f4

github-actions Bot added CORE DSTREAM ML MLLIB PYTHON SQL STRUCTURED STREAMING labels Dec 13, 2021

HyukjinKwon reviewed Dec 14, 2021

View reviewed changes

Comment thread python/pyspark/ml/stat.py

HyukjinKwon reviewed Dec 14, 2021

View reviewed changes

Comment thread python/pyspark/sql/pandas/typehints.py

Bring back nested getattr

c2a78a8

zero323 added 2 commits December 14, 2021 11:05

Bring back nested getattr in pyspark.sql.pandas.typehints

4025cc3

Update pandas.groupby.normalize_keyword_aggregation doctest

6af93b8

zero323 marked this pull request as ready for review December 14, 2021 13:03

zero323 changed the title ~~[WIP][SPARK-37632][PYTHON] Drop code that targets Python < 3.7~~ [SPARK-37632][PYTHON] Drop code that targets Python < 3.7 Dec 14, 2021

Update description of normalize_keyword_aggregation

084a1cf

HyukjinKwon approved these changes Dec 15, 2021

View reviewed changes

HyukjinKwon reviewed Dec 15, 2021

View reviewed changes

Comment thread python/pyspark/pandas/typedef/typehints.py

zero323 added 4 commits December 15, 2021 11:58

Drop Python 3.6 from classifiers

fc4f344

Drop Python < 3.7 warning from context

04c50d1

Drop dict ordering notes

29aafcf

Don't reference Python 3.6 in flaky tests

1ae8792

Update python/pyspark/sql/pandas/typehints.check_tuple_annotataion co…

33f1901

…mment

ueshin reviewed Dec 15, 2021

View reviewed changes

Comment thread python/pyspark/pandas/frame.py

Comment thread python/pyspark/pandas/series.py

Comment thread python/setup.py

zero323 added 2 commits December 15, 2021 21:42

Drop GenericMeta.__getitem__ workaround

424cd9c

Update python_requires

f9f0cfd

ueshin approved these changes Dec 15, 2021

View reviewed changes

HyukjinKwon approved these changes Dec 16, 2021

View reviewed changes

HyukjinKwon closed this in 6fb417e Dec 16, 2021

zero323 deleted the SPARK-37632 branch December 16, 2021 09:13

harupy mentioned this pull request Dec 17, 2021

Use python 3.7 when running dev cross version tests mlflow/mlflow#5180

Merged

29 tasks

Conversation

zero323 commented Dec 13, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

zero323 commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

SparkQA commented Dec 14, 2021

Uh oh!

itholic commented Dec 15, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ueshin commented Dec 15, 2021

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

HyukjinKwon commented Dec 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon commented Dec 15, 2021 •

edited

Loading