Skip to content

Fix import error on Python 3.10+#475

Merged
HyukjinKwon merged 1 commit intopy4j:masterfrom
shadchin:fix_python_310
Mar 17, 2022
Merged

Fix import error on Python 3.10+#475
HyukjinKwon merged 1 commit intopy4j:masterfrom
shadchin:fix_python_310

Conversation

@shadchin
Copy link
Copy Markdown
Contributor

@shadchin shadchin commented Mar 17, 2022

No description provided.

HyukjinKwon
HyukjinKwon reviewed Mar 17, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks man. Can we actually just avoid using Callable? It's only for assert. We can just check if it's a weakref or not in the assert.

@HyukjinKwon
Copy link
Copy Markdown
Member

HyukjinKwon commented Mar 17, 2022 •
edited
Loading

We should also set up the CI w/ Python 3.10 too .. I didn't realise that it's removed in Python 3.10.

@shadchin
Copy link
Copy Markdown
Contributor Author

shadchin commented Mar 17, 2022 •
edited
Loading

Maybe switch on callable() ? https://docs.python.org/3/library/functions.html#callable
UPD: this Python 3 only

@HyukjinKwon HyukjinKwon mentioned this pull request Mar 17, 2022
Closed
@HyukjinKwon HyukjinKwon merged commit a687435 into py4j:master Mar 17, 2022
@HyukjinKwon
Copy link
Copy Markdown
Member

HyukjinKwon commented Mar 17, 2022

I am adding @gabe-l-hart as a co-author since he investigated/reported this issue first, and made an almost similar fix w/ this one at #474.

HyukjinKwon pushed a commit that referenced this pull request Mar 17, 2022
@shadchin @gabe-l-hart @HyukjinKwon
Fix import error on Python 3.10+ (#475)
919170d 
Closes #474

Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
@HyukjinKwon HyukjinKwon mentioned this pull request Mar 17, 2022
Merged
@HyukjinKwon HyukjinKwon added this to the 0.10.9.5 milestone Mar 18, 2022
@HyukjinKwon HyukjinKwon mentioned this pull request Mar 18, 2022
Closed
HyukjinKwon added a commit to apache/spark that referenced this pull request Mar 18, 2022
@HyukjinKwon
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
97335ea 
### What changes were proposed in this pull request?

This PR is a retry of #35871 with bumping up the version to 0.10.9.5.
It was reverted because of Python 3.10 is broken, and Python 3.10 was not officially supported in Py4J.

In Py4J 0.10.9.5, the issue was fixed (py4j/py4j#475), and it added Python 3.10 support officially with CI set up (py4j/py4j#477).

### Why are the changes needed?

See #35871

### Does this PR introduce _any_ user-facing change?

See #35871

### How was this patch tested?

Py4J sets up Python 3.10 CI now, and I manually tested PySpark with Python 3.10 with this patch:

```bash
./bin/pyspark
```

```
import py4j
py4j.__version__
spark.range(10).show()
```

```
Using Python version 3.10.0 (default, Mar  3 2022 03:57:21)
Spark context Web UI available at http://172.30.5.50:4040
Spark context available as 'sc' (master = local[*], app id = local-1647571387534).
SparkSession available as 'spark'.
>>> import py4j
>>> py4j.__version__
'0.10.9.5'
>>> spark.range(10).show()
+---+
| id|
+---+
...
```

Closes #35907 from HyukjinKwon/SPARK-38563-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon added a commit to apache/spark that referenced this pull request Mar 18, 2022
@HyukjinKwon
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
7bb1d6f 
### What changes were proposed in this pull request?

This PR is a retry of #35871 with bumping up the version to 0.10.9.5.
It was reverted because of Python 3.10 is broken, and Python 3.10 was not officially supported in Py4J.

In Py4J 0.10.9.5, the issue was fixed (py4j/py4j#475), and it added Python 3.10 support officially with CI set up (py4j/py4j#477).

### Why are the changes needed?

See #35871

### Does this PR introduce _any_ user-facing change?

See #35871

### How was this patch tested?

Py4J sets up Python 3.10 CI now, and I manually tested PySpark with Python 3.10 with this patch:

```bash
./bin/pyspark
```

```
import py4j
py4j.__version__
spark.range(10).show()
```

```
Using Python version 3.10.0 (default, Mar  3 2022 03:57:21)
Spark context Web UI available at http://172.30.5.50:4040
Spark context available as 'sc' (master = local[*], app id = local-1647571387534).
SparkSession available as 'spark'.
>>> import py4j
>>> py4j.__version__
'0.10.9.5'
>>> spark.range(10).show()
+---+
| id|
+---+
...
```

Closes #35907 from HyukjinKwon/SPARK-38563-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 97335ea)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon added a commit to apache/spark that referenced this pull request Mar 18, 2022
@HyukjinKwon
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
864b89b 
This PR is a retry of #35871 with bumping up the version to 0.10.9.5.
It was reverted because of Python 3.10 is broken, and Python 3.10 was not officially supported in Py4J.

In Py4J 0.10.9.5, the issue was fixed (py4j/py4j#475), and it added Python 3.10 support officially with CI set up (py4j/py4j#477).

See #35871

See #35871

Py4J sets up Python 3.10 CI now, and I manually tested PySpark with Python 3.10 with this patch:

```bash
./bin/pyspark
```

```
import py4j
py4j.__version__
spark.range(10).show()
```

```
Using Python version 3.10.0 (default, Mar  3 2022 03:57:21)
Spark context Web UI available at http://172.30.5.50:4040
Spark context available as 'sc' (master = local[*], app id = local-1647571387534).
SparkSession available as 'spark'.
>>> import py4j
>>> py4j.__version__
'0.10.9.5'
>>> spark.range(10).show()
+---+
| id|
+---+
...
```

Closes #35907 from HyukjinKwon/SPARK-38563-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 97335ea)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@shadchin shadchin deleted the fix_python_310 branch March 19, 2022 05:15
kazuyukitanimura pushed a commit to kazuyukitanimura/spark that referenced this pull request Aug 10, 2022
@HyukjinKwon @viirya
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
e3b151d 
This PR is a retry of apache#35871 with bumping up the version to 0.10.9.5.
It was reverted because of Python 3.10 is broken, and Python 3.10 was not officially supported in Py4J.

In Py4J 0.10.9.5, the issue was fixed (py4j/py4j#475), and it added Python 3.10 support officially with CI set up (py4j/py4j#477).

See apache#35871

See apache#35871

Py4J sets up Python 3.10 CI now, and I manually tested PySpark with Python 3.10 with this patch:

```bash
./bin/pyspark
```

```
import py4j
py4j.__version__
spark.range(10).show()
```

```
Using Python version 3.10.0 (default, Mar  3 2022 03:57:21)
Spark context Web UI available at http://172.30.5.50:4040
Spark context available as 'sc' (master = local[*], app id = local-1647571387534).
SparkSession available as 'spark'.
>>> import py4j
>>> py4j.__version__
'0.10.9.5'
>>> spark.range(10).show()
+---+
| id|
+---+
...
```

Closes apache#35907 from HyukjinKwon/SPARK-38563-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 97335ea)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

0.10.9.5

Development

Successfully merging this pull request may close these issues.

2 participants

@shadchin @HyukjinKwon