Package apache-airflow-providers-apache-spark
Release: 6.1.0
This is a provider package for apache.spark provider. All classes for this provider package
are in airflow.providers.apache.spark python package.
You can find package information and changelog for the provider in the documentation.
You can install this package on top of an existing Airflow installation (see Requirements below
for the minimum Airflow version supported) via
pip install apache-airflow-providers-apache-spark
The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
| PIP package | Version required |
|---|---|
apache-airflow |
>=2.11.0 |
apache-airflow-providers-common-compat |
>=1.12.0 |
pyspark-client |
>=4.0.0 |
grpcio-status |
>=1.67.0 |
requests |
>=2.32.0 |
tenacity |
>=8.3.0 |
Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified providers in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
pip install apache-airflow-providers-apache-spark[cncf.kubernetes]| Dependent package | Extra |
|---|---|
| apache-airflow-providers-cncf-kubernetes | cncf.kubernetes |
| apache-airflow-providers-common-compat | common.compat |
| Extra | Dependencies |
|---|---|
cncf.kubernetes |
apache-airflow-providers-cncf-kubernetes>=7.4.0 |
openlineage |
apache-airflow-providers-openlineage |
pyspark |
pyspark>=4.0.0 |
The changelog for the provider package can be found in the changelog.