-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Description
Version: Airflow: apache-airflow-backport-providers-google==2020.11.13
Problem
The operator gcs to bigquery fails because it calls the method download with positional parameters in the wrong order.
It calls in this way gcs_hook.download(self.bucket, self.schema_object). See more in the line
The signature for the download is gcs_hook.download(self, object_name, bucket_name). See more in the line
Solution
- Change the gcs_to_bigquery operator so it calls the method download with the keyword parameters rather than positional.
- Change the gcs_to_bigquery operator so it calls the method download with the right order of the parameters.
- Change the signature of download method in the gcs_hook to be
def download(self, bucket_name, object_name, filename)rather thandef dowload(self, object_name, bucket_name, filename)
I would prefer the option 3 because the method upload start with bucket_name see line and it makes sense to follow the same convention. Also most of the method in the gcs hook start with bucket name.
Also in Airflow 1.10.12 the method download starts with bucket_name and it would make sense not to change the order of the parameters (even if it change the name of the parameters). See here
Issues Related
This issue is related with #12335. It was closed but it was not fixed. I left a comments but there was not reply.