Skip to content

The operator GCS to BigQuery fails to download the schema because download is called with the wrong order of parameters. #12439

@owlphi

Description

@owlphi

Version: Airflow: apache-airflow-backport-providers-google==2020.11.13

Problem

The operator gcs to bigquery fails because it calls the method download with positional parameters in the wrong order.
It calls in this way gcs_hook.download(self.bucket, self.schema_object). See more in the line

The signature for the download is gcs_hook.download(self, object_name, bucket_name). See more in the line

Solution

  1. Change the gcs_to_bigquery operator so it calls the method download with the keyword parameters rather than positional.
  2. Change the gcs_to_bigquery operator so it calls the method download with the right order of the parameters.
  3. Change the signature of download method in the gcs_hook to be def download(self, bucket_name, object_name, filename) rather than def dowload(self, object_name, bucket_name, filename)

I would prefer the option 3 because the method upload start with bucket_name see line and it makes sense to follow the same convention. Also most of the method in the gcs hook start with bucket name.

Also in Airflow 1.10.12 the method download starts with bucket_name and it would make sense not to change the order of the parameters (even if it change the name of the parameters). See here

Issues Related

This issue is related with #12335. It was closed but it was not fixed. I left a comments but there was not reply.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:bugThis is a clearly a bugprovider:googleGoogle (including GCP) related issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions