Fix MyPy for Google Bigquery #20329
Fix MyPy for Google Bigquery #20329
Conversation
5830441 to
869ba74
Compare
869ba74 to
c38d741
Compare
c38d741 to
1fa04d2
Compare
|
One of the last ones for google (I think we have 50 left after this is merged :)) |
|
likewise looking for reviewer :) |
1fa04d2 to
89b27f7
Compare
There was a problem hiding this comment.
I recently changed gcs_hooks's return value as the solution to an issue.
So if you are not passing the filename to the download() method it will return only bytes otherwise file name.
Therefore the variable schema_fields_bytes_or_string always will be as bytes, but unfortunately gcs_hook.download's return type is bytes | str.
So how do you think could we skip this check if hasattr(schema_fields_bytes_or_string, 'decode')?
Also, maybe we could use here something like that?
gcs_hook.get_conn().bucket(gcs_bucket).blob(blob_name=gcs_object).download_as_bytes()
There was a problem hiding this comment.
Good point. I changed it in the way that I created a separate method in the hook that only returns bytes (and does not pass filename) and cast it to bytes - with appropriate comment.
89b27f7 to
a61dbee
Compare
|
Would love to merge all google stuff ! |
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Part of #19891
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.