Skip to content

Conversation

@r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Oct 6, 2022

What this does

  • Removes <2 limit on google-cloud-storage for google provider

Why

  • newer versions of google-cloud-storage are required from some other python libraries
  • the breaking changes made to 2.0 was the removal of python 2.x support which airflow no longer supports regardless
  • the breaking changes made to 2.1 was the removal of python 3.6 support which airflow no longer supports regardless

Misc notes

  • Made my best attempt to update all places where this could be referenced, unable to run pre-commit --run-all due to the following
  • I attempted to use vscode in open in containers but hit error 1
Workspace folder not specified in devcontainer.json.
  • After resolving error 1 by adding workspace_folder=/var/lib/docker/codespacemount/workspace/airflow I ran into error 2
[2022-10-06T18:58:11.339Z] Stop (53 ms): Run in container: /bin/sh
[2022-10-06T18:58:11.339Z] Start: Run in container: cat /etc/passwd
[2022-10-06T18:58:11.339Z] Stdin closed!
[2022-10-06T18:58:11.340Z] Shell server terminated (code: 1, signal: null)

Error response from daemon: Container 32d84eed30c292a7bd8b42d3b9a58e241e435f51388dbd5e437d20c32c108eef is not running

@potiuk
Copy link
Member

potiuk commented Oct 6, 2022

@r-richmond - you do not need to use vscode/codespaces to run pre-commits (and there is no need usually for you to run --all-files). You should just install pre-commit where your code is checked-out and run pre-commit install then only necessary pre-commits will be run automatically when you run commit. Pre-commits are supposed to be run locally.

@r-richmond
Copy link
Contributor Author

and there is no need usually for you to run --all-files

👍 wasn't sure given the nature of the pr, ty for the info

you do not need to use vscode/codespaces to run pre-commits ... Pre-commits are supposed to be run locally.

100% correct. I just wanted to avoid installing all the airflow dependencies locally (lots of them, multiple sub-dependencies mysql etc.) + the promise of the pre-configured workspace via vs-code remote container dev was too shiny to resist trying for this project.

I'm hoping I'll get dumb lucky and there are no other requirements for this type of pr and it will pass tests if not I'm happy to let someone else take my branch / pr over. & lastly

  1. Do you see any issues with this modification in general given the points I outlined above?

@potiuk
Copy link
Member

potiuk commented Oct 6, 2022

100% correct. I just wanted to avoid installing all the airflow dependencies locally (lots of them, multiple sub-dependencies mysql etc.) + the promise of the pre-configured workspace via vs-code remote container dev was too shiny to resist trying for this project.

Understood. We have the devcontainer/codespaces implemented but we do not actively check if they continue to work - because vast majority of our contributors does not really have codespaces access (it is not free for most people). It should generally work, so maybe would be a good thing to try to fix it. If you would like to, that would be great :).

Do you see any issues with this modification in general given the points I outlined above?

Nope. Just one test failed randomly (but I believe it was a temporary/random failure. so I re-run it.

@potiuk
Copy link
Member

potiuk commented Oct 6, 2022

It would also be great if you could to run System tests for GCS to check if they still work - I assume you have access to a GCS cloudl ? (see TESTING.rst for information about system tests)

@potiuk
Copy link
Member

potiuk commented Oct 6, 2022

LGTM. If anyone (maybe you @r-richmond) can confirm that system tests are working for it ? Then we can merge it.

@r-richmond
Copy link
Contributor Author

We have the devcontainer/codespaces implemented but we do not actively check if they continue to work - because vast majority of our contributors does not really have codespaces access (it is not free for most people).

  1. Codespaces isn't free but vscode dev-containers are :).
  2. Yea I'll give it a go now since it would make it a lot easier to get the dev environment setup. (I've dug more into the failure logs and am reading about a file not found but I'll move that convo to slack / off this pr. /entrypoint: line 6: /workspaces/airflow/scripts/in_container/_in_container_script_init.sh: No such file or directory )

(maybe you r-richmond) can confirm that system tests are working for it

If I can figure out the dev-containers I'll give that a go too. But in the meantime if it is easy for someone else to do this I would be greatly appreciative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants