Description

For more convenient collaboration, we need a consistent set of variables to be shared across developers working on the same datasets or pipelines. To do this, we're introducing a feature to compare locally set Airflow variables with those defined in Cloud Composer (remote).

Cases that are covered include

• If local variables exist but remote variables do not, import the local variables into Cloud Composer.
• If both local and remote variables exist, fetch the remote variable and ask the user for a strategy: one of merge, use remote or use local.
• If both local and remote variables are empty, skip the variable import step into Cloud Composer.
• If remote variables exist but local variables are empty, fetch the remote variable set the local vars to be equal to it. In this way, the user will have the same exact variables as used in the Composer environment.

I've also added an optimization in the tests: skip formatting the code (takes a few seconds for every test case) for the setup step in test_deploy_dag.py.

Checklist

Note: If an item applies to you, all of its sub-items must be fulfilled

• (Required) This pull request is appropriately labeled
• Please merge this pull request after it's approved
• I'm adding or editing a feature
  • I have updated the README accordingly
  • I have added tests for the feature
• I'm refactoring or cleaning up some code