Description

Unified variables

This PR supports an optional variables file (.vars.{ENV}.yaml) under every dataset folder, that can contain all the values of variables needed for that dataset's infra and the pipeline configurations.

The namespaces supported in the YAML file are

• infra: a set of key-value pairs that are copied as Terraform variables under infra/terraform.tfvars
• pipelines: a JSON object that contains dataset-specific variables (Airflow variables), copied to .{env}/datasets/{DATASET}/pipelines/{dataset}_variables.json

Because these YAML files might contain sensitive information, they aren't checked into the repo.

Support for IAM policies for GCS buckets and BQ datasets

This PR also adds support for adding IAM policies into the Terraform resource definitions of GCS buckets and BQ datasets using the YAML variables above. The convention is to use

infra:
  iam_policies:
    storage_buckets:
      bucket_name:
        - role: roles/storage.objectViewer
          members:
            - user:some-user@google.com
    bigquery_datasets:
      dataset_name:
        - role: roles/bigquery.dataViewer
          members:
            - allAuthenticatedUsers
            - user:another-user@example.com

in the YAML variables file to associate a list of IAM roles to a specific GCS bucket or BQ dataset. These IAM roles will be included in the generated Terraform files as IAM policy resources.

Checklist

Note: If an item applies to you, all of its sub-items must be fulfilled

• (Required) This pull request is appropriately labeled
• Please merge this pull request after it's approved
• I'm adding or editing a feature
  • I have updated the README accordingly
  • I have added tests for the feature
• I'm adding or editing a dataset
  • The Google Cloud Datasets team is aware of the proposed dataset
  • I put all my code inside datasets/<DATASET_NAME> and nothing outside of that directory
• I'm adding/editing documentation
• I'm submitting a bugfix
  • I have added tests to my bugfix (see the tests folder)
• I'm refactoring or cleaning up some code