(work in progress)

related: #31137

Summary

This PR adds support for retrieving data from Google Cloud Storage buckets with Requester Pays enabled.

Adds optional user_project argument to below operators and GCSHook methods.

Notes:

• Add optional user_project parameter to each GCSHook method, and pass to calls to client.bucket
• For operations that span multiple buckets, only one user_project can be supplied (all operations charged to this project)

Testing

Amazon provider:

• GCSToS3Operator

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_DEFAULT_REGION=
export GOOGLE_APPLICATION_CREDENTIALS=
export GCP_PROJECT_ID=
pytest --system amazon tests/system/providers/amazon/aws/example_gcs_to_s3.py

Google provider:

• GCSSynchronizeBucketsOperator
• GCSToLocalFilesystemOperator
• GCSListObjectsOperator
• GCSToGCSOperator
• GCSDeleteObjectsOperator
• GCSDeleteBucketOperator
• LocalFilesystemToGCSOperator

export SYSTEM_TESTS_ENV_ID=
export SYSTEM_TESTS_GCP_PROJECT=
export GOOGLE_APPLICATION_CREDENTIALS=
pytest --system google tests/system/providers/google/cloud/gcs/example_gcs_copy_delete.py
pytest --system google tests/system/providers/google/cloud/gcs/example_gcs_upload_download.py

• GCSToGoogleDriveOperator (requires: Google Workspace Admin account, Google Drive API)

export SYSTEM_TESTS_ENV_ID=
export SYSTEM_TESTS_GCP_PROJECT=
export GOOGLE_APPLICATION_CREDENTIALS=
pytest --system google tests/system/providers/google/cloud/gcs/example_gcs_to_gdrive.py