bk: use OIDC to tear-down the cloud resources#2567
Conversation
This will run in parallel while we do the migration Avoid using service accounts and use the Keyless/OIDC approach Uses https://github.com/elastic/oblt-aws-auth-buildkite-plugin
this should help with the lack of support for OIDC in cloud-reaper
| # NOTE: this approach is deprecated and will be removed in the near future. | ||
| # see https://github.com/elastic/observability-robots/issues/2771 (only accessible by Elastic employees) | ||
| if [[ "$BUILDKITE_PIPELINE_SLUG" == "elastic-package-cloud-cleanup" && "$BUILDKITE_STEP_KEY" == "cloud-cleanup" ]]; then | ||
| AWS_SERVICE_ACCOUNT_SECRET_PATH=kv/ci-shared/platform-ingest/aws_ingest_ci |
There was a problem hiding this comment.
let's define the variable where it's used
| @@ -0,0 +1,229 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
I copied it from .buildkite/scripts/cloud-cleanup.sh and added the support for OIDC.
| options: | ||
| key: '${ACCOUNT_KEY}' | ||
| secret: '${ACCOUNT_SECRET}' | ||
| key: '${AWS_ACCESS_KEY_ID}' |
There was a problem hiding this comment.
To help with using the existing env variables created by https://github.com/elastic/oblt-aws-auth-buildkite-plugin
| # As long as cloud reaper does not support OIDC authentication. | ||
| create_aws_ephemeral_user() { | ||
| # Generate a unique name for the ephemeral IAM user. | ||
| EPHEMERAL_USER="ephemeral-admin-$(date +%s)" | ||
| echo "Creating IAM user: ${EPHEMERAL_USER}" | ||
| aws iam create-user --user-name "${EPHEMERAL_USER}" \ | ||
| --tags Key=ephemeral,Value=true Key=division,Value=engineering Key=org,Value=obs Key=environment,Value=ci Key=repo,Value=elastic-package Key=created_at,Value="$(date -Is)" | ||
|
|
||
| echo "Attaching AdministratorAccess policy to ${EPHEMERAL_USER}..." | ||
| aws iam attach-user-policy --user-name "${EPHEMERAL_USER}" --policy-arn arn:aws:iam::aws:policy/AdministratorAccess | ||
|
|
||
| echo "Creating access keys for ${EPHEMERAL_USER}..." | ||
| creds_json=$(aws iam create-access-key --user-name "${EPHEMERAL_USER}") | ||
| AWS_ACCESS_KEY_ID_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId') | ||
| AWS_SECRET_ACCESS_KEY_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.SecretAccessKey') | ||
| export EPHEMERAL_USER AWS_ACCESS_KEY_ID_EPHEMERAL AWS_SECRET_ACCESS_KEY_EPHEMERAL | ||
| } | ||
|
|
||
| # Define cleanup function to delete the ephemeral IAM user regardless of script outcome. | ||
| cleanup_ephemeral_user() { | ||
| echo "Cleaning up ephemeral IAM user: ${EPHEMERAL_USER}" | ||
| aws iam detach-user-policy --user-name "${EPHEMERAL_USER}" --policy-arn arn:aws:iam::aws:policy/AdministratorAccess | ||
| key_id=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId') | ||
| aws iam delete-access-key --user-name "${EPHEMERAL_USER}" --access-key-id "${key_id}" | ||
| aws iam delete-user --user-name "${EPHEMERAL_USER}" | ||
| echo "Ephemeral IAM user ${EPHEMERAL_USER} deleted." | ||
| } | ||
| trap cleanup_ephemeral_user EXIT | ||
|
|
||
| cloud_reaper_aws() { | ||
| echo "--- Configuring ephemeral user" | ||
| create_aws_ephemeral_user |
There was a problem hiding this comment.
This is what I added to help with the OIDC when running cloud-reaper
| echo "--- Installing awscli" | ||
| with_aws_cli | ||
|
|
There was a problem hiding this comment.
Run earlier, to help with using the aws cli to create the ephemeral service account
| AWS_ACCESS_KEY_ID_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId') | ||
| AWS_SECRET_ACCESS_KEY_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.SecretAccessKey') | ||
| export EPHEMERAL_USER AWS_ACCESS_KEY_ID_EPHEMERAL AWS_SECRET_ACCESS_KEY_EPHEMERAL |
There was a problem hiding this comment.
Should AWS_ACCESS_KEY_ID_EPHEMERAL and AWS_SECRET_ACCESS_KEY_EPHEMERAL be renamed to follow the patterns so their contents are redacted ?
https://buildkite.com/docs/pipelines/configure/managing-log-output#redacted-environment-variables
Maybe something like AWS_ACCESS_KEY_ID_EPHEMERAL_SECRET and AWS_SECRET_ACCESS_KEY_EPHEMERAL_SECRET ? Or is it not needed in this scenario ?
There was a problem hiding this comment.
It does not matter at all, those env variables are not masked, unless they are set in the pre-command.
When creating env variable on the fly, there is no way to redact values
|
I'm running this build to validate if the recent changes with |
💚 Build Succeeded
History
cc @v1v |
What
Why
Avoid using service accounts and use the Keyless/OIDC approach.
Test
See https://buildkite.com/elastic/elastic-package-cloud-cleanup/builds/459
Further details
Uses https://github.com/elastic/oblt-aws-auth-buildkite-plugin
Requires #2568, so i can test this PR in isolation before merging it.