Skip to content

Batch get secrets#3483

Merged
moskyb merged 8 commits into
mainfrom
batch-secrets
Sep 18, 2025
Merged

Batch get secrets#3483
moskyb merged 8 commits into
mainfrom
batch-secrets

Conversation

@moskyb

@moskyb moskyb commented Sep 16, 2025

Copy link
Copy Markdown
Contributor

Description

This PR updates the buildkite-agent secret get command to be able to concurrently fetch multiple secrets — eg, where previously you might've had to do something like:

for key in "DATABASE_URL" "API_TOKEN" "SOME_OTHER_SECRET"; do
  val=$(buildkite-agent secret get key)
  declare "$key=$val"
done

and wait for each secret fetch to happen serially. Now, the secret get command can fetch the secrets concurrently, and output them in an easy-to-parse format like JSON or env:

buildkite-agent secret get --skip-redaction "DATABASE_URL" "API_TOKEN" "SOME_OTHER_SECRET"
{"DATABASE_URL": "database.postgres:5432", "API_TOKEN": "hunter2", "SOME_OTHER_SECRET": "ssh!"}
# or
buildkite-agent secret get --skip-redaction --format env "DATABASE_URL" "API_TOKEN" "SOME_OTHER_SECRET"
DATABASE_URL="database.postgres:5432"
API_TOKEN="hunter2"
SOME_OTHER_SECRET="ssh!"

Note that under normal circumstances (where the --skip-redaction flag is omitted) the values of the secrets would be redacted from buildkite job logs.

Additionally, secrets embdedded in the buildkite pipeline YAML will also be fetched concurrently.

Context

PS-1064
Slack convo

Changes

Usage:

    buildkite-agent secret get [options...] [key1] [key2] ...

Description:

Gets a list of secrets from Buildkite and prints them to stdout. Key names are case
insensitive in this command, and secret values are automatically redacted in the build logs
unless the ′skip-redaction′ flag is used.

If any request for a secret fails, the command will return a non-zero exit code and print details of all failed secrets.

If only a single key is provided, the secret value will be printed without any formatting.

If multiple keys are provided, the output format will be as defined by the ′format′ flag, which defaults to JSON.

Examples:

    # Secret keys are case insensitive
    $ buildkite-agent secret get deploy_key
    "..."
    $ buildkite-agent secret get DEPLOY_KEY
    "..."

    # Format is ignored when only a single key is provided
    $ buildkite-agent secret get --format env deploy_key
    "..."

    # JSON is the default format when multiple keys are provided
    $ buildkite-agent secret get deploy_key github_api_token
    {"deploy_key": "...", "github_api_token": "..."}

    # ... but you can also request env format, which can be piped into ′source′, ′declare -x′, and friends
    $ buildkite-agent secret get --format env deploy_key github_api_token
    DEPLOY_KEY="..."
    GITHUB_API_TOKEN="..."

Options:

  --no-color                  Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
  --debug                     Enable debug mode. Synonym for ′--log-level debug′. Takes precedence over ′--log-level′ [$BUILDKITE_AGENT_DEBUG]
  --log-level value           Set the log level for the agent, making logging more or less verbose. Defaults to notice. Allowed values are: debug, info, error, warn, fatal (default: "notice") [$BUILDKITE_AGENT_LOG_LEVEL]
  --experiment value          Enable experimental features within the buildkite-agent [$BUILDKITE_AGENT_EXPERIMENT]
  --profile value             Enable a profiling mode, either cpu, memory, mutex or block [$BUILDKITE_AGENT_PROFILE]
  --agent-access-token value  The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
  --endpoint value            The Agent API endpoint (default: "https://agent.buildkite.com/v3") [$BUILDKITE_AGENT_ENDPOINT]
  --no-http2                  Disable HTTP2 when communicating with the Agent API. [$BUILDKITE_NO_HTTP2]
  --debug-http                Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]
  --trace-http                Enable HTTP trace mode, which logs timings for each HTTP request. Timings are logged at the debug level unless a request fails at the network level in which case they are logged at the error level [$BUILDKITE_AGENT_TRACE_HTTP]
  --job value                 Which job should should the secret be for [$BUILDKITE_JOB_ID]
  --format value              The output format, either 'default', 'json', or 'env'. When 'default', a single secret will print just the value, while multiple secrets will print JSON. When 'json' or 'env', secrets will be printed as key-value pairs in the requested format (default: "default") [$BUILDKITE_AGENT_SECRET_GET_FORMAT]
  --skip-redaction            Skip redacting the retrieved secret from the logs. Then, the command will print the secret to the Job's logs if called directly. [$BUILDKITE_AGENT_SECRET_GET_SKIP_SECRET_REDACTION]

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go fmt ./...)

Disclosures / Credits

It's all artisanal, handcrafted slop from me.

This command concurrently requests a list of secrets from the agent API, redacts them, then prints them to the terminal in a configurable format. This should significantly speed up operations where lots of secrets are being fetched.
@DrJosh9000 DrJosh9000 self-requested a review September 16, 2025 23:23
Comment thread internal/secrets/secret.go

@DrJosh9000 DrJosh9000 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good, I have some tweaks

Comment thread internal/secrets/secret_test.go Outdated

keys := []string{"DATABASE_URL", "API_TOKEN"}
secrets, errs := FetchSecrets(context.Background(), apiClient, "test-job-id", keys, false)
secrets, errs := FetchSecrets(context.Background(), apiClient, "test-job-id", keys, 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you're here, why not update all these:

Suggested change
secrets, errs := FetchSecrets(context.Background(), apiClient, "test-job-id", keys, 1)
secrets, errs := FetchSecrets(t.Context(), apiClient, "test-job-id", keys, 1)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have another branch locally to clean up the tests in this file, will submit along with that

Comment thread clicommand/secret_batch_read.go Outdated
Comment thread clicommand/secret_batch_read.go Outdated
Comment thread clicommand/secret_batch_read.go Outdated
Comment thread internal/secrets/secret.go Outdated
@moskyb moskyb mentioned this pull request Sep 17, 2025
2 tasks

@DrJosh9000 DrJosh9000 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread clicommand/secret_get.go Outdated
Comment thread clicommand/secret_get.go Outdated
Comment thread clicommand/secret_get.go Outdated
Comment thread clicommand/secret_get.go Outdated
@buildkite buildkite deleted a comment from blaknite Sep 18, 2025
@dabarrell dabarrell dismissed their stale review September 18, 2025 03:34

Addressed

Co-authored-by: David Barrell <david@buildkite.com>

@DrJosh9000 DrJosh9000 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM. Great work!

@moskyb moskyb enabled auto-merge September 18, 2025 03:55
@moskyb moskyb merged commit da1cbfa into main Sep 18, 2025
1 check passed
@moskyb moskyb deleted the batch-secrets branch September 18, 2025 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants