disable AWS GO SDK logging #2289
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
18 tasks
Contributor
|
@paulbehrisch I'm working on a PR to make it more configurable from helmfile & vals with PR - helmfile/vals#893 & #2290 |
aditmeno
added a commit
to aditmeno/helmfile
that referenced
this pull request
Nov 24, 2025
This PR fixes issue helmfile#2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR helmfile#2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR helmfile#2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR helmfile#893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: helmfile#2270 Related: helmfile#2288, helmfile#2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno
added a commit
to aditmeno/helmfile
that referenced
this pull request
Nov 24, 2025
This PR fixes issue helmfile#2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR helmfile#2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR helmfile#2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR helmfile#893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: helmfile#2270 Related: helmfile#2288, helmfile#2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno
added a commit
to aditmeno/helmfile
that referenced
this pull request
Nov 24, 2025
This PR fixes issue helmfile#2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR helmfile#2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR helmfile#2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR helmfile#893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: helmfile#2270 Related: helmfile#2288, helmfile#2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno
added a commit
to aditmeno/helmfile
that referenced
this pull request
Nov 24, 2025
This PR fixes issue helmfile#2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR helmfile#2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR helmfile#2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR helmfile#893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: helmfile#2270 Related: helmfile#2288, helmfile#2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
aditmeno
added a commit
to aditmeno/helmfile
that referenced
this pull request
Nov 24, 2025
This PR fixes issue helmfile#2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR helmfile#2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR helmfile#2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR helmfile#893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: helmfile#2270 Related: helmfile#2288, helmfile#2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
Author
@aditmeno I haven't read the entire vals repo, but i'd assume we only want to use DEBUG when we run helmfile with the debug flag? Otherwise the default log level should be INFO? |
yxxhero
pushed a commit
that referenced
this pull request
Nov 24, 2025
) * fix: make AWS SDK debug logging configurable (issue #2270) This PR fixes issue #2270 where AWS SDK debug logs expose sensitive credentials in helmfile output, by adding flexible, configurable AWS SDK logging with secure defaults. Problem: -------- Despite PR #2288's fix, AWS SDK debug logs still appeared in helmfile output, exposing sensitive information: - AWS tokens and authorization headers - Request/response bodies containing credentials - Secret metadata from vals providers Root Cause: ----------- 1. PR #2288 only suppressed vals' own logging via LogOutput: io.Discard 2. AWS SDK v2 uses separate logging (AWS_SDK_GO_LOG_LEVEL, WithClientLogMode) 3. Vals library defaulted to verbose logging (aws.LogRetries | aws.LogRequest) 4. No programmatic way to control AWS SDK logging Solution: --------- Two-part fix in conjunction with vals PR #893: 1. Vals library enhancement (helmfile/vals#893): - Added Options.AWSLogLevel field for programmatic control - Changed default from verbose to secure (no logging) - Added preset levels: off, minimal, standard, verbose - Maintains AWS_SDK_GO_LOG_LEVEL precedence 2. Helmfile changes (this PR): - Added HELMFILE_AWS_SDK_LOG_LEVEL environment variable - Enhanced vals configuration to use new AWSLogLevel field - Added conditional AWS SDK log suppression in remote.go (3 locations) - Comprehensive unit tests (15 test cases) Configuration: -------------- Preset levels via HELMFILE_AWS_SDK_LOG_LEVEL: - "off" (default) - No logging, secure, prevents credential leakage - "minimal" - Log retries only - "standard" - Log retries + requests (previous default behavior) - "verbose" - Log everything (requests, responses, bodies, signing) - Custom - Comma-separated values (e.g., "request,response") Priority order: 1. AWS_SDK_GO_LOG_LEVEL env var (highest) 2. HELMFILE_AWS_SDK_LOG_LEVEL env var 3. Secure default ("off") Testing: -------- Added comprehensive unit tests: - pkg/plugins/vals_test.go: 9 test cases * TestAWSSDKLogLevelConfiguration - all preset levels * TestEnvironmentVariableReading - env var parsing - pkg/remote/remote_test.go: 6 test cases * TestAWSSDKLogLevelInit - init() logic All tests passing: - pkg/plugins: PASS (3/3 test suites) - pkg/remote: PASS (all test suites) - golangci-lint: 0 issues Files changed: 7 files, 271 insertions(+), 31 deletions(-) Security: --------- Before: Credentials exposed by default (aws.LogRetries | aws.LogRequest) After: Credentials protected by default (no logging unless explicitly enabled) Follows security principles: - Secure by default - Principle of least privilege - Explicit opt-in for sensitive logging - Defense in depth Dependency: ----------- Depends on: helmfile/vals#893 Currently using: aditmeno/vals@a97336ce2bf6 (via go.mod replace) After vals PR merges: Update to official release Fixes: #2270 Related: #2288, #2289, helmfile/vals#893 Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> * chore: update vals to use parameter-based AWS log level configuration Updated vals dependency to commit 06d7cd29 which implements clean parameter-based AWS SDK logging configuration instead of using global state mutation. Changes in vals implementation: - AWS log level passed through function parameters to each provider - No os.Setenv() - no environment mutation - No package-level global variables - No sync/atomic dependency needed - Thread-safe by design - each provider instance has its own log level This maintains the same functionality as before but with a cleaner implementation that avoids global state mutation. Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> * deps: update vals to upstream v0.42.6 Update from vals fork (aditmeno/vals) to official release v0.42.6. Remove replace directive now that vals PR #893 has been merged upstream. This brings in the AWS SDK log level configuration improvements: - SetDefaultLogLevel() package-level function - Options.AWSLogLevel field support - Secure default (no logging) - Preset log levels (off, minimal, standard, verbose) Also updates related dependencies: - Azure SDK and auth libraries - AWS SDK config and credentials - OAuth2 library Signed-off-by: Aditya Menon <amenon@canarytechnologies.com> --------- Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
Contributor
I've set the default to not log at all, but you can set AWS_SDK_GO_LOG_LEVEL to the level you require - helmfile/vals#893 |
Member
|
fixed |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#2270 is still occurring for me with helmfile 1.2.1
❯ helmfile version
▓▓▓ helmfile
Version v1.2.1
Git Commit "brew"
Build Date 23 Nov 25 16:32 +07 (15 hours ago)
Commit Date 23 Nov 25 16:32 +07 (15 hours ago)
Dirty Build no
Go version 1.25.4
Compiler gc
Platform darwin/arm64
❯
❯ helmfile diff -l cluster=staging
SDK 2025/11/24 07:54:06 DEBUG Request
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
As I said this started happening with helmfile 1.1.6 and this PR would be a workaround.
https://github.com/helmfile/vals/releases/tag/v0.42.1
Vals 0.42.1 migrated to the new Go SDK version. I believe by default the AWS Go SDK logs as INFO.