Skip to content

fix(secret-manager): rewrite secret manager to async/await with await of initialize()#6380

Closed
eagleeye wants to merge 1 commit intogoogleapis:mainfrom
eagleeye:secret-manager-rewrite-to-await
Closed

fix(secret-manager): rewrite secret manager to async/await with await of initialize()#6380
eagleeye wants to merge 1 commit intogoogleapis:mainfrom
eagleeye:secret-manager-rewrite-to-await

Conversation

@eagleeye
Copy link
Contributor

@eagleeye eagleeye commented May 26, 2025

Hi there! I've find a lot if issues with async code, that are causing uncaught exceptions in our production systems, so I've fixed them.
The last change to the secret manager did not improve stability of the code, here I've explained why:
#6188 (review)
And showed it with a small test
image

List of changes:

  • mark most of the functions as async - this helps to track Promises in the code, allows to use await in that functions
  • changed calls this.initialize() to await this.initialized
  • replaced some Promise chains to await, that allowed to flatten the code
  • fixed tslint errors related to commas and formatting

Original error, that happens super rarely, near in 1/1000 times, but it causes our containers restart during start, because it happens out of secret-manager call stack, so we can't retry it.
The error was spotted in secret manager v5.6.0, that is not the latest and I'm aware, that some fixes were added in 6.0.1 with google-gax update, but anyway, current master of secret manager has issues in handling promises, that must be fixed.

2025-05-19 1251.300 EEST
Error: 2 UNKNOWN: Getting metadata from plugin failed with error: Could not refresh access token: A Forbidden error was returned while attempting to retrieve an access token for the Compute Engine built-in service account. This may be because the Compute Engine instance does not have the correct permission scopes specified: Could not refresh access token: Unsuccessful response status code. Request failed with status code 403 at callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js19) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js76) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js141) at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js181) at /usr/src/app/node_modules/@grpc/grpc-js/build/src/resolving-call.js78 at process.processTicksAndRejections (node77:11)
2025-05-19 1251.300 EEST
for call at
2025-05-19 1251.300 EEST
at ServiceClientImpl.makeUnaryRequest (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js32)
2025-05-19 1251.300 EEST
at ServiceClientImpl.anonymous (/usr/src/app/node_modules/@grpc/grpc-js/build/src/make-client.js19)
2025-05-19 1251.300 EEST
at /usr/src/app/node_modules/@google-cloud/secret-manager/build/src/v1/secret_manager_service_client.js29
2025-05-19 1251.300 EEST
at /usr/src/app/node_modules/google-gax/build/src/normalCalls/timeout.js16
2025-05-19 1251.300 EEST
at repeat (/usr/src/app/node_modules/google-gax/build/src/normalCalls/retries.js25)
2025-05-19 1251.300 EEST
at /usr/src/app/node_modules/google-gax/build/src/normalCalls/retries.js13
2025-05-19 1251.300 EEST
at OngoingCallPromise.call (/usr/src/app/node_modules/google-gax/build/src/call.js27)
2025-05-19 1251.300 EEST
at NormalApiCaller.call (/usr/src/app/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js19)
2025-05-19 1251.300 EEST
at /usr/src/app/node_modules/google-gax/build/src/createApiCall.js30
2025-05-19 1251.300 EEST
at process.processTicksAndRejections (node95:5) {
2025-05-19 1251.300 EEST
code: 2,
2025-05-19 1251.300 EEST
  details: 'Getting metadata from plugin failed with error: Could not refresh access token: A Forbidden error was returned while attempting to retrieve an access token for the Compute Engine built-in service account. This may be because the Compute Engine instance does not have the correct permission scopes specified: Could not refresh access token: Unsuccessful response status code. Request failed with status code 403',

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
    The issues with async code are obvious and I believe that they do not require discussions
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)
    The code didn't changed the signature of the methods. The methods were async, the tests didn't changed.

Fixes #<issue_number_goes_here> 🦕
(No issue created)

@eagleeye eagleeye requested review from a team and yoshi-approver as code owners May 26, 2025 14:42
@eagleeye eagleeye force-pushed the secret-manager-rewrite-to-await branch from 9563064 to 09828b6 Compare May 26, 2025 14:54
@eagleeye eagleeye changed the title fix: rewrite secret manager to async/await with await of initialize() fix(secret-manager): rewrite secret manager to async/await with await of initialize() May 26, 2025
@product-auto-label product-auto-label bot added the api: secretmanager Issues related to the Secret Manager API. label May 27, 2025
@eagleeye eagleeye force-pushed the secret-manager-rewrite-to-await branch 6 times, most recently from 617444f to da51579 Compare June 3, 2025 10:47
@eagleeye eagleeye force-pushed the secret-manager-rewrite-to-await branch from da51579 to ca4bb7c Compare June 3, 2025 10:47
@eagleeye
Copy link
Contributor Author

eagleeye commented Jun 3, 2025

@sofisl I've fixed some more issues in the code, I've missed few awaits, please trigger test again.

@sofisl
Copy link
Contributor

sofisl commented Jun 3, 2025

/gcbrun

@sofisl sofisl added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jun 4, 2025
@sofisl
Copy link
Contributor

sofisl commented Jun 4, 2025

@eagleeye Thanks for your contributions. This library isn't where the changes go but I did want to check if tests would pass. I need to confirm with my team what the best move is here but I'll reopen this as an issue in the generator for tracking: googleapis/google-cloud-node-core#543

@sofisl sofisl closed this Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: secretmanager Issues related to the Secret Manager API. do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants