test(policy): address timeouts and flakiness#14773
Merged
Conversation
Policy tests are very flaky. Currently one of the main culprits is that service account creation sometimes isn't caught as an event by the watcher, blocking `await_service_account` until it times out after 60s. We already have in place up to 3 retries when calling `cargo nextest`, but these tests are sequential and the 60s timeouts start accumulating until we reach the CI job timeout at 20min. This change lowers the service account creation timeout down to 15s, understanding that if the watcher catches that event it will do pretty quickly or else block indefinitely. So better to fail faster and trigger the test retry ASAP.
adleong
reviewed
Dec 2, 2025
Member
adleong
left a comment
There was a problem hiding this comment.
I wonder if the reduction in flakiness we get here is just because checking for the service account with a synchronous call takes time, allowing for more time for the namespace to be persisted. I.e. I wonder if the service account get is roughly equivalent to a sleep here.
I also wonder if awaiting for the namespace to show up in a watch could get rid of the flakiness entirely by guaranteeing that kubernetes is ready for us to initiate the namespaced service account watch.
adleong
approved these changes
Dec 3, 2025
Signed-off-by: Scott Fleener <scott@buoyant.io>
sfleen
approved these changes
Dec 4, 2025
Member
Author
|
Ok I finally added the namespace watcher and refactored things a bit to avoid repetition, and CI is doing great. I also had to incorporate #14777, that just surfaced and was blocking CI. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Policy tests are very flaky. Currently one of the main culprits is that service account creation sometimes isn't caught as an event by the watcher, blocking
await_service_accountuntil it times out after 60s. We already have in place up to 3 retries when callingcargo nextest, but these tests are sequential and the 60s timeouts start accumulating until we reach the CI job timeout at 20min.This change first lowers the service account creation timeout down to 15s, understanding that if the watcher catches that event it will do pretty quickly or else block indefinitely. So better to fail faster and trigger the test retry ASAP.
With this change,
test-policy (v1.34, linkerd, experimental)is finally passing, taking 17m due to the large number of retries:After having measured this, we also added a check in
await_service_accountto bypass the watcher logic if the SA is already in place. This resulted in the same tests taking only 12m with far less flakiness: