[Backport release-0.14] Account for two different kinds of consistency issues#289
Merged
negz merged 2 commits intorelease-0.14from Sep 7, 2021
Merged
[Backport release-0.14] Account for two different kinds of consistency issues#289negz merged 2 commits intorelease-0.14from
negz merged 2 commits intorelease-0.14from
Conversation
This commit is intended to address two issues that we diagnosed while investigating crossplane-contrib/provider-aws#802. The first issue is that controller-runtime does not guarantee reads from cache will return the freshest version of a resource. It's possible we could create an external resource in one reconcile, then shortly after trigger another in which it appears that the managed resource was never created because we didn't record its external-name. This only affects the subset of managed resources with non-deterministic external-names that are assigned during creation. The second issue is that some external APIs are eventually consistent. A newly created external resource may take some time before our ExternalClient's observe call can confirm it exists. AWS EC2 is an example of one such API. This commit attempts to address the first issue by making an Update to a managed resource immediately before Create it called. This Update call will be rejected by the API server if the managed resource we read from cache was not the latest version. It attempts to address the second issue by allowing managed resource controller authors to configure an optional grace period that begins when an external resource is successfully created. During this grace period we'll requeue and keep waiting if Observe determines that the external resource doesn't exist, rather than (re)creating it. Signed-off-by: Nic Cope <negz@rk0n.org> (cherry picked from commit a3a59c9)
The retry logic we use to persist critical annotations makes it difficult to delete an annotation without potentially also deleting annotations added by another controller (e.g. the composition logic). This commit therefore changes the way we detect whether we might have created an external resource but not recorded the result. Previously we relied on the presence of the 'pending' annotation to detect this state. Now we check whether the 'pending' annotation is newer than any 'succeeded' or 'failed' annotation. Signed-off-by: Nic Cope <negz@rk0n.org> (cherry picked from commit 8e780ec)
2 tasks
negz
approved these changes
Sep 7, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #283 to
release-0.14.