Skip to content

[Backport release-0.14] Account for two different kinds of consistency issues#289

Merged
negz merged 2 commits intorelease-0.14from
backport-283-to-release-0.14
Sep 7, 2021
Merged

[Backport release-0.14] Account for two different kinds of consistency issues#289
negz merged 2 commits intorelease-0.14from
backport-283-to-release-0.14

Conversation

@github-actions
Copy link
Copy Markdown

@github-actions github-actions bot commented Sep 7, 2021

Description

Backport of #283 to release-0.14.

negz added 2 commits September 7, 2021 22:30
This commit is intended to address two issues that we diagnosed while
investigating crossplane-contrib/provider-aws#802.

The first issue is that controller-runtime does not guarantee reads from cache
will return the freshest version of a resource. It's possible we could create an
external resource in one reconcile, then shortly after trigger another in which
it appears that the managed resource was never created because we didn't record
its external-name. This only affects the subset of managed resources with
non-deterministic external-names that are assigned during creation.

The second issue is that some external APIs are eventually consistent. A newly
created external resource may take some time before our ExternalClient's observe
call can confirm it exists. AWS EC2 is an example of one such API.

This commit attempts to address the first issue by making an Update to a managed
resource immediately before Create it called. This Update call will be rejected
by the API server if the managed resource we read from cache was not the latest
version.

It attempts to address the second issue by allowing managed resource controller
authors to configure an optional grace period that begins when an external
resource is successfully created. During this grace period we'll requeue and
keep waiting if Observe determines that the external resource doesn't exist,
rather than (re)creating it.

Signed-off-by: Nic Cope <negz@rk0n.org>
(cherry picked from commit a3a59c9)
The retry logic we use to persist critical annotations makes it difficult to
delete an annotation without potentially also deleting annotations added by
another controller (e.g. the composition logic). This commit therefore changes
the way we detect whether we might have created an external resource but not
recorded the result. Previously we relied on the presence of the 'pending'
annotation to detect this state. Now we check whether the 'pending' annotation
is newer than any 'succeeded' or 'failed' annotation.

Signed-off-by: Nic Cope <negz@rk0n.org>
(cherry picked from commit 8e780ec)
@negz negz merged commit a82d9e9 into release-0.14 Sep 7, 2021
@negz negz deleted the backport-283-to-release-0.14 branch September 7, 2021 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant