Why not marshal into a JSON blob and send a byte stream via grpc?

Good question! I can see the general idea making sense to keep the format consistent all the way through instead of transforming back and forth. I'll need to learn a bit more details about gRPC connections and their transfer modes though, i.e. what may we lose or have to do differently by using that transfer mode instead.

I remember there was a separate discussion with @luxas on whether we can use the change logs feature to record data about all the reconciliations (for example, reconciliation timestamps), and not just about those reconciliations which attempt to correct drifts. Then the after observation would become optional (only recorded for those reconciliations that update the external resource). crossplane-runtime may not need to be opinionated on this, we would extend crossplane-runtime, and if needed, the package manager and the like, so that the side car (the gRPC server) and the client are properly setup, and the provider could use this infrastructure to implement some related features. The proposed arbitrary data field opens up the possibility for similar cases but the crossplane-runtime had better be un-opinionated about them, if we would like to allow other use cases.

Re-observing for the after-update external resource state would be the issue in this case, if we take the control from the managed reconciler. I think we would not like to have the external client to make a second observation on its own. One idea could be to allow the external client to be able to signal to the managed reconciler the need for an immediate reconciliation so that the managed reconciler requeues an immediate rate limited reconciliation request without waiting for the next poll period.

How do you folks feel about allowing the providers to record more reconciliation data, not limited to updates?

How do you folks feel about allowing the providers to record more reconciliation data, not limited to updates?

I don't feel like I understand well enough to answer the question. What are some examples of reconciliation data other than timestamps?

Good question! I can see the general idea making sense to keep the format consistent all the way through instead of transforming back and forth. I'll need to learn a bit more details about gRPC connections and their transfer modes though, i.e. what may we lose or have to do differently by using that transfer mode instead.

Additional Observe means one more extra call to the cloud provider API and will increase probability of API ratelimiting. At least for Creates, we always queue in the managed reconcilers and have a consecutive Observe, which we can leverage somehow.

I think the additional Observe() call is a real concern, we already get complaints that Crossplane is making too many API calls. I think it would be logical to requeue after a CUD operation and use that Observe() call as that "after" state for the previous operation. Maybe use an "operation in-progress" flag to know that there has been a CUD request that needs verification.

I definitely hear the concerns about the extra Observe() call. The usefulness of it is even being called into question with #5822 (comment) 😊

I'm not against cutting the "after" state completely, at least from this initial alpha implementation. I think we still tell an interesting story about WHY we made a change without having that data.

Not implementing an "after" state may be preferable instead of having to manage state across multiple reconciles to link together the Observe() results from a subsequent reconcile with the Create() or Update() operation from a previous reconcile. As things are currently implemented (crossplane/crossplane-runtime@master...jbw976:crossplane-runtime:change-logs), everything is contained within a single reconcile and is definitely more simple for that 😇

So in this alpha implementation, we could:

1. perform the extra Observe() call and see the feedback from early adopters
2. cut the "after" state functionality and do nothing
3. code something more complex that connects Observe() state to Create()/Update()/Delete() observations across reconciliations

Strong opinions from you all? #3 is my least preferred I think 😂

I think alpha is a great time to see what the impact of the second Observe() call would be. If it's bad we can always cut it when it goes beta.

You could always put it under it's own nested flag so the feature could run with or without the second Observe() call.

Chatted with @negz a bit more about "after" state and a 2nd Observe() call:

• From the first Observe() call, we should have a very up to date status.atProvider for any modern managed resource
• The difference between spec.forProvider and status.atProvider should fairly conclusively indicate why we are performing an operation
• The content of an "after" state collection from a 2nd Observe() is likely to not be any different from the "before", and therefore have limited value as called out in design: cloud change logs #5822 (comment)

Therefore, it's reasonable to remove from this design the "after" state and thus any reason to perform a 2nd Observe(). Without it, we still have a useful record of why we are making changes, whether that be because the spec's desired state has changed, we detected drift in the external resource, etc.

Perhaps Providers could be extended further in the future to provide more details in their ExternalObservation about why they concluded ResourceUpToDate: false and we can include that information in the change logs. For example, there is a ExternalObservation.Diff field, but I don't believe many Providers are currently setting it.

I think the additional Observe() call is a real concern, we already get complaints that Crossplane is making too many API calls. I think it would be logical to requeue after a CUD operation and use that Observe() call as that "after" state for the previous operation. Maybe use an "operation in-progress" flag to know that there has been a CUD request that needs verification.

Any chance we can include resource labels?

Assume we aggregate these changelog events into a search index. We'd then want to correlate them with higher-level events (ArgoCD or FluxCD resource events) and finally point to a commit (or set of commits).

Having the resource labels as KVs in this schema would allow us to build the relationship with those higher-level events more directly, providing a solid end-to-end view into how changes at the higher-end of the abstraction flow to changes at the lowest end.