About my comment, I want to propose a few solutions. Firstly, let me summarize my point again. When we set a Selector, we are selecting reference resources and putting them to the Reference field. In our Resolve functions, the Reference field is checking firstly, and after resolving references by using this Reference field we return early. This means that, Reference field is used as cache of Selector. So this blocks 'resolving always scenario' for Selector field (editing Selector and expecting some changes). Some proposed solutions:

• Changing the order of Reference and Selector fields. By this way, the Selector field will be dominant and for every loop, if there is a change in Selector field, Reference field will be updated.
• Now, we are writing the selected reference from Selector to Reference field. The another option can be that, differentiate the two APIS (Reference and Selector). I mean that, selected reference resource name from Selector cannot be written to the Reference field. By this way, we can protect the current logic in Resolve function.
• We may consider that, do not return early after resolving the Reference field. We can let that resolving process can continue for the Selector field also. With this approach, both of the two fields will be tried to resolve for every loop. As a behavioral change of this approach, we will support the using two APIs together.

To be honest, I don't have a strong feeling about which of these would be the better approach. What are your thoughts @negz about this point? Do you have any other approach for addressing this? cc @muvaf

Changing the order of Reference and Selector fields. By this way, the Selector field will be dominant and for every loop, if there is a change in Selector field, Reference field will be updated.

I think this option is more intuitive from the perspective of users since the main job of selector is to select a reference and it makes it straight-forward to understand. If we add policy field to the selector as well, similar to reference, I think users could configure it while being aware of the risk that if a new resource with the same labels is added, then the list will change. I'd personally not opt for that risk but it's at least clear. In practice, I think it'll likely to be used temporarily during operations when a resource is stuck or wrong references are resolved and once it's done, the user would revert it back.

I think this option is more intuitive from the perspective of users since the main job of selector is to select a reference and it makes it straight-forward to understand. If we add policy field to the selector as well, similar to reference, I think users could configure it while being aware of the risk that if a new resource with the same labels is added, then the list will change. I'd personally not opt for that risk but it's at least clear. In practice, I think it'll likely to be used temporarily during operations when a resource is stuck or wrong references are resolved and once it's done, the user would revert it back.

@muvaf I added the new API for Selector and also changed the order of Reference and Selector fields. Now, we are resolving the Selector field firstly.

About this PR, I want ot share some points. We are trying to support new capabilities for the reference resoultion by providing policies. First one is the resolving references for every reconciliation loop (Always policy), the another one is that resolving references as Optional. We have some challenges about them, especially Always one.

I think, the existing API is very appropriate for the current behaviors:

• Resolving references only once
• Cannot be ignored the errors that occurred while reference. (it blocks the circular dependency)

Although this API works very successful for many cases, it seems that, for our new cases that we want support it has some weak parts. All the concerns that I will talk about a little later have been expressed from different angles in the issues. My purpose here is to write about these concerns in this PR and explore whether there is a better way.

I want to share two points (of course, we can talk about many difference cases). These two cases are about the new Always policy.

• Some resources support multiple references like DBSubnetGroup (AWS). DBSubnetGroup can refer more than one Subnet resources. With this PR, we are providing a new API field for every Reference: Policy. So for every Subnet reference in the DBSubnetGroup we can determine different policies. Let me share an example:

SubnetIDRefs:
- name: subnet-1
    policy:
      resolve: Always
- name: subnet-2
- name: subnet-3
- name: subnet-4
    policy:
      resolve: Always

As you can see, some inconsistent situations may occur when the new API is used. For now, when we see an Always policy for an element of list, we are resolving all references for every reconcile loop. In other words, it is sufficient for only one of the elements in the list have an Always policy for resolution in each loop. This might seem intuitively reasonable. However, it is theoretically possible to implement the opposite of this behavior. So it's actually a preference, and it might be reasonable to implement a behavior otherwise.

• When we use the Selector field with the Always policy, a scenario can be possible with the new behavior.

vpcIdSelector:
  matchLabels:
    key: value
  policy:
    resolve: Always

After first reconciliation loop:

vpcIdReference:
- name: vpc-example
vpcIdSelector:
  matchLabels:
    key: value
  policy:
    resolve: Always

Then user edited the Reference field value by using kubectl:

vpcIdReference:
- name: MANUALLY-CHANGED-vpc-example
vpcIdSelector:
  matchLabels:
    key: value
  policy:
    resolve: Always

In next reconciliation loop, because of the Always policy, selector will resolve the reference again and override the changed desired state of this resource:

vpcIdReference:
- name: vpc-example
vpcIdSelector:
  matchLabels:
    key: value
  policy:
    resolve: Always

For the first state of the resource, the Reference field was empty so filling this via controller is the late-initialization concept. However, after changed this value manually, this means that the user gives a new desired state. So I am not sure that we must override this.

So, for now, the new Policies are make sense and resolve the issues. Also I think this is an improvement for the reference resolution pipeline. However, we may discuss re-designing this API for providing better concept for our new cases.

💯

As you can see, some inconsistent situations may occur when the new API is used. For now, when we see an Always policy for an element of list, we are resolving all references for every reconcile loop. In other words, it is sufficient for only one of the elements in the list have an Always policy for resolution in each loop.

@sergenyalcin Could you open an issue regarding this? I don't love that each element has its own policy but there isn't really a good way of adding a policy to the whole array without a breaking change. Maybe an annotation to point to the array field and say policy of this field is Always?

For the first state of the resource, the Reference field was empty so filling this via controller is the late-initialization concept. However, after changed this value manually, this means that the user gives a new desired state. So I am not sure that we must override this.

I think this is pretty much the core use case users want with Always policy. In an ideal case, because the reference is changed by a controller, it would appear under status but then that reference field itself is actually a parameter. So, it's really hard to get both to work. Though personally, I feel better that it's configurable and default is consistent and IMO both policies will be used seldom.