/test

Ah, right, I remember what the problem is here: the cached DNS rules include the IP of the dns server, a.k.a the peer pod, which can change at any point in time, independent of the lifecycle of the subject endpoint.

So, DNS rules can be out-of-date even when no endpoint-specific edges have been missed. This is a bummer.

The solution, IMO, is to replace the Trigger with a Controller that runs periodically. That way we can be reasonably sure that the set of rules in question is up-to-date.

Ah, right, I remember what the problem is here: the cached DNS rules include the IP of the dns server, a.k.a the peer pod, which can change at any point in time, independent of the lifecycle of the subject endpoint.

So, DNS rules can be out-of-date even when no endpoint-specific edges have been missed. This is a bummer.

The solution, IMO, is to replace the Trigger with a Controller that runs periodically. That way we can be reasonably sure that the set of rules in question is up-to-date.

So do you also recommend calling the controller when there are new rules being added Or solely rely on the periodic update ? I guess there can be a scenario where rules are applied but state in fs is not updated as controller is yet to run. I think that's what you meant by reasonably sure part.
How frequent can/should the controller run ? Maybe have that configurable too.

Ah, right, I remember what the problem is here: the cached DNS rules include the IP of the dns server, a.k.a the peer pod, which can change at any point in time, independent of the lifecycle of the subject endpoint.

So, DNS rules can be out-of-date even when no endpoint-specific edges have been missed. This is a bummer.

The solution, IMO, is to replace the Trigger with a Controller that runs periodically. That way we can be reasonably sure that the set of rules in question is up-to-date.

Writing the DNS rules to disk is ultimately best effort because it's done synchronously and if the pod is deleted in the meantime before the latest rules can be synced to disk, then

Ah, right, I remember what the problem is here: the cached DNS rules include the IP of the dns server, a.k.a the peer pod, which can change at any point in time, independent of the lifecycle of the subject endpoint.
So, DNS rules can be out-of-date even when no endpoint-specific edges have been missed. This is a bummer.
The solution, IMO, is to replace the Trigger with a Controller that runs periodically. That way we can be reasonably sure that the set of rules in question is up-to-date.

So do you also recommend calling the controller when there are new rules being added Or solely rely on the periodic update ? I guess there can be a scenario where rules are applied but state in fs is not updated as controller is yet to run. I think that's what you meant by reasonably sure part. How frequent can/should the controller run ? Maybe have that configurable too.

I'm still not sure I understand the problem being solved here.

Basically, SyncEndpointHeaderFile() is called each time the endpoint makes a DNS request. SyncEndpointHeaderFile() triggers code with a 5s interval meaning the sync to the fs only occurs once every 5s.

Therefore, I don't yet see the arguement for why we should sync DNS rules to the fs on policy calculation (endpoint regeneration) when the code already syncs the DNS rules when there's a DNS request made, once every 5s.

@christarazi @vipul-21

Therefore, I don't yet see the arguement for why we should sync DNS rules to the fs on policy calculation (endpoint regeneration) when the code already syncs the DNS rules when there's a DNS request made, once every 5s.

There are two things cached in the endpoint directory for fast recovery on startup:

1. The list of domain names + IPs seen by this endpoint
2. The list of ports + dns server pod IPs that are allowed

No. 1 cannot change without a DNS request, so it makes sense that this is updated per-request. However, no. 2 has a completely unrelated lifecycle. In fact, the lifecycle may be completely independent of the node itself, as CoreDNS being rescheduled on a remote node requires this to be updated.

So, that's why we should periodically refresh the cached DNS information (a.k.a "write the header file"). Either we build a mechanism to somehow propagate dns rules selector updates (hard!), or we just periodically update endpoint cached DNS info.

Make sense?

@christarazi @vipul-21

Therefore, I don't yet see the arguement for why we should sync DNS rules to the fs on policy calculation (endpoint regeneration) when the code already syncs the DNS rules when there's a DNS request made, once every 5s.

There are two things cached in the endpoint directory for fast recovery on startup:

1. The list of domain names + IPs seen by this endpoint

2. The list of ports + dns server pod IPs that are allowed

No. 1 cannot change without a DNS request, so it makes sense that this is updated per-request. However, no. 2 has a completely unrelated lifecycle. In fact, the lifecycle may be completely independent of the node itself, as CoreDNS being rescheduled on a remote node requires this to be updated.

So, that's why we should periodically refresh the cached DNS information (a.k.a "write the header file"). Either we build a mechanism to somehow propagate dns rules selector updates (hard!), or we just periodically update endpoint cached DNS info.

Make sense?

That clarifies things a bit, thanks. What is the consequence if we don't update (2) on a periodic basis? I understand the rules are "out of date / out of sync" on the fs, but what is the actual consequence of them being out of date? Does the list of ports + DNS server pods IPs (coredns) never get updated in the endpoint caches?

@christarazi

What is the consequence if we don't update (2) on a periodic basis?

As of now there shouldn't be any consequence, but this was discussed in sig-policy meeting in June: https://docs.google.com/document/d/1p6LuzoKR55_HgQJTWFInwj2FlIeo7_uAFXYuXe_Lw7Q/edit related to HA DNS Proxy CFP.

Need to determine L7 policy per-endpoint. Can we use epconfig.json to load per-endpoint policy, or do we need another discoverability mechanism?
This is not fast enough, but a PR to write epconfig.json on regeneration would be reasonable

Will try to do the controller approach as suggested.

@christarazi

What is the consequence if we don't update (2) on a periodic basis? [...] Does the list of ports + DNS server pods IPs (coredns) never get updated in the endpoint caches?

That's exactly right. This is caching the set of allowed IPs so we can correctly serve proxied DNS requests while the agent is starting up.

Since DNS rules are per port + destination, an endpoint could potentially have different DNS rules per destination. We also would like to be able to serve DNS requests very quickly when restarting the agent. Since a fresh agent will not know IP:Identity mappings, we cache IPs rather than identities. But then we never refresh the cache.

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

@vipul-21 have you had a chance to look at this? It would be a nice fix to have!

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

This pull request has not seen any activity since it was marked stale.
Closing.

@vipul-21 have you had a chance to look at this? It would be a nice fix to have!

@squeed Sorry for the delay.(busy with some other stuff).
Yes, let me update this.

@squeed I updated the branch with the controller, but not seeing an option to reopen this PR ? Should I create a new one ?
vipul-21@a76b698