PolicyAuditMode Disabled x

...

... only configuration options that decide what logs are sent to cilium monitor can be changed for endpoints ...

Not sure this is consistent with the code :-)

$ git grep -A 7 AUDIT bpf/lib/
bpf/lib/policy.h:#ifdef POLICY_AUDIT_MODE
bpf/lib/policy.h-       if (IS_ERR(ret)) {
bpf/lib/policy.h-               ret = CTX_ACT_OK;
bpf/lib/policy.h-               *audited = 1;
bpf/lib/policy.h-       }
bpf/lib/policy.h-#endif
bpf/lib/policy.h-
bpf/lib/policy.h-       return ret;
--
bpf/lib/policy.h:#ifdef POLICY_AUDIT_MODE
bpf/lib/policy.h-       if (IS_ERR(ret)) {
bpf/lib/policy.h-               ret = CTX_ACT_OK;
bpf/lib/policy.h-               *audited = 1;
bpf/lib/policy.h-       }
bpf/lib/policy.h-#endif
bpf/lib/policy.h-       return ret;
bpf/lib/policy.h-}

That said I suspect that you're specifically interested in allowing POLICY_AUDIT_MODE to influence the host endpoint.

Taking a step back and looking at the reasoning behind the config restrictions endpoints with reserved:___ labels, I had originally only thought as far as the health endpoint. For that endpoint, I figured it's entirely managed by Cilium internally and users would not (and should not) tinker with it; At the time we were considering extending it to perform regular services / policy checks to validate underlying connectivity and I didn't want to somehow end up in a situation where we rely on specific datapath configuration in the logic that manages the health checks, but then a user inadvertently modifies one of the endpoints and causes those checks to fail.

I don't think there's much of a security barrier question here, because even if we were to continue to disallow something like the CONNTRACK or this POLICY_AUDIT_MODE option from being configured via the API, someone with local access could still import directly a policy that allows some set of traffic. Cilium daemon would argue with it and likely clear such a policy out, re-syncing with k8s from time to time but I guess my point is that local socket access to the API is not considered a barrier between security domains; someone with access to the Cilium unix socket is likely to be able to do much more dangerous things than tweaking these specific datapath options.

At this point I don't think many of the original arguments really apply. Even for the health endpoint, I don't think we got to a point where it really made a difference and most likely the only folks who know about this API or regularly use it would be Cilium developers anyway. So from that perspective I'm OK with loosening these restrictions.

Regarding enabling datapath debug, I think it's reasonable to locally get access to a Cilium socket and enable debug via CLI / API like this. I think if we really wanted to make this usable we'd probably support some kind of pod or node annotation that Cilium respected, but I'd guess that's more effort than it's worth for what you're trying to achieve here.

PolicyAuditMode Disabled x

...

... only configuration options that decide what logs are sent to cilium monitor can be changed for endpoints ...

Not sure this is consistent with the code :-)

Ouch, yes. I've updated the commit message to clarify that audit mode can also be changed.

That said I suspect that you're specifically interested in allowing POLICY_AUDIT_MODE to influence the host endpoint.

Yes, I'd like to be able to switch the host firewall in audit mode without switching all of Cilium. There's still some work required to allow that AFAICS.
I'm also interested in being able to easily switch the host endpoint's debug mode and general verbosity (ex. log aggregation), but that's mostly for debugging and troubleshooting.

In general, my approach was to only allow setting configurations that shouldn't have a huge impact on forwarding (except in case of complexity issues of course...) to prevent users from shooting themselves in the foot too easily. The audit mode is a bit of a corner case here, but as far as I can see, switching it on shouldn't result in any connectivity breakage.

test-me-please