Description:
Policy Attachment and HTTPRouteFilters represent two different extension points in Gateway API to implement features outside the core/extended APIs provided by the upstream project.

Envoy Gateway is moving forward with implementing more advanced features on top of the core Gateway API and part of this is developing guidelines for implementers/users on how the project will use Policies and Filters.

In this document I wrote up a design/discussion for Contour on differences between using Policies and Filters: projectcontour/contour#4749 which should be relevant to this project as well. In particular the differences between implementing the case-study spikes for rate limiting as a Policy vs. Filter.

Some other key points to discuss that may or may not be included in the above document:

• The Policy Attachment spec has content about Policies and Filters serving different purposes here: https://gateway-api.sigs.k8s.io/references/policy-attachment/#interaction-with-custom-route-filters (particular about them not being used to set the same configuration fields)
  • If we want to use Policies to set defaults or overrides for Filters, we need to address this
• How users know what configuration is actually applied/in effect on their data-plane is an open issue, especially for Policy Attachment
  • See: Policy Attachment: Priority/order of generated config when multiple Policies applied to a resource kubernetes-sigs/gateway-api#1498 (comment)
  • While we can create a powerful override/default hierarchy to allow flexible configuration, if users do not know what configuration is actually in effect, that is not a great experience
  • Adding Policies influencing how Filters work adds another dimension to this, possibly convoluted user experience
• Mentioned in the document linked above, but Filters in the HTTPRoute sense to me do not directly translate to Envoy HTTP Filters, as they are configured at a much different scope
  • They may work in some cases where there are per-Route incarnations of these filters in Envoy, but that does not exist for every filter, rather most filters are created per HTTPConnectionManager FilterChain