Skip to content

router: new methods for stream filter callback to get route per filter config#21525

Merged
mattklein123 merged 28 commits intoenvoyproxy:mainfrom
wbpcode:dev-pre-work-for-custom-route-config
Jun 23, 2022
Merged

router: new methods for stream filter callback to get route per filter config#21525
mattklein123 merged 28 commits intoenvoyproxy:mainfrom
wbpcode:dev-pre-work-for-custom-route-config

Conversation

@wbpcode
Copy link
Copy Markdown
Member

@wbpcode wbpcode commented Jun 1, 2022

Commit Message: router: new methods for stream filter callback to get route per filter config
Additional Description:

The final goal is to support searching for route level filter config from typed_per_filter_config by the custom filter config name or even dynamic value from request.

Check #12274 and #21466 for complete context.

Regardless of which policy is uesed to select the route config searching key, there is some pre-work need to be addresed for the implementation.

This initial commit is used as a quick example for discussions.

Risk Level: Low.
Testing: Waiting.
Docs Changes: n/a.
Release Notes: n/a.
Platform Specific Features: n/a.

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 1, 2022

It would be hard that let every filter handle the filter name/custom name key searching by itself.

IMO, it would be better that using Http::StreamFilterCallbacks as a proxy and provides filters a method by the Http::StreamFilterCallbacks to get route config.
The filters need to use the Http::StreamFilterCallbacks's method rather than the method provided by the Route.

It means that the HTTP filter should get route config by the way as following code.

  // Http::StreamDecoderFilter
  Http::FilterHeadersStatus decodeHeaders(Http::RequestHeaderMap&, bool) override {
    auto route_config = decode_callback_->mostSpecificPerFilterConfig();
    return Http::FilterHeadersStatus::Continue;
  }

How?

First, we need make the custom name/filter name available for the Http::StreamFilterCallbacks. We can add new method to Http::StreamFilterBase to provide these name.

class StreamFilterBase {
public:
  /**
   * Get custom filter name of current filter instance. This name typically should be the
   * name of HttpFilter proto configuration.
   */
  virtual absl::string_view customName() const { return {}; }

  /**
   * Get filter name of current filter instance.
   */
  virtual absl::string_view filterName() const { return {}; }
};

Second, we can add method mostSpecificPerFilterConfig and traversePerFilterConfig to the Http::StreamFilterCallbacks.

class StreamFilterCallbacks {
  /**
   * This is a helper to get the route's per-filter config if it exists, otherwise the virtual
   * host's. Or nullptr if none of them exist.
   */
  virtual const RouteSpecificFilterConfig* mostSpecificPerFilterConfig() const PURE;

  /**
   * Fold all the available per route filter configs, invoking the callback with each config (if
   * it is present). Iteration of the configs is in order of specificity. That means that the
   * callback will be called first for a config on a Virtual host, then a route, and finally a route
   * entry (weighted cluster). If a config is not present, the callback will not be invoked.
   */
  virtual void traversePerFilterConfig(std::function<void(const Router::RouteSpecificFilterConfig&)> cb) const PURE;
};

Third, update FilterFactoryCb to provide custom filter name. Then when we create the filter instance, we can set the custom name to the filter.

/**
 * This function is used to wrap the creation of an HTTP filter chain for new streams as they
 * come in. Filter factories create the function at configuration initialization time, and then
 * they are used at runtime.
 * @param custom_name custom name of the filter that created by this factory.
 * @param callbacks supplies the callbacks for the stream to install filters to. Typically the
 * function will install a single filter, but it's technically possibly to install more than one
 * if desired.
 */
using FilterFactoryCb = std::function<void(absl::string_view custom_name, FilterChainFactoryCallbacks& callbacks)>;

Http::FilterFactoryCb FilterFactory::createFilterFactoryFromProtoTyped(const FakePb&, const std::string& stats_prefix,
    Server::Configuration::FactoryContext& context) {
  return [config](absl::string_view custom_name, Http::FilterChainFactoryCallbacks& callbacks) -> void {
    auto fitler = std::make_shared<CorsFilter>(config);
    filter->setNames(custom_name, name());
    callbacks.addStreamFilter(std::move(filter));
  };
}

Fouth, upate every filter to use new mostSpecificPerFilterConfig and traversePerFilterConfig provided by the Http::StreamFilterCallbacks.

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 1, 2022

cc @mattklein123 Could you take a quick look to this simple proposal?

@kyessenov
Copy link
Copy Markdown
Contributor

Looks reasonable to me. One question I have is whether we have to expose the custom name method. The filter manager should have that info, and only share it with the callbacks implementation, without adding another public method.

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 2, 2022

Looks reasonable to me. One question I have is whether we have to expose the custom name method. The filter manager should have that info, and only share it with the callbacks implementation, without adding another public method.

Thanks for you comment. But in the current implementation, the filter instance is added to filter chain by the callback as following. So when we create the filter chain, it's hard to make the custom name available directly without change the callback or add new another public method.

class FilterChainFactoryCallbacks {
  virtual void addStreamDecoderFilter(Http::StreamDecoderFilterSharedPtr filter) PURE;
};

@mattklein123
Copy link
Copy Markdown
Member

So when we create the filter chain, it's hard to make the custom name available directly without change the callback or add new another public method.

As long as you are working in this area, can you please fix all debug lines like this one:

ENVOY_STREAM_LOG(trace,
"decodeHeaders filter iteration aborted due to local reply: filter={}",
*this, static_cast<const void*>((*entry).get()));

To print out the name instead of the pointer address of the filter? This will make debugging multiple filters so much easier. I actually started to do this at one point but gave up because it would require a new public method which I think this solves.

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 7, 2022

But since we all agree that the current configuration is too heavy, I will respect the community's opinion.

👌 I will update all these log line.

wbpcode added 2 commits June 8, 2022 09:34
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
wbpcode added 2 commits June 9, 2022 17:34
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
@wbpcode wbpcode marked this pull request as draft June 9, 2022 17:43
wbpcode added 9 commits June 10, 2022 02:34
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
…an work correctly

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 13, 2022

This is outdated.

Inherited relationships before this PR of HTTP stream filter.

graph TD
  BA(StreamFilterBase)
  SD(StreamDecoderFilter)
  SE(StreamEncoderFilter)
  SF(StreamFilter)
  PD(PassthroughDecoderFilter)
  PE(PassthroughEncoderFilter)
  PF(PassthroughFilter)

  BA --> SD
  BA --> SE
  SD --virtual--> PD
  SD --virtual--> SF
  SE --virtual--> SF
  SE --virtual--> PE
  PD --> PF
  SF --> PF
  PE --> PF

Loading

Inherited relationships after this PR of HTTP stream filter:

graph TD
  BA(StreamFilterBase)
  SD(StreamDecoderFilter)
  SE(StreamEncoderFilter)
  SF(StreamFilter)
  PD(PassthroughDecoderFilter)
  PE(PassthroughEncoderFilter)
  PF(PassthroughFilter)

  BA --virtual--> SD
  BA --virtual--> SE
  SD --> PD
  SD --> SF
  SE --> SF
  SE --> PE
  SF --> PF

Loading

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Copy link
Copy Markdown
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait-any

// Struct of canonical filter name and HTTP stream filter factory callback.
struct NamedHttpFilterFactoryCb {
// Canonical filter name.
std::string name;
Copy link
Copy Markdown
Member

@soulxu soulxu Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to put this name to the FilterConfigProvider also?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to merge this in some way. I'm not familiar with this code but it seems like there is a bunch of duplication with the FilterFactoryContext also? Can this be shared somehow?

Copy link
Copy Markdown
Member Author

@wbpcode wbpcode Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have just tried to add a method to expose this name in the FilterConfigProvider. But it just make the code more complex and no better than current implementation.

Because it's supported to used different type filters by the ECDS, so the canonical filter name of current provider may be changed when the filter config is updateing by the ECDS. We must make the canonical filter name thread local and update it just as we update factory function in the FilterConfigProvider.
So it's hard to used FilterFactoryContext to store these name in the FilterConfigProvider.

cc @mattklein123 @soulxu

Copy link
Copy Markdown
Member Author

@wbpcode wbpcode Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message ExtensionConfigSource {
  ConfigSource config_source = 1 [(validate.rules).any = {required: true}];

  // Optional default configuration to use as the initial configuration if
  // there is a failure to receive the initial extension configuration or if
  // `apply_default_config_without_warming` flag is set.
  google.protobuf.Any default_config = 2;

  // Use the default config as the initial configuration without warming and
  // waiting for the first discovery response. Requires the default configuration
  // to be supplied.
  bool apply_default_config_without_warming = 3;
  
  // **Here the repeated type_urls make the filter type can be changed by the ECDS. A ratelimit filter can be** 
  // **updated to ext_authz.**
  // A set of permitted extension type URLs. Extension configuration updates are rejected
  // if they do not match any type URL in the set.
  repeated string type_urls = 4 [(validate.rules).repeated = {min_items: 1}];
}

IMO, it would be better to add some restrictions here. For example, only multiple version APIs of same type filter can be used here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, it would be better to add some restrictions here. For example, only multiple version APIs of same type filter can be used here.

I guess even though there can be multiple extension type URLs, they should pointed to the same factory, or they should point to the same canonical filter name?

for (const auto& type_url : config_discovery.type_urls()) {
auto factory_type_url = TypeUtil::typeUrlToDescriptorFullName(type_url);
auto* factory = Registry::FactoryRegistry<
Server::Configuration::NamedHttpFilterConfigFactory>::getFactoryByType(factory_type_url);
if (factory == nullptr) {
throw EnvoyException(
fmt::format("Error: no factory found for a required type URL {}.", factory_type_url));
}
}

Then you can get factory->name(), and pass that to the dynamic filter config provider.

auto filter_config_provider = filter_config_provider_manager_.createDynamicFilterConfigProvider(
config_discovery, name, context_, stats_prefix_, last_filter_in_current_config,
filter_chain_type, nullptr);

@adisuissa probably knows the usecase of multiple extension type urls.

Copy link
Copy Markdown
Member Author

@wbpcode wbpcode Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess even though there can be multiple extension type URLs, they should pointed to the same factory, or they should point to the same canonical filter name?

It seems that there is no any restriction for now. It would be great if we can add this restriction. But it would be a break change.
So, I think we can keep this PR's implemantation and after this restriction is shipped someday, we can update the implemantation with a new minor patch.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, it seems like this is the best we have now. Thanks!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kyessenov Can probably shed some light on multiple extensions type-urls

Copy link
Copy Markdown
Member

@soulxu soulxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this way. Just one more comment about whether we can put the all the name to the FilterConfigProvider. Then we needn't that NamedHttpFilterFactoryCb wrapper.

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 21, 2022

I like this way. Just one more comment about whether we can put the all the name to the FilterConfigProvider. Then we needn't that NamedHttpFilterFactoryCb wrapper.

It's ok for me. But I am just not sure if we need to expose this canonical name for every type extension. Until for now, the canonical name is only used by the HTTP filters to get route config. And we keep it for the backwards compatibility. So I am not sure if it's worth it to update the APIs of ExtensionConfigProvider.

But I'm not against this either. I can update it if the community wishes to do so.

cc @mattklein123 cc @soulxu

@soulxu
Copy link
Copy Markdown
Member

soulxu commented Jun 21, 2022

I like this way. Just one more comment about whether we can put the all the name to the FilterConfigProvider. Then we needn't that NamedHttpFilterFactoryCb wrapper.

It's ok for me. But I am just not sure if we need to expose this canonical name for every type extension. Until for now, the canonical name is only used by the HTTP filters to get route config. And we keep it for the backwards compatibility. So I am not sure if it's worth it to update the APIs of ExtensionConfigProvider.

Since I saw the FilterConfigProvider already holds the custom name,https://github.com/envoyproxy/envoy/pull/21525/files#diff-9ffd7a166e0bd6afee8cbe95c75544a26060e9d2bc814ba3309fa4c04d8860ecR738
so I thought the FilterConfigProvider can hold the canonical name also

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 21, 2022

I like this way. Just one more comment about whether we can put the all the name to the FilterConfigProvider. Then we needn't that NamedHttpFilterFactoryCb wrapper.

It's ok for me. But I am just not sure if we need to expose this canonical name for every type extension. Until for now, the canonical name is only used by the HTTP filters to get route config. And we keep it for the backwards compatibility. So I am not sure if it's worth it to update the APIs of ExtensionConfigProvider.

Since I saw the FilterConfigProvider already holds the custom name,https://github.com/envoyproxy/envoy/pull/21525/files#diff-9ffd7a166e0bd6afee8cbe95c75544a26060e9d2bc814ba3309fa4c04d8860ecR738 so I thought the FilterConfigProvider can hold the canonical name also

custom name is used by the ECDS, but canonical name is unnecessary for most of extention config providers.

Any way, it ok for me to udpate it or not. So waiting for some more suggessions from matt.

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 21, 2022

/retest

@repokitteh-read-only
Copy link
Copy Markdown

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #21525 (comment) was created by @wbpcode.

see: more, trace.

Copy link
Copy Markdown
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks in general this LGTM. Flushing some small comments. Thank you!

/wait

// Struct of canonical filter name and HTTP stream filter factory callback.
struct NamedHttpFilterFactoryCb {
// Canonical filter name.
std::string name;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to merge this in some way. I'm not familiar with this code but it seems like there is a bunch of duplication with the FilterFactoryContext also? Can this be shared somehow?

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 22, 2022

/retest

@repokitteh-read-only
Copy link
Copy Markdown

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #21525 (comment) was created by @wbpcode.

see: more, trace.

Copy link
Copy Markdown
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks LGTM with one question.

/wait-any

Comment on lines +1089 to +1091
// Canonical filter name. This is used to indicate type of filter. For example,
// "envoy.filters.http.buffer" for the HTTP buffer filter.
std::string filter_name;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we don't use this anymore? Should this actually be the type URL? cc @kyessenov @phlax as this also relates to #21707 I think?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still necessary for now. Because this name will be used to search the per filter config, as a fallback of config name. 🤔

If we deprecate the fallback policy and used the config name only someday, we can remove this name.

Copy link
Copy Markdown
Contributor

@kyessenov kyessenov Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the phrasing needs to be better.

DEPRECATED: Filter extension qualified name. Used as a fallback for `config_name`.  E.g. ...

This is needed to support extension names in per_filter_config I think, so we can't change it to anything but what it was (which was extension qualified name).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get it.

soulxu
soulxu previously approved these changes Jun 23, 2022
Copy link
Copy Markdown
Member

@soulxu soulxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

Signed-off-by: wbpcode <wangbaiping@corp.netease.com>
Copy link
Copy Markdown
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@wbpcode
Copy link
Copy Markdown
Member Author

wbpcode commented Jun 23, 2022

/retest

@repokitteh-read-only
Copy link
Copy Markdown

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #21525 (comment) was created by @wbpcode.

see: more, trace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants