Inject Filter SEGVs during aborted gRPC request to Inject Server #1
Description
Inject filter tries to talk to gRPC service to get the headers to Inject using the AsyncClientStreamImpl (kicked off in inject filter decodeHeaders). There are handshake issues with a stubbed out golang implementation of that inject.proto service (SSL WRONG VERSION, separate issue) which causes the gRPC request to to reset which then seems to trigger issues in an apparently under-configured router (?). Likely a mistake in inject decodeHeaders/Filter.
Help please @mattklein123 if you have a few cycles
To create a core dump, checkout this project, init the git submodule as per README.md, then inside the lyft/envoy-build Linux container image with this source mounted, in envoy-filter-example folder (this repo):
bazel build --define=signal_trace=disabled -c dbg //:envoy
ulimit -c unlimited
bazel-bin/envoy -l trace -c inject_server.json &
curl -v -H "x-myco-authn: a" -H "x-myco-jwt: 339" localhost:8080/service/ffo
Boom, envoy segv -> core file
gdb bazel-bin/envoy core
(gdb) bt
Shows that it fails here (router not init'd with decodeHeaders call?):
router.cc
void Filter::chargeUpstreamCode(const Http::HeaderMap& response_headers,
Upstream::HostDescriptionConstSharedPtr upstream_host) {
if (config_.emit_dynamic_stats_ && !callbacks_->requestInfo().healthCheck()) {
const Http::HeaderEntry* upstream_canary_header = response_headers.EnvoyUpstreamCanary();
const Http::HeaderEntry* internal_request_header = downstream_headers_->EnvoyInternalRequest(); <-- downstream_headers_ is null. boom
If you protect that for null it fails here:
http/utility.cc
void Utility::sendLocalReply(StreamDecoderFilterCallbacks& callbacks, Code response_code,
const std::string& body_text) {
HeaderMapPtr response_headers{
new HeaderMapImpl{{Headers::get().Status, std::to_string(enumToInt(response_code))}}};
if (!body_text.empty()) {
response_headers->insertContentLength().value(body_text.size());
response_headers->insertContentType().value(Headers::get().ContentTypeValues.Text);
}
callbacks.encodeHeaders(std::move(response_headers), body_text.empty());
if (!body_text.empty()) {
Buffer::OwnedImpl buffer(body_text);
callbacks.encodeData(buffer, true); <-- here. Callbacks non-null but has crap ptr inside it
}
}
Both above calls are kicked off at end this method of router.cc:
void Filter::onUpstreamReset(UpstreamResetType type, const Optional<Http::StreamResetReason>& reset_reason) {
...
chargeUpstreamCode(code, upstream_host); <- boom
Http::Utility::sendLocalReply(*callbacks_, code, body); <- boom
}
}