Skip to content

Envoy crashing with NOT_REACHED_GCOVR_EXCL_LINE when loading YAML config using --config-yaml #3990

@scaldwell-atlassian

Description

@scaldwell-atlassian

Bug Template

Title: One line description

Description:

Problem Statement:
When using —config-yaml , we encounter a panic in json_loader.cc at line 315 (NOT_REACHED_GCOVR_EXCL_LINE), which is defined as a panic. This panic is defined in assert.h at line 60 as an outcome we should never see unless code execution either (a) follows an unexpected path or (b) a “horrendous” bug exists.
Investigating the bug further, we find that the error is in a switch…case statement in json_loader.cc (which starts at line 246) during buildRapidJsonDocument(). This appears to be a type checking block, which tests for Arrays, Objects and Null values. The method has likely reached some input on
the YAML file which is not one of these data types, but the output does not tell us anything further.

Expected behavior:
Envoy should return an error message that guides the administrator in correcting the problem.

Repro steps:

Use minimum YAML config from documents with --config-yaml option and observe output to stdout/err when running envoy from commandline. At best there will be a silent error and the proxy will start. At worst case, the service will not start and an error will be seen on screen (seel below).

Config:

Issue was reproduced using google_com_proxy.yaml and google_com_proxy_v2.yaml included with software.

Logs:

Include the access logs and the Envoy logs.

root@89279f2e0b72:/# $(which envoy) --v2-config-only -l DEBUG --config-yaml /etc/envoy/envoy.yaml
[2018-07-30 21:40:04.499][243][info][main] source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:185] statically linked extensions:
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:187] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:190] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:193] filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:196] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:198] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:200] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:203] transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:206] transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-07-30 21:40:04.500][243][critical][assert] source/common/json/json_loader.cc:315] panic: not reached
[2018-07-30 21:40:04.500][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0xf3
[2018-07-30 21:40:04.500][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<0> obj</lib/x86_64-linux-gnu/libc.so.6> (If unsymbolized, use tools/stack_decode.py):
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #0 0x7ff6dff4d428 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #1 0x7ff6dff4f029 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</usr/local/bin/envoy>
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #2 0x9acc2c (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #3 0x9ad287 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #4 0x9b3ab0 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #5 0x9a2687 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #6 0x6a8f00 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #7 0x6a98a0 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #8 0x6ac655 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #9 0x477e66 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #10 0x4781e2 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #11 0x418015 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #12 0x7ff6dff3882f (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</usr/local/bin/envoy>
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #13 0x46e1c8 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 0
Aborted (core dumped)

Call Stack:

see above

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugquestionQuestions that are neither investigations, bugs, nor enhancements

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions