Skip to content

router: consume all retry related headers#11913

Merged
snowp merged 7 commits intoenvoyproxy:masterfrom
numerodix:consume-retry-headers-from-request
Jul 8, 2020
Merged

router: consume all retry related headers#11913
snowp merged 7 commits intoenvoyproxy:masterfrom
numerodix:consume-retry-headers-from-request

Conversation

@numerodix
Copy link
Copy Markdown
Contributor

Signed-off-by: Martin Matusiak numerodix@gmail.com

Commit Message: router: consume all retry related headers
Additional Description: Update the router to consume all retry related headers, to prevent them being propagated to the upstream.
Risk Level: low
Testing: Added new unit tests & did manual testing (with the runtime feature both on and off).
Docs Changes: None
Release Notes: Added
Runtime guard: envoy.reloadable_features.consume_all_retry_headers

I was testing retries when I discovered what seems like a bug with the handling of retry headers. If I make a request setting lots of different retry related headers like so:

curl -v \
  -H 'x-envoy-retry-on: retriable-status-codes,retriable-header-names' \
  -H 'x-envoy-retriable-status-codes: 500' \
  -H 'x-envoy-retriable-header-names: X-Retry-Always' \
  -H 'x-envoy-max-retries: 2' \
  -H 'x-envoy-hedge-on-per-try-timeout: true' \
  -H 'x-envoy-upstream-rq-per-try-timeout-ms: 2' \
localhost:8080

:8080 is envoy proxying to a trivial http server that just echoes what it receives. What I'm seeing envoy send is:

host: localhost:8080
user-agent: curl/7.68.0
accept: */*
x-envoy-retriable-status-codes: 500
x-envoy-retriable-header-names: X-Retry-Always
x-forwarded-for: 192.168.1.110
x-forwarded-proto: http
x-envoy-internal: true
x-request-id: fdc5b240-d3e5-4ec3-8c2e-5ceaa15901eb
content-length: 0

Most of the x-envoy-XXX headers have been removed/consumed by the retry implementation, but x-envoy-retriable-status-codes and x-envoy-retriable-header-names have not.

I see two issues with this:

  • This behavior does not seem "hygienic". Envoy is not ignoring these headers (eg. because they are untrusted, or because it doesn't think they are meant for itself), it is acting on them. But it is also letting them propagate.
  • These headers could be used (inadvertently or intentionally) to control the "next" envoy. If upstream is also an envoy, and has configured retry_on: retriable_status_codes,retriable_header_names then its policy could be partly overridden.

@numerodix
Copy link
Copy Markdown
Contributor Author

/assign @alyssawilk

@repokitteh-read-only
Copy link
Copy Markdown

numerodix is not allowed to assign users.

🐱

Caused by: a #11913 (comment) was created by @numerodix.

see: more, trace.

@numerodix
Copy link
Copy Markdown
Contributor Author

/retest

@repokitteh-read-only
Copy link
Copy Markdown

🤷‍♀️ nothing to rebuild.

🐱

Caused by: a #11913 (comment) was created by @numerodix.

see: more, trace.

@numerodix
Copy link
Copy Markdown
Contributor Author

numerodix commented Jul 7, 2020

/retest

1 similar comment
@numerodix
Copy link
Copy Markdown
Contributor Author

/retest

@repokitteh-read-only
Copy link
Copy Markdown

🤷‍♀️ nothing to rebuild.

🐱

Caused by: a #11913 (comment) was created by @numerodix.

see: more, trace.

Copy link
Copy Markdown
Contributor

@snowp snowp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this looks good, just a couple nits

@numerodix
Copy link
Copy Markdown
Contributor Author

Woops, looks like I just missed 1.15.0.
I'll need to wait for docs/root/version_history/current.rst to be renamed to docs/root/version_history/v1.15.0.rst and then insert the release note into the new docs/root/version_history/current.rst.

numerodix added 3 commits July 8, 2020 09:39
Also remove `x-envoy-retriable-header-names` and
`x-envoy-retriable-status-codes` to avoid these headers being
propagated to the upstream.

Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
@numerodix numerodix force-pushed the consume-retry-headers-from-request branch from 2eddc3d to 4e840f1 Compare July 7, 2020 23:57
@numerodix
Copy link
Copy Markdown
Contributor Author

Rebased, updated the release note and force pushed.

@numerodix
Copy link
Copy Markdown
Contributor Author

/retest

@repokitteh-read-only
Copy link
Copy Markdown

🤷‍♀️ nothing to rebuild.

🐱

Caused by: a #11913 (comment) was created by @numerodix.

see: more, trace.

numerodix added 2 commits July 8, 2020 11:55
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
@numerodix
Copy link
Copy Markdown
Contributor Author

/retest

@repokitteh-read-only
Copy link
Copy Markdown

🤷‍♀️ nothing to rebuild.

🐱

Caused by: a #11913 (comment) was created by @numerodix.

see: more, trace.

numerodix added 2 commits July 8, 2020 15:48
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Copy link
Copy Markdown
Contributor

@snowp snowp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@snowp snowp merged commit a3ef3e7 into envoyproxy:master Jul 8, 2020
scheler pushed a commit to scheler/envoy that referenced this pull request Aug 4, 2020
Also remove `x-envoy-retriable-header-names` and
`x-envoy-retriable-status-codes` to avoid these headers being
propagated to the upstream.

Signed-off-by: Martin Matusiak <numerodix@gmail.com>
Signed-off-by: scheler <santosh.cheler@appdynamics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants