Skip to content

Ensure http channel validator does not alter ES thread context#95498

Merged
albertzaharovits merged 3 commits intoelastic:mainfrom
albertzaharovits:netty4-http-header-validator-with-thread-context
Apr 25, 2023
Merged

Ensure http channel validator does not alter ES thread context#95498
albertzaharovits merged 3 commits intoelastic:mainfrom
albertzaharovits:netty4-http-header-validator-with-thread-context

Conversation

@albertzaharovits
Copy link
Copy Markdown
Contributor

This PR ensures that the validator (of http requests over the netty
channel) is unable to alter the ES context of netty worker threads.
The netty worker threads are oblivious to the ES context, but they
eventually call into the ES code, at which point any tampering of
the context during the validation is visible and potentially dangerous.

@albertzaharovits albertzaharovits marked this pull request as ready for review April 24, 2023 15:05
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team. label Apr 24, 2023
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

}

private void forwardFullRequest(ChannelHandlerContext ctx) {
Transports.assertDefaultThreadContext(threadContext);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ on the asserts here (and anywhere for the thread context - it is very helpful). however, could be outside the context of this PR but it would be great if only had 1 definition of what default thread context means.

org.elasticsearch.transport.Transports#assertDefaultThreadContext defines it as

    assert threadContext.getRequestHeadersOnly().isEmpty()
            || REQUEST_HEADERS_ALLOWED_ON_DEFAULT_THREAD_CONTEXT.containsAll(threadContext.getRequestHeadersOnly().keySet())
            : "expected empty context but was " + threadContext.getRequestHeadersOnly() + " on " + Thread.currentThread().getName();

and org.elasticsearch.common.util.concurrent.ThreadContext#isDefaultContext defines it as

return threadLocal.get() == DEFAULT_CONTEXT;

I understand they are slightly different, but ideally they would not be and we would only have a single definition. Not needed to address on this PR , just wanted to get your thoughts if it is possible to collapse these 2 into 1.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally they would not be and we would only have a single definition.

++ Looked into this briefly too. I would've expected threadLocal.get() == DEFAULT_CONTEXT to be used everywhere inplace of threadContext.getRequestHeadersOnly().isEmpty() ..... I tried it and it failed intermitently on one of the license IT, where the thread context contained the license expiration warning response header. I think that response headers might be the reason for the different ways to check the default context....
I haven't investigated further... it might be a bug with response headers leaking....

I'll make a note to follow-up on this when I'll feel out the woods...

Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@albertzaharovits
Copy link
Copy Markdown
Contributor Author

Thank you for the quick review, Jake!

@albertzaharovits
Copy link
Copy Markdown
Contributor Author

@elasticsearchmachine update branch

@albertzaharovits albertzaharovits merged commit da9d698 into elastic:main Apr 25, 2023
@albertzaharovits albertzaharovits deleted the netty4-http-header-validator-with-thread-context branch April 25, 2023 14:36
albertzaharovits added a commit to albertzaharovits/elasticsearch that referenced this pull request Jun 13, 2023
…ic#95498)

This ensures that the validator (of http requests over the netty
channel) is unable to alter the ES context of netty worker threads.
The netty worker threads are oblivious to the ES context, but they
eventually call into the ES code, at which point any tampering of
the context during the validation is visible and potentially dangerous.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Network Http and internode communication implementations >non-issue Team:Distributed Meta label for distributed team. v8.8.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants