Implementing XFCC header#1087
Conversation
| return temporaryFileSubstitute(path, ParamMap(), port_map, version); | ||
| } | ||
|
|
||
| std::string TestEnvironment::temporaryFileSubstitute(const std::string& path, |
There was a problem hiding this comment.
I don't want to pick on this change, as the problem was building up before, but we've reached the point where my initial hack in temporaryFileSubstitute to do some simple regex in a Jinja-style is now not scaling all that well. We can iterate over the same config files probably O(10-100) times now, which doesn't seem great. I don't think you need to fix it, but maybe put in a TODO(htuch): Switch to a single pass variable substitution..
There was a problem hiding this comment.
Yes I agree. So I will put a TODO for you, and I'll keep this change because I need to make this json file configurable in my integration tests.
There was a problem hiding this comment.
Sure, a TODO and filing an issue is the right thing here.
There was a problem hiding this comment.
Can you add this TODO please and file a related issue @myidpt ?
There was a problem hiding this comment.
Done. I put the TODO in the .h file but I can move it to here.
|
|
||
|
|
||
| if (connection.ssl() || config.forwardClientCert() == Http::ForwardClientCertType::AlwaysForwardOnly) { | ||
| std::string clientCertDetails = ""; |
| request_headers.removeForwardedClientCert(); | ||
| break; | ||
| case Http::ForwardClientCertType::SanitizeSet: | ||
| request_headers.removeForwardedClientCert(); |
There was a problem hiding this comment.
not needed, insert will replace existing one.
| break; | ||
| case Http::ForwardClientCertType::AppendForward: | ||
| // Get the Cert info. | ||
| if (!request_headers.ForwardedClientCert()->value().empty()) { |
There was a problem hiding this comment.
request_headers.ForwardedClientCert() might be nullptr
htuch
left a comment
There was a problem hiding this comment.
Looks like this is coming together nicely.
|
|
||
| /** | ||
| * @return ClientCertDetailsType the details of the client cert to forward to the next hop. | ||
| */ |
There was a problem hiding this comment.
The name of this method is strange. From its name, it seems to imply it is mutating (i.e. it sets something), but it takes no arguments. Can you explain how this is supposed to be used more clearly? Also, it returns a list of ClientCertDetailsType, which is also a bit confusing, since this is basically just a Subject/SAN enum, why a list of this?
There was a problem hiding this comment.
Yes, the naming is simply following the routine to be the same name of the field in config. This function is directly returning the set_client_cert_details field in the configuration, that indicates which information from the current client cert should be set into the x-forwarded-client-cert header. It can be a list because there are cases you want to set both subject and SAN into the header, and in the future more information may be needed as well.
| // TODO. Get Cert Subject. | ||
| } else if (detail == Http::ClientCertDetailsType::SAN) { | ||
| // Currently, we only support a single SAN field. | ||
| clientCertDetails += ";SAN=" + connection.ssl()->uriSanPeerCertificate(); |
There was a problem hiding this comment.
We need to document this header format and the configuration of this features in https://github.com/lyft/envoy/tree/master/docs somewhere.
source/common/ssl/connection_impl.cc
Outdated
| void ConnectionImpl::onConnected() { ASSERT(!handshake_complete_); } | ||
|
|
||
| bool ConnectionImpl::isMtls() { | ||
| if (SSL_get_peer_certificate(ssl_.get())) { |
There was a problem hiding this comment.
Is it an invariant that if we've received a client cert that this cert has been appropriately verified and trusted?
There was a problem hiding this comment.
That's a good point. I think this will just return whether the client cert is present, regardless of whether the client cert is valid.
And I don't think this function should depend on whether the client cert is valid, just like the other function sha256PeerCertificateDigest().
There was a problem hiding this comment.
The implementation here is wrong in case of the connection is a client connection. We could call this peerCertificatePresented instead of isMtls, or check both certificate.
Also note the X509 pointer returned by SSL_get_peer_certificate need to be freed. Use bssl::UniquePtr.
There was a problem hiding this comment.
That makes sense. Done.
|
|
||
| if (config.hasObject("forward_client_cert")) { | ||
| // TODO: Check mTLS. | ||
| const std::string forward_client_cert = config.getString("forward_client_cert"); |
There was a problem hiding this comment.
Can you write this as const std::string forward_client_cert = config.getString("forward_client_cert", Http::ForwardClientCertType::Sanitize); to simplify default case handling?
| forward_client_cert_ = Http::ForwardClientCertType::Sanitize; | ||
| } | ||
|
|
||
| if (config.hasObject("set_client_cert_details")) { |
There was a problem hiding this comment.
BTW, can you contribute the new config stuff to https://github.com/lyft/envoy-api as well? Thanks.
There was a problem hiding this comment.
Sure. I will do that.
| @@ -0,0 +1,243 @@ | |||
| #include "xfcc_integration_test.h" | |||
There was a problem hiding this comment.
It seems overkill to create an entire new test config file for every feature we want to integrate. I think this is a general issue with integration tests today - there's no easy way to templatize the creation of the configs to turns on/off individual features while retaining a common base. This results in a lot of boiler plate to maintain. @mattklein123 for comment.
There was a problem hiding this comment.
I haven't looked through this change yet, but, it seems unnecessary for a whole new config. Can it just be part of the SSL integration test? In general, the fact there are multiple different integration test configs is really arbitrary. They could all be in a single config which just has more listeners and more cluster definitions. I'm certainly happy to figure out a way to simplify things there, but nothing comes to mind quickly, and that is orthogonal to this change. In this case I would probably just move the integration tests into the SSL one even if new listeners and upstreams need to be added.
There was a problem hiding this comment.
I originally kept it separately because the xfcc header can also be forwarded without ssl (the "always_forward_only" option). So if you take a look at the config, I have two listeners, one with ssl_context and the other not. But if it's overkill, I can merge the xfcc tests with ssl, because they are very related.
There was a problem hiding this comment.
I don't have an opinion on this. I agree there is a lot of boilerplate but at the same time it's not that bad. Whatever you both want is fine with me.
There was a problem hiding this comment.
I've created #1162 to track, let's stick with what is there for now.
| return temporaryFileSubstitute(path, ParamMap(), port_map, version); | ||
| } | ||
|
|
||
| std::string TestEnvironment::temporaryFileSubstitute(const std::string& path, |
There was a problem hiding this comment.
Sure, a TODO and filing an issue is the right thing here.
| --------------- | ||
|
|
||
| *x-forwarded-clinet-cert* (XFCC) is a proxy header which indicates certificate information of part | ||
| or all of the clients that a request has flowed through on its way from the client to the server. |
|
|
||
| The following keys are supported: | ||
|
|
||
| 1. ``BY`` The Subject Alternative Name (SAN) (Common Name if SAN is not available) of the current |
| proxy's certificate. | ||
| 2. ``Hash`` The SHA 256 diguest of the current client certificate. | ||
| 3. ``SAN`` The SAN field (URI type) of the current client certificate. There can be multiple SANs. | ||
| 4. ``Subject`` The Subject field of the current Client certificate. The value is quoted. |
There was a problem hiding this comment.
s/Client certificate/client certificate/
|
|
||
| .. _config_http_conn_man_headers_x-forwarded-for: | ||
|
|
||
| x-forwarded-client-cert |
| or all of the clients that a request has flowed through on its way from the client to the server. | ||
| A proxy may choose to sanitize/append/forward the XFCC header before proxying the reqeust. | ||
|
|
||
| The XFCC header value is a comma (",") separated string. Each substring is an XFCC element, which |
There was a problem hiding this comment.
Can any of the key/values contains = or ,? Is there a need to consider escaping?
There was a problem hiding this comment.
Yes, the value should be double-quoted. Like the "Subject" field. Updated the doc. Thanks for the catch.
|
|
||
| if ((connection.ssl() && connection.ssl()->isMtls()) || | ||
| config.forwardClientCert() == Http::ForwardClientCertType::AlwaysForwardOnly) { | ||
| std::string clientCertDetails; |
There was a problem hiding this comment.
Extract this part to a function and call it from inside switch below, so you don't always build clientCertDetails even if it is not needed below.
There was a problem hiding this comment.
Quick related drive by comment (not full review): Can we pull all of this code out of mutateRequestHeaders() into a new function that can be self contained tested? This function is getting too big as it is.
| clientCertDetails += "By=" + connection.ssl()->uriSanLocalCertificate(); | ||
| } | ||
| if (!connection.ssl()->sha256PeerCertificateDigest().empty()) { | ||
| clientCertDetails += ";Hash=" + connection.ssl()->sha256PeerCertificateDigest(); |
There was a problem hiding this comment.
If uriSanLocalCertificate() is empty, then you don't need ; here. Same applies to below.
| clientCertDetails += ";Hash=" + connection.ssl()->sha256PeerCertificateDigest(); | ||
| } | ||
| for (auto detail : config.setCurrentClientCertDetails()) { | ||
| if (detail == Http::ClientCertDetailsType::Subject) { |
source/common/ssl/connection_impl.cc
Outdated
| void ConnectionImpl::onConnected() { ASSERT(!handshake_complete_); } | ||
|
|
||
| bool ConnectionImpl::isMtls() { | ||
| if (SSL_get_peer_certificate(ssl_.get())) { |
There was a problem hiding this comment.
The implementation here is wrong in case of the connection is a client connection. We could call this peerCertificatePresented instead of isMtls, or check both certificate.
Also note the X509 pointer returned by SSL_get_peer_certificate need to be freed. Use bssl::UniquePtr.
| } | ||
|
|
||
| if (config.hasObject("set_current_client_cert_details")) { | ||
| for (const std::string detail : config.getStringArray("set_current_client_cert_details")) { |
| /** | ||
| * @return list of ClientCertDetailsType the configuration of the current client cert's details to | ||
| * be forwarded. | ||
| */ |
There was a problem hiding this comment.
nit: const std::list<Http::ClientCertDetailsType>&
include/envoy/ssl/connection.h
Outdated
| @@ -17,7 +17,7 @@ class Connection { | |||
| /** | |||
| * @return whether the connection is Mutual TLS. | |||
|
@myidpt is this ready for wider review? I will take a look tomorrow. |
|
@mattklein123 Yes, it is ready. |
mattklein123
left a comment
There was a problem hiding this comment.
This is really excellent work, thanks. Some small nits. @htuch did you want to take another pass? I think this is pretty much ready to go.
source/common/json/config_schemas.cc
Outdated
| "use_remote_address" : {"type" : "boolean"}, | ||
| "forward_client_cert" : { | ||
| "type" : "string", | ||
| "enum" : ["forward_only", "append_forward", "sanitize", "sanitize_set", "always_forward_only", "always_append_forward"] |
There was a problem hiding this comment.
nit: please break at ~100 cols
| Tracing::HttpTracerUtility::mutateHeaders(request_headers, runtime); | ||
| } | ||
|
|
||
| if ((connection.ssl() && connection.ssl()->peerCertificatePresented()) || |
There was a problem hiding this comment.
nit: I still think it might be better to have this code be a separate routine, even if private. It's a lot to look through and the mutateRequestHeaders function is already huge. (If it was a separate routine I think it would probably be less nested because you could do early return). I could go either way on this but thought I would mention.
There was a problem hiding this comment.
Separated to a private function.
| Http::ConnectionManagerTracingStats tracing_stats_; | ||
| bool use_remote_address_{}; | ||
| Http::ForwardClientCertType forward_client_cert_; | ||
| std::list<Http::ClientCertDetailsType> set_current_client_cert_details_; |
There was a problem hiding this comment.
nit: I would make this std::vector for slight perf boost
| client_san_); | ||
| } | ||
|
|
||
| /* |
There was a problem hiding this comment.
Oh sorry, forgot to remove it :P
| #include "gtest/gtest.h" | ||
|
|
||
| namespace Envoy { | ||
| using testing::NiceMock; |
There was a problem hiding this comment.
nit: please moving using statements outside of Envoy namespace (elsewhere also)
mattklein123
left a comment
There was a problem hiding this comment.
Few remaining tiny nits. LGTM pending comments from anyone else.
| // provides behavior that most consumers would expect. | ||
| static std::atomic<uint64_t> next_stream_id_; | ||
|
|
||
| static void mutateXfccRequestHeader(Http::HeaderMap& request_headers, |
There was a problem hiding this comment.
nit: function definitions go above variable definitions
| request_headers.removeForwardedClientCert(); | ||
| return; | ||
| } | ||
| std::string clientCertDetails; |
There was a problem hiding this comment.
nit: This function is super dense and a bit hard to dive into. Can we get some high level comments before line 140, 145, and 178 on what is about to happen? Would help for others coming along.
| using testing::Return; | ||
|
|
||
| namespace Envoy { | ||
|
|
There was a problem hiding this comment.
super small nit: del new line between namespace decls when nothing in between (same in other files)
| public: | ||
| virtual ~Connection() {} | ||
|
|
||
| /** |
| request_headers.removeForwardedClientCert(); | ||
| return; | ||
| } | ||
| std::string clientCertDetails; |
There was a problem hiding this comment.
Since clientCertDetails is only used when SanitizeSet or AppendForward set, process other cases before building it. Also, append directly to header->value() might be more efficient.
There was a problem hiding this comment.
Reorganized the code. I don't know what you meant by append directly to header->value(), that seems exactly what I'm doing.
There was a problem hiding this comment.
I meant you can directly use append(), since you're only doing += for string, i.e. something similar to here:
https://github.com/lyft/envoy/blob/master/source/common/grpc/common.cc#L137
Instead of even building a std::string. That will avoid to copy that when you append at the end. It is OK to optimize that based on perf analysis.
There was a problem hiding this comment.
I see. I think that can be done as part of the code optimization (I added a TODO here). We may want to have an inline function to do that.
htuch
left a comment
There was a problem hiding this comment.
Looks great. Final round of minor comments.
| virtual std::string uriSanLocalCertificate() PURE; | ||
|
|
||
| /** | ||
| * @return the SHA256 digest of the peer certificate. Returns "" if there is no peer certificate |
There was a problem hiding this comment.
What is a "server side connection"?
There was a problem hiding this comment.
I think the original author means TLS instead of mTLS. I will update it.
| return temporaryFileSubstitute(path, ParamMap(), port_map, version); | ||
| } | ||
|
|
||
| std::string TestEnvironment::temporaryFileSubstitute(const std::string& path, |
There was a problem hiding this comment.
Can you add this TODO please and file a related issue @myidpt ?
|
|
||
| Each XFCC element is a semicolon ";" separated string. Each substring is a key-value pair, grouped | ||
| together by an equals ("=") sign. The keys are case-insensitive, the values are case-sensitive. If | ||
| ",", ";" or "=" appear in a value, the value should be double-quoted. |
There was a problem hiding this comment.
What if " appears inside the double-quoted string? Is there a need to escape this? If it can't appear, this should probably also be made explicit here.
| for (auto detail : config.setCurrentClientCertDetails()) { | ||
| switch (detail) { | ||
| case Http::ClientCertDetailsType::Subject: | ||
| if (!clientCertDetails.empty()) { |
There was a problem hiding this comment.
This (and below) can be lifted above the switch.
| } | ||
| if (!connection.ssl()->sha256PeerCertificateDigest().empty()) { | ||
| if (!clientCertDetails.empty()) { | ||
| clientCertDetails += ";"; |
There was a problem hiding this comment.
Why not just build a list of clientCertDetails components and then do a https://github.com/lyft/envoy/blob/master/source/common/common/utility.h#L140 with ";"? A join is what's happening logically here.
| } | ||
| clientCertDetails += "Hash=" + connection.ssl()->sha256PeerCertificateDigest(); | ||
| } | ||
| for (auto detail : config.setCurrentClientCertDetails()) { |
source/common/json/config_schemas.cc
Outdated
| "sanitize", | ||
| "sanitize_set", | ||
| "always_forward_only", | ||
| "always_append_forward" |
There was a problem hiding this comment.
This doesn't appear to be referenced anywhere in docs or code?
There was a problem hiding this comment.
Good call. It's not a valid option. I will remove it.
source/common/ssl/connection_impl.cc
Outdated
|
|
||
| bool ConnectionImpl::peerCertificatePresented() { | ||
| bssl::UniquePtr<X509> cert(SSL_get_peer_certificate(ssl_.get())); | ||
| if (cert) { |
There was a problem hiding this comment.
You can't do something like return cert and get an implicit cast?
There was a problem hiding this comment.
Yes I think that's better.
| } else if (config.forwardClientCert() == Http::ForwardClientCertType::SanitizeSet) { | ||
| request_headers.insertForwardedClientCert().value(client_cert_details_str); | ||
| } | ||
| // It should never reach here. |
There was a problem hiding this comment.
Then crash with NOT_REACHED
| } | ||
|
|
||
| // TODO(myidpt): Handle the special characters in By and SAN fields. | ||
| std::vector<std::string> client_cert_details; |
There was a problem hiding this comment.
FYI this is going to be substantially slower than what you had before. This is related to what @lizan mentioned with doing direct append into the header value. In the interest of getting this out and not doing premature optimization, let's leave this simpler code, but please add a TODO that we should look at perf in this case. (Basically anything that is done on every request is something that we need to look at perf for).
There was a problem hiding this comment.
Yeah, the join will be slower. My main aim in suggesting this was to avoid the repeated boilerplate pattern in the code. I think it could also have been made an inline function or something like that as well.
There was a problem hiding this comment.
OK. I will leave a TODO here.
There was a problem hiding this comment.
Just curious, do we have the performance benchmarks?
There was a problem hiding this comment.
Nothing official currently. Official stuff is being worked on in https://github.com/lyft/envoy-perf
source/common/ssl/connection_impl.cc
Outdated
|
|
||
| bool ConnectionImpl::peerCertificatePresented() { | ||
| bssl::UniquePtr<X509> cert(SSL_get_peer_certificate(ssl_.get())); | ||
| return cert.get(); |
htuch
left a comment
There was a problem hiding this comment.
LGTM modulo @mattklein123's remaining comments.
|
@htuch @alyssawilk in the interest of moving this along and unblocking people, can we merge this and the cleanup/refactor for SSL utility code actually be done in #1183 ? |
|
Fine by me
…On Thu, Jun 29, 2017 at 2:47 PM, Matt Klein ***@***.***> wrote:
@htuch <https://github.com/htuch> @alyssawilk
<https://github.com/alyssawilk> in the interest of moving this along and
unblocking people, can we merge this and the cleanup/refactor for SSL
utility code actually be done in #1183
<#1183> ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1087 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ARYFvbOqG7UB1cG9lSm9ixxQgr1gZLgkks5sI_E0gaJpZM4N2vPv>
.
|
|
@myidpt et al, thanks for the work on this - will merge it into our fork and test it out! |
|
Thank you folks for the review :) |
This PR adds a new set of `onEngineRunning` callbacks to the platform (Swift/Kotlin) interfaces. The intent behind these callbacks is to allow platform consumers to observe when Envoy Mobile has finished setting itself up internally. When `build()` is called from an `EnvoyEngineBuilder`, internal library setup is not complete when the allocation returns. Instead, Envoy actually does additional setup before being called back with a "post init" step from the core Envoy layer indicating that it's ready to send requests. In practice, this doesn't have any direct effect on consumers since any requests sent into the library during this timeframe are queued for sending automatically once setup is complete. Based on my localized runs of the library, I noticed 800-1000ms between the start of allocation and the time at which Envoy Mobile has completed its setup. There is room for improvement here which will be looked to in the future. Changes: - Add the new interface to platform layers - Implement the core wiring for this callback and pipe that all the way through JNI / Objective-C up to Kotlin/Swift - Update sample apps to utilize this functionality - Add core unit test to validate the core callback completes Signed-off-by: Michael Rebello <me@michaelrebello.com> Co-authored-by: Alan Chiu <achiu@lyft.com> Signed-off-by: JP Simard <jp@jpsim.com>
This PR adds a new set of `onEngineRunning` callbacks to the platform (Swift/Kotlin) interfaces. The intent behind these callbacks is to allow platform consumers to observe when Envoy Mobile has finished setting itself up internally. When `build()` is called from an `EnvoyEngineBuilder`, internal library setup is not complete when the allocation returns. Instead, Envoy actually does additional setup before being called back with a "post init" step from the core Envoy layer indicating that it's ready to send requests. In practice, this doesn't have any direct effect on consumers since any requests sent into the library during this timeframe are queued for sending automatically once setup is complete. Based on my localized runs of the library, I noticed 800-1000ms between the start of allocation and the time at which Envoy Mobile has completed its setup. There is room for improvement here which will be looked to in the future. Changes: - Add the new interface to platform layers - Implement the core wiring for this callback and pipe that all the way through JNI / Objective-C up to Kotlin/Swift - Update sample apps to utilize this functionality - Add core unit test to validate the core callback completes Signed-off-by: Michael Rebello <me@michaelrebello.com> Co-authored-by: Alan Chiu <achiu@lyft.com> Signed-off-by: JP Simard <jp@jpsim.com>
…ns (#1087) **Description** This commit corrects the missing unmarshal support for token embeddings which otherwise crash with 502. To ensure no other bugs, it backfills embeddings recordings for all main scenarios. To simplify maintenance, this reduces work in test setup when the proxy response is exactly the same as upstream. **Related Issues/PRs (if applicable)** Backfills tests for #1085 Signed-off-by: Adrian Cole <adrian@tetrate.io>
This implements XFCC as discussed in issue #794.
Fixes #794