Skip to content

eds: allow duplicate internal addresses#22420

Closed
stevenctl wants to merge 1 commit intoenvoyproxy:mainfrom
stevenctl:allow-dupe-upstream
Closed

eds: allow duplicate internal addresses#22420
stevenctl wants to merge 1 commit intoenvoyproxy:mainfrom
stevenctl:allow-dupe-upstream

Conversation

@stevenctl
Copy link
Copy Markdown

Commit Message: allow duplicate internal endpoints
Additional Description: endpoints that use internal address should not be subject to deduplication as passthrough metadata is usually what distinguishes them
Risk Level: Low
Testing: TODO unit test
Docs Changes: N/A
Release Notes: should I add one? are there releases including internal address at this point?
Platform Specific Features: N/A

Fixes #22417

@repokitteh-read-only
Copy link
Copy Markdown

Hi @stevenctl, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #22420 was opened by stevenctl.

see: more, trace.

@zuercher
Copy link
Copy Markdown
Member

// An exception is made for internal addresses as passthrough metadata is usually what makes them unique.
const auto address_as_string = address->asString();
if (all_new_hosts.count(address_as_string) > 0) {
if (address->envoyInternalAddress() == nullptr && all_new_hosts.count(address_as_string) > 0) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be a bit improved:

if (address->type() != Type::EnvoyInternal) {
  const auto address_as_string = address->asString();
  if (all_new_hosts.count(...)) {
    return;
  }
  all_new_hosts.emplace(address_as_string);
}

That will avoid counting internal address and unnecessarily creating a string.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as a cpp newb.. wouldn't internaladdr != null already avoid the counting? Just double checking.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but you're still putting internal hosts into a map for no reason.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The emplace call has to happen regardless of address->type(), so seems we should always create the string.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't see this comment ^

Are we certain this map is only used for dedupe? It eventually gets passed here:

[&all_new_hosts, &new_hosts_for_current_priority,
&hosts_with_updated_locality_for_current_priority, &final_hosts,
&max_host_weight](const HostSharedPtr& p) {
// This host has already been added as a new host in the
// new_hosts_for_current_priority. Return false here to make sure that host
// reference with older locality gets cleaned up from the priority.
if (hosts_with_updated_locality_for_current_priority.contains(p->address()->asString())) {
return false;
}
if (all_new_hosts.contains(p->address()->asString()) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it's trying to correlate the priority set and the new hosts map, so we should keep them consistent. I wonder if we should make the map be from a pointer instead. Let me look deeper at the usage.

Copy link
Copy Markdown
Contributor

@kyessenov kyessenov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this is not going to work as expected for addition/removal of internal address hosts with the same address. All the complex logic for diffing host sets relies on HostMap which itself relies on addressAsString. So something is likely to break (either removal not causing drain, or addition not affecting priority set). I don't see how we can easily change to not use addressAsString because hosts are re-created it seems on every EDS update. So what we need is a structural hash hash(internal_address_name, endpoint metadata) to be the address-as-string. I think we could simply append hash of the endpoint metadata to internal addresses? WDYT?

@kyessenov
Copy link
Copy Markdown
Contributor

Another idea: change host address to support "extensions" to internal address identifier. That would be a protobuf change, but we could just add an opaque string as a designator of "host uniqueness" without affecting "network identity".

Adds a "logical port" to internal addresses to help differentiate them
from each other when they have differences in metadata.

Could also be accomplished by appending a hash to the dedupe key.

Signed-off-by: Steven Landow <landow@google.com>
@stevenctl
Copy link
Copy Markdown
Author

2022-08-01T11:08:32.074168Z     debug   envoy filter    original_dst: new connection accepted
2022-08-01T11:08:32.074204Z     debug   envoy filter    workload metadata: new connection accepted
2022-08-01T11:08:32.074238Z     debug   envoy filter    original_dst: new connection accepted
2022-08-01T11:08:32.074325Z     debug   envoy filter    [C18] new tcp proxy session
2022-08-01T11:08:32.074358Z     debug   envoy filter    [C18] Creating connection to cluster spiffe://cluster.local/ns/echo/sa/a_to_http_b.echo.svc.cluster.local_outbound_internal
2022-08-01T11:08:32.074426Z     debug   envoy misc      Allocating TCP conn pool
2022-08-01T11:08:32.074468Z     debug   envoy pool      trying to create new connection
2022-08-01T11:08:32.074481Z     debug   envoy pool      creating a new connection (connecting=0)
2022-08-01T11:08:32.074626Z     debug   envoy connection        [C19] connecting to envoy://outbound_tunnel_lis_spiffe://cluster.local/ns/echo/sa/a:10.244.1.5_18080
2022-08-01T11:08:32.074651Z     debug   envoy io        user namespace handle 0x55ff14f12000 connect to previously closed peer envoy://outbound_tunnel_lis_spiffe://cluster.local/ns/echo/sa/a:10.244.1.5_18080.
2022-08-01T11:08:32.074664Z     debug   envoy connection        [C19] immediate connect error: Invalid argument
2022-08-01T11:08:32.074696Z     debug   envoy conn_handler      [C18] new connection from 10.244.1.4:42088
2022-08-01T11:08:32.074740Z     debug   envoy connection        [C19] raising immediate error
2022-08-01T11:08:32.074747Z     debug   envoy connection        [C19] closing socket: 0
2022-08-01T11:08:32.074758Z     debug   envoy pool      [C19] client disconnected, failure reason: immediate connect error: Invalid argument
2022-08-01T11:08:32.074777Z     debug   envoy filter    [C18] Creating connection to cluster spiffe://cluster.local/ns/echo/sa/a_to_http_b.echo.svc.cluster.local_outbound_internal
2022-08-01T11:08:32.074792Z     debug   envoy connection        [C18] closing data_to_write=0 type=1
2022-08-01T11:08:32.074802Z     debug   envoy connection        [C18] closing socket: 1
2022-08-01T11:08:32.074859Z     debug   envoy conn_handler      [C18] adding to cleanup list
2022-08-01T11:08:32.074899Z     debug   envoy pool      invoking idle callbacks - is_draining_for_deletion_=false
[2022-08-01T11:08:32.074Z] "- - -" 0 UF,URX - - "immediate_connect_error:_Invalid_argument" 0 0 0 - "-" "-" "-" "-" "envoy://outbound_tunnel_lis_spiffe://cluster.local/ns/echo/sa/a:10.244.1.5_18080" spiffe://cluster.local/ns/echo/sa/a_to_http_b.echo.svc.cluster.local_outbound_internal - 10.96.8.66:80 10.244.1.4:42088 - - outbound capture listener
[2022-08-01T11:08:32.074Z] "- - -" 0 UF,URX - - "immediate_connect_error:_Invalid_argument" 0 0 0 - "-" "-" "-" "-" "envoy://outbound_tunnel_lis_spiffe://cluster.local/ns/echo/sa/a:10.244.1.5_18080" spiffe://cluster.local/ns/echo/sa/a_to_http_b.echo.svc.cluster.local_outbound_internal - 10.96.8.66:80 10.244.1.4:42088 - - capture outbound (no pep)

I think changing asString on address has other implications.

@repokitteh-read-only
Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @htuch
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #22420 was synchronize by stevenctl.

see: more, trace.

@kyessenov
Copy link
Copy Markdown
Contributor

I can't tell what the issue is from the log. Can you reproduce it in a test with a standalone envoy?

@kyessenov
Copy link
Copy Markdown
Contributor

There's a difference between downstream and upstream. You don't want to have logical port on the downstream, so we need to strip the port there.

@kyessenov
Copy link
Copy Markdown
Contributor

I think we need to separate listener binding from address like in https://github.com/envoyproxy/envoy/pull/22274/files

* For IPv4 addresses: "1.2.3.4:80"
* For IPv6 addresses: "[1234:5678::9]:443"
* For pipe addresses: "/foo"
* For internal addresses: "serverListenerName-<hash of metadata>"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the port directly available as a property of the Address?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have another API PR that also changes how binding works #22274, because we don't want to bind to individual "ports". It's merely an endpoint identifier for diff tracking.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kyessenov will let you guide the review for API here and let me know when ready for another pass. It seems #22274 is solving the broader problem here.

@stevenctl
Copy link
Copy Markdown
Author

Seems like #22274 is the fix

@stevenctl stevenctl closed this Aug 4, 2022
kyessenov added a commit that referenced this pull request Aug 10, 2022
…er binding (#22274)

Signed-off-by: Kuat Yessenov <kuat@google.com>

Commit Message: Expose internal listener functionality. Marks the fields as implemented. Adds the following functional changes prior to publishing:
* do not use the `address` field per #21132, and instead use the listener `name` field as originally expected.
* add an `endpoint_id` field to upstream address to fix #22420; it's not possible to have multiple internal addresses to the same listener in a single pool without some extra identifier beyond the listener name, because pools identify addresses using `asString()`.
* fixes #20665 by using stat prefix `listener.envoy_internal_<listener_name>`.

Adds a few sample configs (and fixes #20335):
* minimal two chained `tcp_proxy` listeners;
* encap HTTP in HTTP CONNECT;
* decap HTTP in HTTP CONNECT;

Risk Level: low, hidden prior
Testing: yes
Docs Changes: yes
Release Notes: yes
lizan pushed a commit to envoyproxy/data-plane-api that referenced this pull request Aug 10, 2022
…er binding (#22274)

Signed-off-by: Kuat Yessenov <kuat@google.com>

Commit Message: Expose internal listener functionality. Marks the fields as implemented. Adds the following functional changes prior to publishing:
* do not use the `address` field per envoyproxy/envoy#21132, and instead use the listener `name` field as originally expected.
* add an `endpoint_id` field to upstream address to fix envoyproxy/envoy#22420; it's not possible to have multiple internal addresses to the same listener in a single pool without some extra identifier beyond the listener name, because pools identify addresses using `asString()`.
* fixes envoyproxy/envoy#20665 by using stat prefix `listener.envoy_internal_<listener_name>`.

Adds a few sample configs (and fixes envoyproxy/envoy#20335):
* minimal two chained `tcp_proxy` listeners;
* encap HTTP in HTTP CONNECT;
* decap HTTP in HTTP CONNECT;

Risk Level: low, hidden prior
Testing: yes
Docs Changes: yes
Release Notes: yes

Mirrored from https://github.com/envoyproxy/envoy @ 02489bbf8990faddb6be1f2e0f22851b64c3fc85
oschaaf pushed a commit to maistra/envoy that referenced this pull request Oct 26, 2022
…er binding (#22274)

Signed-off-by: Kuat Yessenov <kuat@google.com>

Commit Message: Expose internal listener functionality. Marks the fields as implemented. Adds the following functional changes prior to publishing:
* do not use the `address` field per envoyproxy/envoy#21132, and instead use the listener `name` field as originally expected.
* add an `endpoint_id` field to upstream address to fix envoyproxy/envoy#22420; it's not possible to have multiple internal addresses to the same listener in a single pool without some extra identifier beyond the listener name, because pools identify addresses using `asString()`.
* fixes envoyproxy/envoy#20665 by using stat prefix `listener.envoy_internal_<listener_name>`.

Adds a few sample configs (and fixes envoyproxy/envoy#20335):
* minimal two chained `tcp_proxy` listeners;
* encap HTTP in HTTP CONNECT;
* decap HTTP in HTTP CONNECT;

Risk Level: low, hidden prior
Testing: yes
Docs Changes: yes
Release Notes: yes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EDS deduping endpoints with same address but different metadata

4 participants