Skip to content

Enable server side control over maximum connection age#10867

Closed
elevran wants to merge 2 commits intoistio:masterfrom
elevran:pilot_envoy_lb_7878
Closed

Enable server side control over maximum connection age#10867
elevran wants to merge 2 commits intoistio:masterfrom
elevran:pilot_envoy_lb_7878

Conversation

@elevran
Copy link
Copy Markdown
Contributor

@elevran elevran commented Jan 10, 2019

Allow configuration of maximum connection age on the gRPC server.

Once a connection's lifetime reaches the maximum age, the server gracefully closes it (i.e., the client is informed and allowed grace time to drain messages). After a grace period, the connection is forcibly closed. Actual connection age is randomized by the gRPC server implementation to avoid connection thundering herds.
By controlling connection lifetime, clients are forced to periodically reconnect, achieving a better load balancing spread across servers - quicker. Since connections are long-lived, server instances that are available earlier in the endpoints pool will get (and retain) a larger portion of the client connections, leaving connection distribution unbalanced.

By default, the existing behavior is maintained (i.e., maximum connection age and grace are set to "infinity"). Users who wish to adopt the new behavior can do so using the added command line parameters.

Documentation changes are still needed and will be done in a separate PR.

Fixes #7878

@elevran elevran added this to the 1.1 milestone Jan 10, 2019
@elevran elevran requested a review from hzxuzhonghu January 10, 2019 13:31
@istio-testing
Copy link
Copy Markdown
Collaborator

@elevran: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
prow/istio-integ-k8s-tests.sh 0a91113 link /test istio-integ-k8s-tests
Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liamawhite
Copy link
Copy Markdown
Member

/lgtm

cc @duderino @ZackButcher

@istio-testing
Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: elevran, liamawhite
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: nmittler

If they are not already assigned, you can assign the PR to them by writing /assign @nmittler in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@linsun
Copy link
Copy Markdown
Member

linsun commented Jan 10, 2019

@elevran could you rebase to release-1.1 branch?

@elevran
Copy link
Copy Markdown
Contributor Author

elevran commented Jan 10, 2019

PR #10870 resubmitted off of release-1.1 branch instead of this one which was off of master (too many conflicts...)

@elevran elevran closed this Jan 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1/4 istio-pilot instances getting basically no load

5 participants