Updating LB docs to the new LB logic by alyssawilk · Pull Request #359 · envoyproxy/data-plane-api

alyssawilk · 2017-12-22T01:57:21Z

No description provided.

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk · 2017-12-22T01:58:37Z

This mostly goes with the upcoming PR depending on envoyproxy/envoy#2244 landing

Will close for now - just wanted the docs PR for 2244

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk · 2018-01-02T20:31:05Z

huh. local check_format complains about spaces in my change.
local fix_format changes 5 files I didn't touch and doesn't update the lb doc.

I'll poke around.

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

mattklein123 · 2018-01-02T21:14:08Z

@alyssawilk I noticed the same thing and fixed here: #381. I'm confused how the stuff got through in the first place.

alyssawilk · 2018-01-02T21:17:42Z

#382 should forward-fix that.

…

On Tue, Jan 2, 2018 at 4:14 PM, Matt Klein ***@***.***> wrote: @alyssawilk <https://github.com/alyssawilk> I noticed the same thing and fixed here: #381 <#381>. I'm confused how the stuff got through in the first place. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#359 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARYFvdq9okApshb2PIhXTcL7_Y1I_sM8ks5tGpwhgaJpZM4RKj5U> .

mattklein123

nice, this is super cool. Some small nits.

mattklein123 · 2018-01-03T00:23:39Z

docs/root/intro/arch_overview/load_balancing.rst

-another are fairly simplistic: a given priority level will be used until it has zero healthy hosts,
-at which point it will hard fail to the next highest priority level.
+level. For each EDS :ref:`LocalityLbEndpoints<envoy_api_msg_LocalityLbEndpoints>` an optional
+priority may also be specifie. When endpoints at the highest priority level (P=0) are healthy, all


typo "specifie"

mattklein123 · 2018-01-03T00:24:08Z

docs/root/intro/arch_overview/load_balancing.rst

-at which point it will hard fail to the next highest priority level.
+level. For each EDS :ref:`LocalityLbEndpoints<envoy_api_msg_LocalityLbEndpoints>` an optional
+priority may also be specifie. When endpoints at the highest priority level (P=0) are healthy, all
+traffic will land on endpoints in that priority leve. As endpoints for the highest priority level


typo: "leve" - do you mind running through spell checker?

Hahaha. I did spell check, but then I ran a buggy version of check_format which replaced "[^.]. " with ". " and so removed the last letter of a bunch of words. Shame on me for not checking diffs before and after.

mattklein123 · 2018-01-03T00:24:38Z

docs/root/intro/arch_overview/load_balancing.rst

+
+Currently, it is assumed that each priority level is over-provisioned by a (hard-coded) factor of
+1.4. So if 80% of the endpoints are healthy, the priority level is still considered healthy because
+80*1.4 > 10. As the number of healthy endpoints dips below 72%, the health of the priority level


htuch

I think this is great at conveying intuition about what is happening. Would be good to also add in the underlying formula for computing how the load is spilled for those .interested.

htuch · 2018-01-03T11:11:30Z

docs/root/intro/arch_overview/load_balancing.rst

+Currently, it is assumed that each priority level is over-provisioned by a (hard-coded) factor of
+1.4. So if 80% of the endpoints are healthy, the priority level is still considered healthy because
+80*1.4 > 10. As the number of healthy endpoints dips below 72%, the health of the priority level
+goes below 100, and any residual traffic will flow to the next priority level.


What is meant by "residual" here?

htuch · 2018-01-03T11:11:48Z

docs/root/intro/arch_overview/load_balancing.rst

+Assume a simple set-up with 2 priority levels, P=1 100% healthy.
+
+----------------------------+---------------------------+----------------------------+
+| Percent healthy endpoints  | Percent of traffic to P=0 |  Percent of traffic to P=1 |


Percent healthy endpoints for P=0?

htuch · 2018-01-03T11:13:58Z

docs/root/intro/arch_overview/load_balancing.rst

+----------------------------+---------------------------+----------------------------+
+| 71                         | 99                        |   1                        |
+----------------------------+---------------------------+----------------------------+
+| 50                         | 70                        |   30                       |


I think it would be helpful to see the expression that is used to compute the residue here.

I used the expressions below to try and compute the 50% health case, I get P_0 traffic at 33% and P_1 traffic at 67%. Maybe I did math wrong, can you double check?

so percent healthy (healthy_PX_backends / total_PX_backends) is 50%
health(P_X) = (140 * .5) = 70
P_0 = min(70, 100) = 70
load to P_1= 100 - (70) = 30

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk · 2018-01-03T14:12:39Z

docs/root/intro/arch_overview/load_balancing.rst

+percent load to P=0 == min(100, health(P=0) * 100 / total_health)
+health(P=X) == (1.4 * healthy_PX_backends / total_PX_backends)
+total_health == min(100, Σ(health(P=0)...health(P=x));
+percent load to P=X == 100 - Σ(percent_load(P0)..percent_load(Px-1))


@htuch this what you were looking for?

Suggestions for improvement welcome.

Yes, this is great! Please verify that when the RST turns to HTML, this is all readable.

htuch · 2018-01-03T14:56:45Z

docs/root/intro/arch_overview/load_balancing.rst

+To sum this up in pseudo algorithms:
+percent load to P=0 == min(100, health(P=0) * 100 / total_health)
+health(P=X) == (1.4 * healthy_PX_backends / total_PX_backends)
+total_health == min(100, Σ(health(P=0)...health(P=x));


Nit: If using fancy Unicode symbols, one minor improvement might be to use P_0 and P_1 subscripts instead of P=0, P=1. Totally an optional readability improvement.

htuch · 2018-01-03T15:00:32Z

docs/root/intro/arch_overview/load_balancing.rst

+-------------+-------------+------------+----------------+----------------+----------------+
+
+To sum this up in pseudo algorithms:
+percent load to P=0 == min(100, health(P=0) * 100 / total_health)


Nit: prefer = to == here, similar to normal expression or function definitions in math.

htuch · 2018-01-03T15:01:57Z

docs/root/intro/arch_overview/load_balancing.rst

+
+To sum this up in pseudo algorithms:
+percent load to P=0 == min(100, health(P=0) * 100 / total_health)
+health(P=X) == (1.4 * healthy_PX_backends / total_PX_backends)


This is a scaled ratio here, but in total_health below the min expression compares to a percentage (100) rather than 1.0. Is this a bug?

htuch · 2018-01-03T15:06:25Z

docs/root/intro/arch_overview/load_balancing.rst

+----------------------------+---------------------------+----------------------------+
+| 71                         | 99                        |   1                        |
+----------------------------+---------------------------+----------------------------+
+| 50                         | 70                        |   30                       |


I used the expressions below to try and compute the 50% health case, I get P_0 traffic at 33% and P_1 traffic at 67%. Maybe I did math wrong, can you double check?

htuch · 2018-01-03T15:06:50Z

docs/root/intro/arch_overview/load_balancing.rst

+percent load to P=0 == min(100, health(P=0) * 100 / total_health)
+health(P=X) == (1.4 * healthy_PX_backends / total_PX_backends)
+total_health == min(100, Σ(health(P=0)...health(P=x));
+percent load to P=X == 100 - Σ(percent_load(P0)..percent_load(Px-1))


Yes, this is great! Please verify that when the RST turns to HTML, this is all readable.

htuch · 2018-01-03T17:07:57Z

Ah, your math is right, I was computing total_health as 140, ignoring the min(100 bit.

htuch · 2018-01-03T17:09:39Z

docs/root/intro/arch_overview/load_balancing.rst

+P=2 would only receive traffic if the combined health of P=0 + P=1 was less than 100.
+
+-----------------------+-----------------------+-----------------------+----------------+----------------+----------------+
+| P=0 healthy endpoints | P=1 healthy endpoints | P=2 healthy endpoints | Traffic to P=0 | Traffic to P=1 | Traffic to P=3 |


Should last column be Traffic to P==2?

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

htuch · 2018-01-03T20:06:54Z

docs/root/intro/arch_overview/load_balancing.rst

+::
+
+  load to P_0 = min(100, health(P_0) * 100 / total_health)
+  health(P_X) = 100, (140 * healthy_PX_backends / total_PX_backends)


Check docs.

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

Changes Envoy load balancing across priority levels from a hard failover to trickling data based on the health percentage of each priority level. Risk Level: Medium Testing: Added thorough unit testing for the lb failover code, as well as fixing up ring hash failover testing and adding ring-hash specific tests. Docs Changes: envoyproxy/data-plane-api#359 Release Notes: n/a: falls under existing note [Optional Fixes #Issue] Fixes #1929

Updating LB docs to new LB logic

adb54d9

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk closed this Dec 22, 2017

alyssawilk mentioned this pull request Dec 22, 2017

load balancing: Disabling zone aware routing for non-zero priority levels. envoyproxy/envoy#2244

Merged

Merge branch 'master' into real_lb

f38ba8e

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk reopened this Jan 2, 2018

alyssawilk mentioned this pull request Jan 2, 2018

lb: gentle failover across load balancing priority levels. envoyproxy/envoy#2290

Merged

Fix format

fd8c213

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

mattklein123 reviewed Jan 3, 2018

View reviewed changes

htuch reviewed Jan 3, 2018

View reviewed changes

alyssawilk added 3 commits January 3, 2018 09:09

Wherein alyssar@ fails at LaTeX in .rst

153edcb

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

Merge branch 'master' into real_lb

7b4aa79

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

fix format with fixed fix format

e3f7a76

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk commented Jan 3, 2018

View reviewed changes

htuch reviewed Jan 3, 2018

View reviewed changes

alyssawilk force-pushed the real_lb branch 12 times, most recently from 3984fd4 to 3621602 Compare January 3, 2018 17:06

alyssawilk force-pushed the real_lb branch 2 times, most recently from 65037b2 to 0dee4bf Compare January 3, 2018 17:08

htuch reviewed Jan 3, 2018

View reviewed changes

alyssawilk force-pushed the real_lb branch from 0dee4bf to f8e6f98 Compare January 3, 2018 17:10

reviewer comments

df0a33d

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk force-pushed the real_lb branch from f8e6f98 to df0a33d Compare January 3, 2018 17:11

3!=2

4f91853

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

htuch reviewed Jan 3, 2018

View reviewed changes

fixing tyop

815b07c

Signed-off-by: Alyssa Wilk <alyssar@chromium.org>

alyssawilk force-pushed the real_lb branch from 1d10d9f to 815b07c Compare January 3, 2018 20:16

htuch approved these changes Jan 3, 2018

View reviewed changes

htuch merged commit a1f2fe9 into envoyproxy:master Jan 3, 2018

Conversation

alyssawilk commented Dec 22, 2017

Uh oh!

alyssawilk commented Dec 22, 2017

Uh oh!

alyssawilk commented Jan 2, 2018

Uh oh!

mattklein123 commented Jan 2, 2018

Uh oh!

alyssawilk commented Jan 2, 2018 via email

Uh oh!

mattklein123 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch commented Jan 3, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants