Revert "Revert "Use heuristic to determine how many events to handle per iter…" by AspirinSJL · Pull Request #14943 · grpc/grpc

AspirinSJL · 2018-04-05T00:07:48Z

To fix a timeout issue reported internally. In that issue, the keepalive ping from the client to the balancer will time out after 10s. The client somehow doesn't receive the ack sent from the balancer.

I have verified this bug can be fixed by #14430, but occurs again after #14688.

Since this bug is blocking the rollout of the reporting team, let's merge this first to unblock that team. Will try to figure out the root cause after this PR.

…per iter…"

grpc-testing · 2018-04-05T00:11:01Z

****************************************************************

libgrpc.so

     VM SIZE                                              FILE SIZE
 ++++++++++++++ GROWING                                ++++++++++++++
  [ = ]       0 [None]                                    +600  +0.0%
  +2.1%    +224 src/core/lib/iomgr/ev_epollex_linux.cc    +224  +2.1%
       +41%    +189 pollable_process_events                   +189   +41%
      +0.7%     +14 pollset_work                               +14  +0.7%
      +4.0%     +13 [Unmapped]                                 +13  +4.0%
       +11%      +8 pollset_init                                +8   +11%

  +0.0%    +224 TOTAL                                     +824  +0.0%


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-04-05T00:20:04Z

[trickle] No significant performance differences

kpayson64

Please add a comment in the code why we are re-introducing this change, because it is not obvious or understood why this is needed for correctness, and likely to be removed otherwise.

grpc-testing · 2018-04-05T01:02:30Z

[microbenchmarks] Performance differences noted:
Benchmark                                                            allocs_per_iteration    atm_add_per_iteration    atm_cas_per_iteration    locks_per_iteration    nows_per_iteration
-------------------------------------------------------------------  ----------------------  -----------------------  -----------------------  ---------------------  --------------------
BM_EmptyCore                                                                                 +28%
BM_PollEmptyPollset                                                                          +66%
BM_SingleThreadPollOneFd                                                                     +66%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/0                                                                                     -16%                   -14%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/1                                                                                     -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/2                                                                                     -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/1                                                                                     -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/2                                                                                     -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/2097152/1                                                                               -5%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/1                                                                                -9%                    -10%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/2                                                                                -6%                    -8%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/1                                                                                 -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/2                                                                                 -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/1                                                                                  -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/2                                                                                  -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/1                                                                                   -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/2                                                                                   -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/1                                                                                    -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/2                                                                                    -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/1                                                                                     -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/2                                                                                     -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/0                                                                                        -16%                   -14%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/1                                                                                        -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/2                                                                                        -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/1                                                                                        -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/2                                                                                        -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/2097152/1                                                                                  -5%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/1                                                                                   -9%                    -10%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/2                                                                                   -6%                    -8%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/1                                                                                    -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/2                                                                                    -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/1                                                                                     -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/2                                                                                     -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/1                                                                                      -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/2                                                                                      -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/1                                                                                       -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/2                                                                                       -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/1                                                                                        -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/2                                                                                        -7%                    -9%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/2097152                                                                                   -13%                   -6%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/0       +107%                   +137%                    +145%                    +151%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/2097152/0                                                                                                          -4%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/0/2097152                                                                                      -11%                   -5%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/134217728/134217728  +121%                   +126%                    +130%                    +121%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/16777216/16777216    +16%                    +8%                      +6%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/2097152/0                                                                                                             -5%

AspirinSJL · 2018-04-05T03:12:37Z

@kpayson64 Added. Thanks!

grpc-testing · 2018-04-05T03:14:14Z

****************************************************************

libgrpc.so

     VM SIZE                                              FILE SIZE
 ++++++++++++++ GROWING                                ++++++++++++++
  [ = ]       0 [None]                                    +600  +0.0%
  +2.1%    +224 src/core/lib/iomgr/ev_epollex_linux.cc    +224  +2.1%
       +41%    +189 pollable_process_events                   +189   +41%
      +0.7%     +14 pollset_work                               +14  +0.7%
      +4.0%     +13 [Unmapped]                                 +13  +4.0%
       +11%      +8 pollset_init                                +8   +11%

  +0.0%    +224 TOTAL                                     +824  +0.0%


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-04-05T03:24:21Z

[trickle] No significant performance differences

grpc-testing · 2018-04-05T04:06:48Z

[microbenchmarks] Performance differences noted:
Benchmark                                                               allocs_per_iteration    atm_add_per_iteration    atm_cas_per_iteration    locks_per_iteration    nows_per_iteration
----------------------------------------------------------------------  ----------------------  -----------------------  -----------------------  ---------------------  --------------------
BM_EmptyCore                                                                                    +28%
BM_PollEmptyPollset                                                                             +66%
BM_PumpStreamServerToClient<TCP>/0                                                              +5%                      +6%
BM_SingleThreadPollOneFd                                                                        +66%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/0                                                                                        -16%                   -14%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/1                                                                                        -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/2                                                                                        -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/1                                                                                        -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/2                                                                                        -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/1                                                                                   -9%                    -10%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/2                                                                                   -6%                    -8%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/1                                                                                    -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/2                                                                                    -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/1                                                                                     -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/2                                                                                     -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/1                                                                                      -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/2                                                                                      -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/1                                                                                       -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/2                                                                                       -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/1                                                                                        -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/2                                                                                        -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/0                                                                                           -16%                   -14%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/1                                                                                           -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/2                                                                                           -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/1                                                                                           -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/2                                                                                           -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/2097152/1                                                                                     -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/1                                                                                      -9%                    -10%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/2                                                                                      -6%                    -8%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/1                                                                                       -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/2                                                                                       -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/1                                                                                        -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/2                                                                                        -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/1                                                                                         -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/2                                                                                         -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/1                                                                                          -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/2                                                                                          -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/1                                                                                           -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/2                                                                                           -7%                    -9%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/0          +122%                   +175%                    +172%                    +130%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/134217728  +83%                    +91%                     +90%                     +76%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/2097152/2097152                                                                                -5%

grpc-testing · 2018-04-05T17:43:01Z

****************************************************************

libgrpc.so

     VM SIZE                                              FILE SIZE
 ++++++++++++++ GROWING                                ++++++++++++++
  [ = ]       0 [None]                                    +600  +0.0%
  +2.1%    +224 src/core/lib/iomgr/ev_epollex_linux.cc    +224  +2.1%
       +41%    +189 pollable_process_events                   +189   +41%
      +0.7%     +14 pollset_work                               +14  +0.7%
      +4.0%     +13 [Unmapped]                                 +13  +4.0%
       +11%      +8 pollset_init                                +8   +11%

  +0.0%    +224 TOTAL                                     +824  +0.0%


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

grpc-testing · 2018-04-05T17:51:12Z

[trickle] No significant performance differences

grpc-testing · 2018-04-05T18:34:28Z

[microbenchmarks] Performance differences noted:
Benchmark                                                               allocs_per_iteration    atm_add_per_iteration    atm_cas_per_iteration    http2_pings_sent_per_iteration    locks_per_iteration    nows_per_iteration
----------------------------------------------------------------------  ----------------------  -----------------------  -----------------------  --------------------------------  ---------------------  --------------------
BM_EmptyCore                                                                                    +28%
BM_PollEmptyPollset                                                                             +66%
BM_SingleThreadPollOneFd                                                                        +66%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/0                                                                                                                          -16%                   -14%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/1                                                                                                                          -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/2                                                                                                                          -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/1                                                                                                                          -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/1/2                                                                                                                          -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/1                                                                                +20%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/1                                                                                                                     -9%                    -10%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/262144/2                                                                                                                     -6%                    -8%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/1                                                                                                                      -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/32768/2                                                                                                                      -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/1                                                                                                                       -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/4096/2                                                                                                                       -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/1                                                                                                                        -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/512/2                                                                                                                        -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/1                                                                                                                         -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/64/2                                                                                                                         -7%                    -9%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/1                                                                                                                          -10%                   -11%
BM_StreamingPingPong<MinTCP, NoOpMutator, NoOpMutator>/8/2                                                                                                                          -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/0                                                                                                                             -16%                   -14%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/1                                                                                                                             -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/0/2                                                                                                                             -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/1                                                                                                                             -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/1/2                                                                                                                             -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/134217728/1                                                                                   +20%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/2097152/1                                                                                                                       -4%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/1                                                                                                                        -9%                    -10%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/262144/2                                                                                                                        -6%                    -8%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/1                                                                                                                         -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/32768/2                                                                                                                         -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/1                                                                                                                          -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/4096/2                                                                                                                          -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/1                                                                                                                           -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/512/2                                                                                                                           -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/1                                                                                                                            -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/64/2                                                                                                                            -7%                    -9%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/1                                                                                                                             -10%                   -11%
BM_StreamingPingPong<TCP, NoOpMutator, NoOpMutator>/8/2                                                                                                                             -7%                    -9%
BM_StreamingPingPongMsgs<MinTCP, NoOpMutator, NoOpMutator>/134217728                                                                              +20%
BM_StreamingPingPongMsgs<TCP, NoOpMutator, NoOpMutator>/134217728                                                                                 +5%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/0/2097152                                                                                                                        -14%                   -5%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/0          +107%                   +114%                    +122%                                                      +129%
BM_UnaryPingPong<MinTCP, NoOpMutator, NoOpMutator>/134217728/134217728  +6%                     +10%                     +9%                                                        +5%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/0/2097152                                                                                                                           -14%                   -7%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/134217728/0             +109%                   +130%                    +129%                                                      +109%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/134217728/134217728     +121%                   +117%                    +119%                                                      +109%
BM_UnaryPingPong<TCP, NoOpMutator, NoOpMutator>/16777216/16777216                               +9%                      +8%

AspirinSJL · 2018-04-06T17:16:06Z

Basic Tests MacOS [opt] (internal CI): [build failure] FAILED: run_tests_python_macos_opt_native #14965
Basic Tests Windows [opt] (internal CI): TIMEOUT: py35_gevent.test.interop._insecure_intraop_test.InsecureIntraopTest #14966
Interop Cloud-to-Cloud Tests (internal CI): TIMEOUT: on many cloud_to_prod and cloud_to_cloud tests #14969
Interop Cloud-to-Prod Tests (internal CI): TIMEOUT: on many cloud_to_prod and cloud_to_cloud tests #14969

Revert "Revert "Use heuristic to determine how many events to handle …

cbfc531

…per iter…"

AspirinSJL requested review from kpayson64 and sreecha April 5, 2018 00:07

kpayson64 suggested changes Apr 5, 2018

View reviewed changes

Add TODO

22c6387

kpayson64 approved these changes Apr 5, 2018

View reviewed changes

AspirinSJL added the kokoro:force-run label Apr 5, 2018

kokoro-team removed the kokoro:force-run label Apr 5, 2018

AspirinSJL merged commit cc4fe1b into master Apr 6, 2018

AspirinSJL deleted the revert-14688-revert_pr branch April 6, 2018 17:17

lock bot locked as resolved and limited conversation to collaborators Jan 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "Revert "Use heuristic to determine how many events to handle per iter…"#14943

Revert "Revert "Use heuristic to determine how many events to handle per iter…"#14943
AspirinSJL merged 2 commits intomasterfrom
revert-14688-revert_pr

AspirinSJL commented Apr 5, 2018 •

edited

Loading

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

kpayson64 left a comment

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

AspirinSJL commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

AspirinSJL commented Apr 6, 2018 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

AspirinSJL commented Apr 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

kpayson64 left a comment

Choose a reason for hiding this comment

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

AspirinSJL commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

grpc-testing commented Apr 5, 2018

Uh oh!

AspirinSJL commented Apr 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AspirinSJL commented Apr 5, 2018 •

edited

Loading

AspirinSJL commented Apr 6, 2018 •

edited

Loading