Tap and Top stops streaming intermittently #2891
Description
Bug Report
What is the issue?
On edge-19.5.4, the linkerd tap and linkerd top commands seem to stop streaming data after a minute or two, despite continuous traffic in the emojivoto application.
How can it be reproduced?
✗ linkerd version
Client version: edge-19.5.4
Server version: edge-19.5.4
# install emojivoto
✗ curl https://run.linkerd.io/emojivoto.yml | linkerd inject - | k apply -f -
# notice that after a minute or two, the data streaming stops
✗ linkerd -n emojivoto top deploy
# meanwhile, the voting service continues to send out traffic
✗ k -n emojivoto logs -f voting-7c8c96b45c-rlsj4 voting-svc
Logs, error output, etc
✗ linkerd logs --control-plane-component tap
+ linkerd linkerd-tap-7779f7d978-48prk › tap
+ linkerd linkerd-tap-7779f7d978-48prk › linkerd-proxy
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:34:24Z" level=info msg="running version edge-19.5.4"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:34:24Z" level=info msg="waiting for caches to sync"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:34:24Z" level=info msg="caches synced"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:34:24Z" level=info msg="starting admin server on :9998"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:34:24Z" level=info msg="starting gRPC server on :8088"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:36:03Z" level=info msg="Tapping 4 pods for target: {Namespace:emojivoto Type:deployment Name: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:36:03Z" level=info msg="Establishing tap on 172.17.0.11:4190"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:36:03Z" level=info msg="Establishing tap on 172.17.0.13:4190"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:36:03Z" level=info msg="Establishing tap on 172.17.0.12:4190"
linkerd linkerd-tap-7779f7d978-48prk tap time="2019-06-04T17:36:03Z" level=info msg="Establishing tap on 172.17.0.10:4190"