Outbound HTTP call failing intermittently #11055
Description
What is the issue?
Outbound HTTP calls to application fails with connection closed error intermittently. Source application from which HTTP calls failing is a scheduler application which makes regular calls to other application based on schedule. After the failure, the next schedule works fine. Please see the logs below from linkerd-proxy
How can it be reproduced?
Linkerd Version: 2.13.4
Logs, error output, etc
[129664.685446s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:39860
[129664.691331s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:39872
[129664.691557s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:39878
[129664.692554s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:39894
[129664.695277s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:39900
[129784.342535s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:42724
[129784.342604s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:42738
[129784.342642s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:42746
[129784.342682s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=service unavailable client.addr=10.50.29.85:42760
output of linkerd check -o short
linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2023-06-25T04:48:41Z
see https://linkerd.io/2.13/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
certificate will expire on 2023-06-24T04:48:45Z
see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
certificate will expire on 2023-06-24T04:48:46Z
see https://linkerd.io/2.13/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
‼ policy-validator cert is valid for at least 60 days
certificate will expire on 2023-06-24T04:48:45Z
see https://linkerd.io/2.13/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints
linkerd-version
---------------
‼ cli is up-to-date
is running version 2.13.4 but the latest stable version is 2.13.5
see https://linkerd.io/2.13/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 2.13.4 but the latest stable version is 2.13.5
see https://linkerd.io/2.13/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5b59946658-pm2wk (stable-2.13.4)
* linkerd-identity-54dfb7648b-hhp2r (stable-2.13.4)
* linkerd-proxy-injector-5fc4795598-4h68n (stable-2.13.4)
see https://linkerd.io/2.13/checks/#l5d-cp-proxy-version for hints
linkerd-viz
-----------
‼ tap API server cert is valid for at least 60 days
certificate will expire on 2023-06-24T04:22:23Z
see https://linkerd.io/2.13/checks/#l5d-tap-cert-not-expiring-soon for hints
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* metrics-api-d6665b56-7hs59 (stable-2.13.4)
* tap-6d45c9cd5c-wj2ft (stable-2.13.4)
* tap-injector-6c95fb4f97-q945w (stable-2.13.4)
* web-7c9756b6b-4bcvw (stable-2.13.4)
see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cp-version for hints
Status check results are √
Environment
- Kubernetes Version: 1.23
- Cluster Environment: EKS
- OS: Linux
- Linkerd Version: 2.13.4
Possible solution
No response
Additional context
No response
Would you like to work on fixing this bug?
None