Inbound connections stop and CLOSE_WAIT skyrockets

## Detailed description of the problem



We're seeing a weird behavior that's been consistent over multiple versions of haproxy (2.1/2.2 and now 2.3) and I'm at the point where I suspect we might be hitting a strange bug.

We have a pool of haproxies that accept traffic from ad exchanges (lots of POSTs) and send those to backend servers to process.  There's no humans involved, and normally traffic is very consistent.

Every so often we'll go from 20k established connections to a much higher number of connections, hundreds of thousands of CLOSE_WAIT connections, and then we'll run out of TCP memory on the node.

I've been working on tuning and trying to stop this from happening, and now I have a new weird scenario to run by you.

1: Server is operating normally, 50k established connections, plenty of traffic.
2: Suddenly new connections to :80 start to fail, dns healthchecks fail.
3: Existing connections continue to send traffic, which is sent to the backend, as normal.
4: Trying to curl from the box itself to http://localhost/ fails

```
10-4-18-176:~$ curl http://localhost/healthz
curl: (28) Failed to connect to localhost port 80: Connection timed out
```

Connections right now:

```
10-4-18-176:~$ ss | grep ESTAB | wc -l
37714
10-4-18-176:~$ ss | grep CLOSE-WAIT | wc -l
348901
```

These connections in CLOSE-WAIT never close.  They just sit there until we give up and stop/start haproxy.

I managed to capture `show sess` and `show fd` but I'd like to share those privately if at all possible.

## Expected behavior



One our healthcheck fails and incoming connections stop, I'd expect haproxy to close these CLOSE-WAITs and recover to the point it can start taking new :80 connections.  This never happens however!

## Steps to reproduce the behavior



I'm not sure at all I could give you reproduction steps.  We can operate for hours/days in a steady state before suddenly hitting some kind of wall and going from 0 CLOSE-WAIT to hundreds of thousands and instantly broken.

## Do you have any idea what may have caused this?

Not a clue, I can share metrics, configs, etc, but it happens suddenly and instantly.

## Do you have an idea how to solve the issue?

Afraid not, all I've done is tune things and try to.

## What is your configuration?



Some quick detail first - we pin irq's to the first 8 cores and then haproxy to the other 8:

```
global
  log /dev/log local0 warning alert
  log /dev/log local1 crit alert
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  maxconn 2000000
  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL). This list is from:
  #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL). This list is from:
  #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  cpu-map auto:1/1-8 8-15

resolvers localdns
  nameserver local1 127.0.0.1:8600
  # enable EDNS0 with larger accepted size to allow for large list of IPs
  accepted_payload_size 8192
  hold nx 40000s
  hold refused 40000s
  hold timeout 40000s
  hold other 40000s
  # do a DNS resolve every 5s both after successfull and failed resolves
  timeout resolve 5s
  timeout retry 5s
  # we keep trying for 300s total before DNS resolution fails
  resolve_retries 60

defaults
  log global
  mode http
  option httplog
  option dontlognull
  timeout connect 5000
  timeout client 120s
  timeout server 120s
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http
  timeout queue 1s
  option splice-request
  option splice-response

frontend admin
  bind *:8100
  mode http
  maxconn 400
  option httplog
  option dontlognull
  log global
  stats uri /admin/stats

frontend stats
  bind *:8404
  mode http
  option http-use-htx
  http-request use-service prometheus-exporter if { path /metrics }
  stats enable
  stats uri /stats
  stats refresh 10s

rontend lb
  bind *:80 process 1/1
  bind *:80 process 1/2
  bind *:80 process 1/3
  bind *:80 process 1/4
  bind *:80 process 1/5
  bind *:80 process 1/6
  bind *:80 process 1/7
  bind *:80 process 1/8
  bind *:443 process 1/1 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/2 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/3 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/4 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/5 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/6 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/7 ssl crt /etc/haproxy_ssl/
  bind *:443 process 1/8 ssl crt /etc/haproxy_ssl/
  mode http
  default_backend fes
  maxconn 60000
  rate-limit sessions 12000
  timeout client-fin 1s
  timeout http-request 120s

backend fes
  mode http
  balance roundrobin
  option httpchk GET /healthz
  timeout server 120s
  timeout connect 10s
  http-reuse aggressive
  retries 0
  retry-on none
  http-request disable-l7-retry
  server-template fes- 400 fes.service.consul:80 check resolvers localdns resolve-prefer ipv4 inter 5s fastinter 1s init-addr none maxconn 360
```

## Output of `haproxy -vv` and `uname -a`



```
A-Proxy version 2.3.0-1ppa1~focal 2020/11/12 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2022.
Known bugs: http://www.haproxy.org/bugs/bugs-2.3.0.html
Running on: Linux 5.4.0-1029-aws #30-Ubuntu SMP Tue Oct 20 10:08:09 UTC 2020 aarch64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-nVVZGf/haproxy-2.3.0=. -fstack-protector-s
trong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-
after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unus
ed-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -
Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +TH
READ -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LI
BCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO
 +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate")
, gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [COMP] compression
        [TRACE] trace
```

This is on ARM, but we see the same issue on regular intel instances:

```
Linux 10-4-18-176 5.4.0-1029-aws #30-Ubuntu SMP Tue Oct 20 10:08:09 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
```

## If HAProxy crashed: Last outputs and backtraces



```
(paste your output here)
```

## Additional information (if helpful)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inbound connections stop and CLOSE_WAIT skyrockets #958

Detailed description of the problem

Expected behavior

Steps to reproduce the behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of `haproxy -vv` and `uname -a`

If HAProxy crashed: Last outputs and backtraces

Additional information (if helpful)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Inbound connections stop and CLOSE_WAIT skyrockets #958

Description

Detailed description of the problem

Expected behavior

Steps to reproduce the behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of haproxy -vv and uname -a

If HAProxy crashed: Last outputs and backtraces

Additional information (if helpful)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `haproxy -vv` and `uname -a`