Docker input plugin: "too many open files" error

### Relevant telegraf.conf:
```toml
[[inputs.docker]]
container_names = []
docker_label_exclude = []
docker_label_include = []
endpoint = "unix:///var/run/docker.sock"
perdevice = true
timeout = "5s"
total = false
```

### System info:
```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

$ telegraf --version
Telegraf 1.18.1 (git: HEAD d3ac9a3f)

$ docker version
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:06 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
```

### Steps to reproduce:
Start telegraf on Ubuntu box with Docker and provided config and wait for "too many open files" error (typically 1-2 days)

### Expected behavior:
No errors, telegraf should work as usual for long period of time

### Actual behavior:
```
$ cat /var/log/telegraf/telegraf.log
2021-05-02T02:01:04Z E! [inputs.docker] Error in plugin: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.21/info": dial unix /var/run/docker.sock: socket: too many open files
2021-05-02T02:01:04Z E! [inputs.docker] Error in plugin: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.21/containers/json?filters=%7B%22status%22%3A%5B%22running%22%5D%7D&limit=0": dial unix /var/run/docker.sock: socket: too many open files
```

### Additional info:
That is connections after few days running:
```
$ lsof +E -aUc telegraf
telegraf 23489 telegraf    1u  unix 0xffff9237d36cc800      0t0 1081192478 type=STREAM ->INO=1081191776 396,systemd-j,17u 1,systemd,18u
telegraf 23489 telegraf    2u  unix 0xffff9237d36cc800      0t0 1081192478 type=STREAM ->INO=1081191776 396,systemd-j,17u 1,systemd,18u
telegraf 23489 telegraf    6u  unix 0xffff9237f425d800      0t0 1081191802 type=DGRAM ->INO=13847 396,systemd-j,4u 1,systemd,127u
telegraf 23489 telegraf   27u  unix 0xffff9236c9b86400      0t0 1081315356 type=STREAM ->INO=1081314394 356,dockerd,27u
telegraf 23489 telegraf   28u  unix 0xffff9236c10eb400      0t0 1081406958 type=STREAM ->INO=1081406959 356,dockerd,46u
telegraf 23489 telegraf   29u  unix 0xffff9236c10e9400      0t0 1081439976 type=STREAM ->INO=1081439977 356,dockerd,47u
telegraf 23489 telegraf   30u  unix 0xffff92375b6aa000      0t0 1081626480 type=STREAM ->INO=1081626481 356,dockerd,60u
telegraf 23489 telegraf   31u  unix 0xffff9236c10e8400      0t0 1081343997 type=STREAM ->INO=1081345479 356,dockerd,41u
telegraf 23489 telegraf   32u  unix 0xffff9236c9b86c00      0t0 1081221327 type=STREAM ->INO=1081220672 356,dockerd,21u
telegraf 23489 telegraf   33u  unix 0xffff9236d3755c00      0t0 1081440323 type=STREAM ->INO=1081440324 356,dockerd,48u
telegraf 23489 telegraf   34u  unix 0xffff9236c2c44400      0t0 1081960086 type=STREAM ->INO=1081960805 356,dockerd,82u
telegraf 23489 telegraf   35u  unix 0xffff9236c10eb000      0t0 1081501764 type=STREAM ->INO=1081501155 356,dockerd,52u
telegraf 23489 telegraf   36u  unix 0xffff9236c10e8c00      0t0 1081375099 type=STREAM ->INO=1081376324 356,dockerd,43u
telegraf 23489 telegraf   37u  unix 0xffff9236c9b85000      0t0 1081283099 type=STREAM ->INO=1081283973 356,dockerd,26u
telegraf 23489 telegraf   38u  unix 0xffff9236d3754c00      0t0 1081252550 type=STREAM ->INO=1081253203 356,dockerd,24u
telegraf 23489 telegraf   39u  unix 0xffff9236c9b85800      0t0 1081193191 type=STREAM ->INO=1081193192 356,dockerd,20u
telegraf 23489 telegraf   40u  unix 0xffff9236c9b86800      0t0 1081192746 type=STREAM ->INO=1081192081 356,dockerd,19u
telegraf 23489 telegraf   41u  unix 0xffff9236ddbe2800      0t0 1081562106 type=STREAM ->INO=1081562107 356,dockerd,56u
telegraf 23489 telegraf   42u  unix 0xffff9236c9b86000      0t0 1081253173 type=STREAM ->INO=1081253174 356,dockerd,23u
telegraf 23489 telegraf   43u  unix 0xffff9236d8c9a400      0t0 1082454244 type=STREAM ->INO=1082454245 356,dockerd,113u
telegraf 23489 telegraf   44u  unix 0xffff9236ddbe1800      0t0 1081595248 type=STREAM ->INO=1081595249 356,dockerd,57u
telegraf 23489 telegraf   45u  unix 0xffff9236c9b87800      0t0 1081221364 type=STREAM ->INO=1081220814 356,dockerd,22u
telegraf 23489 telegraf   46u  unix 0xffff9236d3757c00      0t0 1081283922 type=STREAM ->INO=1081283069 356,dockerd,25u
telegraf 23489 telegraf   47u  unix 0xffff9236c9b84800      0t0 1081343966 type=STREAM ->INO=1081343967 356,dockerd,34u
telegraf 23489 telegraf   48u  unix 0xffff9236d3757400      0t0 1081314450 type=STREAM ->INO=1081315378 356,dockerd,28u
telegraf 23489 telegraf   49u  unix 0xffff9236d3754000      0t0 1081471233 type=STREAM ->INO=1081471234 356,dockerd,49u
telegraf 23489 telegraf   50u  unix 0xffff9236c10ebc00      0t0 1081406932 type=STREAM ->INO=1081408295 356,dockerd,45u
telegraf 23489 telegraf   51u  unix 0xffff9236d3ffec00      0t0 1081869655 type=STREAM ->INO=1081869656 356,dockerd,76u
telegraf 23489 telegraf   52u  unix 0xffff92375b6aa800      0t0 1081686490 type=STREAM ->INO=1081685723 356,dockerd,64u
telegraf 23489 telegraf   53u  unix 0xffff9236ddbe2c00      0t0 1081595272 type=STREAM ->INO=1081596405 356,dockerd,58u
telegraf 23489 telegraf   54u  unix 0xffff9236c10e9c00      0t0 1081470582 type=STREAM ->INO=1081470583 356,dockerd,50u
telegraf 23489 telegraf   55u  unix 0xffff9236d3755400      0t0 1081376285 type=STREAM ->INO=1081376286 356,dockerd,42u
telegraf 23489 telegraf   56u  unix 0xffff92375b6aa400      0t0 1081656084 type=STREAM ->INO=1081656085 356,dockerd,62u
telegraf 23489 telegraf   57u  unix 0xffff92375b6a9000      0t0 1081656016 type=STREAM ->INO=1081656017 356,dockerd,61u
telegraf 23489 telegraf   58u  unix 0xffff9236c10e8000      0t0 1081501735 type=STREAM ->INO=1081501101 356,dockerd,51u
telegraf 23489 telegraf   59u  unix 0xffff9236c2c47400      0t0 1081776893 type=STREAM ->INO=1081778006 356,dockerd,70u
telegraf 23489 telegraf   60u  unix 0xffff9236d3757000      0t0 1081531372 type=STREAM ->INO=1081531595 356,dockerd,53u
telegraf 23489 telegraf   61u  unix 0xffff9236c2c47000      0t0 1081930181 type=STREAM ->INO=1081930944 356,dockerd,80u
telegraf 23489 telegraf   62u  unix 0xffff9236ddbe0800      0t0 1081562082 type=STREAM ->INO=1081562361 356,dockerd,55u
telegraf 23489 telegraf   63u  unix 0xffff9236d3754400      0t0 1081532442 type=STREAM ->INO=1081531649 356,dockerd,54u
telegraf 23489 telegraf   64u  unix 0xffff92375b6a8800      0t0 1081686384 type=STREAM ->INO=1081685630 356,dockerd,63u
telegraf 23489 telegraf   65u  unix 0xffff9236ddbe1000      0t0 1081747047 type=STREAM ->INO=1081747510 356,dockerd,68u
telegraf 23489 telegraf   66u  unix 0xffff92371914b800      0t0 1082238229 type=STREAM ->INO=1082236747 356,dockerd,100u
telegraf 23489 telegraf   67u  unix 0xffff9236c9ad6400      0t0 1082298684 type=STREAM ->INO=1082298685 356,dockerd,104u
telegraf 23489 telegraf   68u  unix 0xffff9236d4a74400      0t0 1083103864 type=STREAM ->INO=1083104319 356,dockerd,154u
telegraf 23489 telegraf   69u  unix 0xffff9236c2c47c00      0t0 1082021508 type=STREAM ->INO=1082021509 356,dockerd,85u
telegraf 23489 telegraf   70u  unix 0xffff92375b6ab800      0t0 1081626455 type=STREAM ->INO=1081626456 356,dockerd,59u
telegraf 23489 telegraf   71u  unix 0xffff9236d3ffe000      0t0 1081899382 type=STREAM ->INO=1081899383 356,dockerd,78u
telegraf 23489 telegraf   72u  unix 0xffff9236d8bdb000      0t0 1082541049 type=STREAM ->INO=1082541753 356,dockerd,118u
telegraf 23489 telegraf   73u  unix 0xffff9236ddbe2400      0t0 1081776847 type=STREAM ->INO=1081777846 356,dockerd,69u
telegraf 23489 telegraf   74u  unix 0xffff9236ddbe3800      0t0 1081715201 type=STREAM ->INO=1081715202 356,dockerd,66u
telegraf 23489 telegraf   75u  unix 0xffff92375b6aac00      0t0 1081806688 type=STREAM ->INO=1081806689 356,dockerd,71u
telegraf 23489 telegraf   76u  unix 0xffff92375b6a9800      0t0 1081716526 type=STREAM ->INO=1081716527 356,dockerd,65u
telegraf 23489 telegraf   77u  unix 0xffff9236d3ffd800      0t0 1081838022 type=STREAM ->INO=1081838976 356,dockerd,74u
telegraf 23489 telegraf   78u  unix 0xffff9236ddbe2000      0t0 1081747021 type=STREAM ->INO=1081747484 356,dockerd,67u
telegraf 23489 telegraf   79u  unix 0xffff9236d3ffe800      0t0 1081806745 type=STREAM ->INO=1081808050 356,dockerd,72u
telegraf 23489 telegraf   80u  unix 0xffff9236c9ad6800      0t0 1082051850 type=STREAM ->INO=1082051851 356,dockerd,87u
telegraf 23489 telegraf   81u  unix 0xffff923719149800      0t0 1082144195 type=STREAM ->INO=1082144887 356,dockerd,94u
telegraf 23489 telegraf   82u  unix 0xffff9236c2c45c00      0t0 1081868902 type=STREAM ->INO=1081868903 356,dockerd,75u
telegraf 23489 telegraf   83u  unix 0xffff9236d3ffe400      0t0 1081991680 type=STREAM ->INO=1081991681 356,dockerd,84u
telegraf 23489 telegraf   84u  unix 0xffff9236d3fff800      0t0 1081899339 type=STREAM ->INO=1081899340 356,dockerd,77u
telegraf 23489 telegraf   85u  unix 0xffff9236c2c45400      0t0 1081838959 type=STREAM ->INO=1081838960 356,dockerd,73u
telegraf 23489 telegraf   86u  unix 0xffff92371914a000      0t0 1082081036 type=STREAM ->INO=1082081037 356,dockerd,89u
telegraf 23489 telegraf   87u  unix 0xffff9236c2c44c00      0t0 1081990560 type=STREAM ->INO=1081990561 356,dockerd,83u
telegraf 23489 telegraf   88u  unix 0xffff9236c9ad7800      0t0 1082174413 type=STREAM ->INO=1082174414 356,dockerd,96u
telegraf 23489 telegraf   89u  unix 0xffff9236c2c44000      0t0 1081930139 type=STREAM ->INO=1081930140 356,dockerd,79u
telegraf 23489 telegraf   90u  unix 0xffff9236d8bd9c00      0t0 1082453910 type=STREAM ->INO=1082454268 356,dockerd,114u
telegraf 23489 telegraf   91u  unix 0xffff92371914ac00      0t0 1082206429 type=STREAM ->INO=1082206430 356,dockerd,97u
telegraf 23489 telegraf   92u  unix 0xffff9236d3ffd000      0t0 1081960791 type=STREAM ->INO=1081960061 356,dockerd,81u
telegraf 23489 telegraf   93u  unix 0xffff9236d4bffc00      0t0 1082392013 type=STREAM ->INO=1082392014 356,dockerd,110u
telegraf 23489 telegraf   94u  unix 0xffff9236d3ffc800      0t0 1082022004 type=STREAM ->INO=1082021561 356,dockerd,86u
telegraf 23489 telegraf   95u  unix 0xffff9236c9ad6000      0t0 1082207440 type=STREAM ->INO=1082206464 356,dockerd,98u
telegraf 23489 telegraf   96u  unix 0xffff923719148800      0t0 1082112684 type=STREAM ->INO=1082113168 356,dockerd,92u
telegraf 23489 telegraf   97u  unix 0xffff9236d8c9a000      0t0 1082786463 type=STREAM ->INO=1082786464 356,dockerd,134u
telegraf 23489 telegraf   98u  unix 0xffff9236c9ad5800      0t0 1082051890 type=STREAM ->INO=1082051891 356,dockerd,88u
telegraf 23489 telegraf   99u  unix 0xffff92371914a400      0t0 1082112617 type=STREAM ->INO=1082112618 356,dockerd,91u
telegraf 23489 telegraf  100u  unix 0xffff9236c9ad6c00      0t0 1082174384 type=STREAM ->INO=1082176116 356,dockerd,95u
telegraf 23489 telegraf  101u  unix 0xffff9236d4bfc400      0t0 1082362412 type=STREAM ->INO=1082362413 356,dockerd,107u
telegraf 23489 telegraf  102u  unix 0xffff923719149000      0t0 1082081097 type=STREAM ->INO=1082081985 356,dockerd,90u
telegraf 23489 telegraf  103u  unix 0xffff9236d8bd9400      0t0 1082392986 type=STREAM ->INO=1082391966 356,dockerd,109u
telegraf 23489 telegraf  104u  unix 0xffff9236d8bd8400      0t0 1082572966 type=STREAM ->INO=1082572161 356,dockerd,119u
...
```

Debug info from dockerd:
```
$ curl --unix-socket /var/run/docker.sock http://./debug/pprof/goroutine\?debug\=2
goroutine 20351474 [IO wait, 52 minutes]:
internal/poll.runtime_pollWait(0x7fb0e78150c0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0011af398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0011af380, 0xc000fb4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0011af380, 0xc000fb4000, 0x1000, 0x1000, 0x0, 0x17, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc00037bcb0, 0xc000fb4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x6a
github.com/docker/docker/cmd/dockerd/hack.(*MalformedHostHeaderOverrideConn).Read(0xc000aea780, 0xc000fb4000, 0x1000, 0x1000, 0xc00132f400, 0xc001249e60, 0x0)
        /go/src/github.com/docker/docker/cmd/dockerd/hack/malformed_host_override.go:110 +0x84
net/http.(*connReader).Read(0xc0012344e0, 0xc000fb4000, 0x1000, 0x1000, 0x558720e8c198, 0x558720e8c1a8, 0x5587218bb101)
        /usr/local/go/src/net/http/server.go:796 +0xf6
bufio.(*Reader).fill(0xc0012c58c0)
        /usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).ReadSlice(0xc0012c58c0, 0x55871f25860a, 0xc000d69680, 0xc0003a39a8, 0x55871f22c9c2, 0xc00132f700, 0x100)
        /usr/local/go/src/bufio/bufio.go:359 +0x3f
bufio.(*Reader).ReadLine(0xc0012c58c0, 0xc0003a39b0, 0xc000072700, 0x7fb0e797f008, 0x0, 0x0, 0xc0011af380)
        /usr/local/go/src/bufio/bufio.go:388 +0x36
net/textproto.(*Reader).readLineSlice(0xc001241ce0, 0xc00132f700, 0x0, 0x0, 0xc0003a3d70, 0xc0011af380)
        /usr/local/go/src/net/textproto/reader.go:57 +0x6e
net/textproto.(*Reader).ReadLine(...)
        /usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc0012c58c0, 0x0, 0xc00132f700, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:1012 +0x94
net/http.(*conn).readRequest(0xc0003ebe00, 0x558721be75c0, 0xc0012f9cc0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:982 +0x161
net/http.(*conn).serve(0xc0003ebe00, 0x558721be75c0, 0xc0012f9cc0)
        /usr/local/go/src/net/http/server.go:1845 +0x6d6
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 20353179 [IO wait, 32 minutes]:
internal/poll.runtime_pollWait(0x7fb0e004a9f0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc001323618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001323600, 0xc00136a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc001323600, 0xc00136a000, 0x1000, 0x1000, 0x0, 0x17, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0010b1678, 0xc00136a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x6a
github.com/docker/docker/cmd/dockerd/hack.(*MalformedHostHeaderOverrideConn).Read(0xc000ae5d80, 0xc00136a000, 0x1000, 0x1000, 0xc001380a00, 0xc0011e1710, 0x0)
        /go/src/github.com/docker/docker/cmd/dockerd/hack/malformed_host_override.go:110 +0x84
net/http.(*connReader).Read(0xc001384de0, 0xc00136a000, 0x1000, 0x1000, 0x558720e8c198, 0x558720e8c1a8, 0x5587218bb101)
        /usr/local/go/src/net/http/server.go:796 +0xf6
bufio.(*Reader).fill(0xc001a263c0)
        /usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).ReadSlice(0xc001a263c0, 0x55871f25860a, 0xc0012b3e00, 0xc0003a39a8, 0x55871f22c9c2, 0xc001380e00, 0x100)
        /usr/local/go/src/bufio/bufio.go:359 +0x3f
bufio.(*Reader).ReadLine(0xc001a263c0, 0xc0003a39b0, 0xc000072700, 0x7fb0e797f6d0, 0x0, 0x1, 0xc001323600)
        /usr/local/go/src/bufio/bufio.go:388 +0x36
net/textproto.(*Reader).readLineSlice(0xc0012e76b0, 0xc001380e00, 0x0, 0x0, 0xc0003a3d70, 0xc001323600)
        /usr/local/go/src/net/textproto/reader.go:57 +0x6e
net/textproto.(*Reader).ReadLine(...)
        /usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc001a263c0, 0x0, 0xc001380e00, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:1012 +0x94
net/http.(*conn).readRequest(0xc0009e6aa0, 0x558721be75c0, 0xc001344400, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:982 +0x161
net/http.(*conn).serve(0xc0009e6aa0, 0x558721be75c0, 0xc001344400)
        /usr/local/go/src/net/http/server.go:1845 +0x6d6
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2957 +0x386
...
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker input plugin: "too many open files" error #9242

Relevant telegraf.conf:

System info:

Steps to reproduce:

Expected behavior:

Actual behavior:

Additional info:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Docker input plugin: "too many open files" error #9242

Description

Relevant telegraf.conf:

System info:

Steps to reproduce:

Expected behavior:

Actual behavior:

Additional info:

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions