I[06-18|21:19:03.613] Served RPC HTTP response                     module=rpc-server method=GET url=/consensus_state status=200 duration=0 remoteAddr=127.0.0.1:33874             2018/06/18 21:19:03 http: Accept error: accept tcp [::]:46657: accept4: too many open files; retrying in 1s
I[06-18|21:19:03.884] HTTPRestRPC                                  module=rpc-server method=/validators args="[<*int64 Value>]" returns="[<*core_types.ResultValidators Value> <error Value>]"
I[06-18|21:19:03.885] Served RPC HTTP response                     module=rpc-server method=GET url=/validators status=200 duration=1 remoteAddr=127.0.0.1:33936                  I[06-18|21:19:03.885] HTTPRestRPC                                  module=rpc-server method=/status args=[] returns="[<*core_types.ResultStatus Value> <error Value>]"
I[06-18|21:19:03.885] Served RPC HTTP response                     module=rpc-server method=GET url=/status status=200 duration=0 remoteAddr=127.0.0.1:33874                      I[06-18|21:19:03.970] Timed out                                    module=consensus dur=3s height=197705 round=0 step=RoundStepPropose
panic: open /home/pz/.gaiad/data/cs.wal/wal: too many open files

goroutine 3001 [running]:                                                                                                                                                         github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile.(*Group).checkHeadSizeLimit(0xc43e18a320)
        /home/pz/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile/group.go:225 +0xa9
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile.(*Group).processTicks(0xc43e18a320)                                                                             /home/pz/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile/group.go:207 +0x2d
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile.(*Group).OnStart                                                                                     /home/pz/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/autofile/group.go:105 +0x3f

Peng:
I did some digging into this `too many open files` error. The issue is the system's max open files per process.
This is the command to find how many files `gaiad` has open:
`ls /proc/(pidof gaiad)/fd/ | wc -l`
For my Debian instance, `ulimit -Sn` (the max open files per process) is `1024`
For my faucet node, which has not gone down yet, `ls /proc/(pidof gaiad)/fd/ | wc -l` returns `963` files open
For my explorer and validator nodes, both have crashed twice already tue to `too many open files`. I surmise that both of those nodes' `gaiad` processes attempted to open more than `1024` files.

My temporary solution will be to edit `vi /etc/security/limits.conf` and increase the system's open files limit per process to 2048 and see what happens.

Ethan:
defntly doesnt seem right that they just keep growing

Peng:
Ok, both nodes crashed again. Validator due to `CONSENSUS FAILURE - too many open files`. Will attempt to increase the fd limit now

Full node crashes with too many RPC connections #1740

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions