crashes with "database is locked" #4857
Description
Issue and Steps to Reproduce
version is 0.10.1
but 0.9.3 did the same, maybe more often.
lightningd emergency exits itself cause the database says its locked during INSERT or UPDATE statements:
$ grep 'database is locked' crash.log.*
crash.log.20211009101318:+498.816259929 lightningdBROKEN: Error executing statement: wallet/wallet.c:3786: INSERT INTO transactions ( id, blockheight, txindex, rawtx) VALUES (?, ?, ?, ?);: database is locked
crash.log.20211009175028:+27421.465744630 lightningdBROKEN: Error executing statement: wallet/wallet.c:768: UPDATE shachains SET num_valid=?, min_index=? WHERE id=?: database is locked
crash.log.20211009175054:+17.143080552 lightningdBROKEN: Error executing statement: wallet/wallet.c:3540: INSERT INTO blocks (height, hash, prev_hash) VALUES (?, ?, ?);: database is locked
crash.log.20211009222937:+16702.140267710 lightningdBROKEN: Error executing statement: wallet/wallet.c:768: UPDATE shachains SET num_valid=?, min_index=? WHERE id=?: database is locked
crash.log.20211010034113:+18687.834439849 lightningdBROKEN: Error executing statement: wallet/wallet.c:768: UPDATE shachains SET num_valid=?, min_index=? WHERE id=?: database is locked
crash.log.20211011032432:+85389.796158637 lightningdBROKEN: Error executing statement: wallet/wallet.c:768: UPDATE shachains SET num_valid=?, min_index=? WHERE id=?: database is locked
Don't know how to reproduce. It happens few times a day, presumably related to how busy the node is, but that is a guess already - I failed to trigger it.
My Digging
Looking generally I find SQLite has a story with locks or here - I can not say whether this applies.
Naturally I suspect "race-condition because of threads" - but can't say anything either, not knowing lightningd's thread model...
On the hardware I can say its rather slow: 2 x 7200rpm HDD combined as zpool mirror. That is low latency compared to SSD. So, if lightningd is rarely used / never stress-tested on such disks, I guess such a thing can go unnoticed.