-
Notifications
You must be signed in to change notification settings - Fork 38.7k
Prevent concurrent savemempool #12842
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
In order to test the new check and error add: --- a/src/rpc/blockchain.cpp
+++ b/src/rpc/blockchain.cpp
@@ -1619,6 +1619,8 @@ UniValue savemempool(const JSONRPCRequest& request)
throw JSONRPCError(RPC_MISC_ERROR, "Currently dumping the mempool");
}
+ MilliSleep(10000);
+
if (!g_is_mempool_loaded) {
throw JSONRPCError(RPC_MISC_ERROR, "The mempool was not loaded yet");
}Then run bitcoin-cli -regtest savemempool & bitcoin-cli -regtest savemempoolThe second should wait for the first to terminate. |
src/rpc/blockchain.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: maybe instead,
"savemempool already in progress"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds better.
conscott
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested ACK 7f722a1f0c82e1cc8dd9f6af9159bf7d7b8f2626
|
After #12863 I'm even more convinced that this might be the wrong approach. Protecting access using a mutex instead of returning these kind of 'busy errors' would avoid having to implement retry-poll-loops client-side (including in the tests). If you want to stick with this you'd at least need to define a new error code that means 'transient error', similar to |
|
@laanwj indeed it's very arguable. I think it's preferable to have the client retrying than reserving resources on the server side. IMO having the client waiting is also not desirable because the client can wait indefinitely and also timeout (but then he can raise the timeout) and we would process the request anyway (?). Getting an error instead of waiting is also more informative and allows the client to explicitly log and retry when he feels like. |
|
OTOH
|
|
I think we had reports that loading the mempool took 1.5 hours or so, just noting without further comment. |
The caller should always have error handling, and this new error would fall in his "unkown error".
We have calls that can hang a lot of time, for instance when the wallet is really big (as in lots of keys, lots of transactions).
Right, but at least a thread and a socket is on hold. And when the mutex is acquired, it might do something that is no longer relevant - in this case dumping the mempool again.. We have this semantic in And regarding #12863, having the client blindingly wait for 1.5 hours should not be an option (?) |
|
I'm also more a fan of blocking until the existing save has finished. Given that no modifications in the mempool can happen in the mean time anyway (I think?), we could even just wait until the existing save operation finishes and then return from both RPC calls (without an additional save). |
Actually it can, |
|
What should be done here? I've to address @conscott comment above, but I guess I should wait for more concept ACKs. |
|
I agree with @laanwj - it should just hold a lock on the file to avoid problems, but otherwise succeed. |
0b68854 to
535be5b
Compare
|
Rebased and updated to block/wait while there is another dump. |
535be5b to
35bdaf7
Compare
|
@luke do you think the current code is enough? |
|
utACK 35bdaf7489cf33d120b3c66f87067c2698ed2e8c |
| The last travis run for this pull request was 57 days ago and is thus outdated. To trigger a fresh travis build, this pull request should be closed and re-opened. |
src/txmempool.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(a) Shouldn't this be a global mutex, since it's specific to the "mempool.dat.new" file whose path is based on the (global) GetDataDir()?
(b) Why a std::mutex instead of a CCriticalSection?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why a CCriticalSection instead of a Mutex?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a) Not sure why that matters. To reduce exposure the mutex could be a static variable in DumpMempool() function;
b) we don't need a recursive mutex.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a) Yeah, a static variable in/next to the DumpMempool() function was what I was thinking. I don't think it matters in practice though.
b) More about trying to just use one set of locking primitives; Mutex/LOCK per MarcoFalke's suggestion would make more sense to me.
src/validation.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could rebase on master, make the type Mutex and use LOCK(...) here?
35bdaf7 to
585b47c
Compare
|
@MarcoFalke @achow101 rebased on master and changed to |
|
re-utACK 585b47c |
|
I really like this clean and simple solution utACK 585b47c |
585b47c rpc: Prevent concurrent savemempool (João Barbosa) Pull request description: Follow up of #12172, this change prevents calling `savemempool` RPC concurrently. Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
585b47c rpc: Prevent concurrent savemempool (João Barbosa) Pull request description: Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently. Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
585b47c rpc: Prevent concurrent savemempool (João Barbosa) Pull request description: Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently. Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
585b47c rpc: Prevent concurrent savemempool (João Barbosa) Pull request description: Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently. Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
Follow up of #12172, this change prevents calling
savemempoolRPC concurrently.