-
Notifications
You must be signed in to change notification settings - Fork 38.7k
coins: use number of dirty cache entries in flush warnings/logs #33512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33512. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
https://github.com/bitcoin/bitcoin/actions/runs/18149289661/job/51656823259?pr=33512#step:9:2059: Run coins_view_db with args ['/home/admin/actions-runner/_work/_temp/build/bin/fuzz', '-runs=1', PosixPath('/home/admin/actions-runner/_work/_temp/ci/scratch/qa-assets/fuzz_corpora/coins_view_db')]INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1436700023
INFO: Loaded 1 modules (611994 inline 8-bit counters): 611994 [0x6082fc8a27f8, 0x6082fc937e92),
INFO: Loaded 1 PC tables (611994 PCs): 611994 [0x6082fc937e98,0x6082fd28e838),
INFO: 2396 files found in /home/admin/actions-runner/_work/_temp/ci/scratch/qa-assets/fuzz_corpora/coins_view_db
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 275937 bytes
INFO: seed corpus: files: 2396 min: 2b max: 275937b total: 27207582b rss: 109Mb
/home/admin/actions-runner/_work/_temp/src/coins.h:288:23: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'size_t' (aka 'unsigned long')
#0 0x6082f97eb6c4 in CoinsViewCacheCursor::NextAndMaybeErase(std::pair<COutPoint const, CCoinsCacheEntry>&) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./coins.h:288:23
#1 0x6082fb3e30bb in CCoinsViewDB::BatchWrite(CoinsViewCacheCursor&, uint256 const&) /home/admin/actions-runner/_work/_temp/build/src/./txdb.cpp:137:21
#2 0x6082f9ec4573 in CCoinsViewCache::Flush() /home/admin/actions-runner/_work/_temp/build/src/./coins.cpp:265:22
#3 0x6082f97cb77f in TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:77:48
#4 0x6082f97cb77f in unsigned long CallOneOf<TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_10>(FuzzedDataProvider&, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_10) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/util.h:42:27
#5 0x6082f97cb77f in TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:56:9
#6 0x6082f97cfc0c in coins_view_db_fuzz_target(std::span<unsigned char const, 18446744073709551615ul>) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:323:5
#7 0x6082f9e6efae in std::function<void (std::span<unsigned char const, 18446744073709551615ul>)>::operator()(std::span<unsigned char const, 18446744073709551615ul>) const /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
#8 0x6082f9e6efae in test_one_input(std::span<unsigned char const, 18446744073709551615ul>) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/util/./test/fuzz/fuzz.cpp:88:5
#9 0x6082f9e6efae in LLVMFuzzerTestOneInput /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/util/./test/fuzz/fuzz.cpp:216:5
#10 0x6082f947d66f in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a8366f) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#11 0x6082f947cc79 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a82c79) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#12 0x6082f947e9e2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a849e2) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#13 0x6082f947ef00 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a84f00) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#14 0x6082f946b585 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a71585) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#15 0x6082f9497946 in main (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a9d946) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#16 0x756d565fa1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
#17 0x756d565fa28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
#18 0x6082f945fb54 in _start (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a65b54) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow /home/admin/actions-runner/_work/_temp/src/coins.h:288:23
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0x1,0xff,0xff,0xff,0xf2,0xfc,0xff,0x2f,0x1f,0xa9,0x5d,0x30,0x30,0x90,0x0,0x0,0x0,0x14,0x69,0x0,0x0,0x0,0x0,0x0,0x0,0xc1,0xc1,0xc1,0xc1,0xc1,0xc1,0xb7,0xc1,0xc1,0x5c,0x91,0x3b,0x72,0x0,0xff,0xfa,0x21,
\001\377\377\377\362\374\377/\037\251]00\220\000\000\000\024i\000\000\000\000\000\000\301\301\301\301\301\301\267\301\301\\\221;r\000\377\372!
artifact_prefix='./'; Test unit written to ./crash-ebc99ca6d40a3c7a37202cdba9a8b36a2cb6007b
Base64: Af////L8/y8fqV0wMJAAAAAUaQAAAAAAAMHBwcHBwbfBwVyRO3IA//oh
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1436700023
INFO: Loaded 1 modules (611994 inline 8-bit counters): 611994 [0x6082fc8a27f8, 0x6082fc937e92),
INFO: Loaded 1 PC tables (611994 PCs): 611994 [0x6082fc937e98,0x6082fd28e838),
INFO: 2396 files found in /home/admin/actions-runner/_work/_temp/ci/scratch/qa-assets/fuzz_corpora/coins_view_db
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 275937 bytes
INFO: seed corpus: files: 2396 min: 2b max: 275937b total: 27207582b rss: 109Mb
/home/admin/actions-runner/_work/_temp/src/coins.h:288:23: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'size_t' (aka 'unsigned long')
#0 0x6082f97eb6c4 in CoinsViewCacheCursor::NextAndMaybeErase(std::pair<COutPoint const, CCoinsCacheEntry>&) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./coins.h:288:23
#1 0x6082fb3e30bb in CCoinsViewDB::BatchWrite(CoinsViewCacheCursor&, uint256 const&) /home/admin/actions-runner/_work/_temp/build/src/./txdb.cpp:137:21
#2 0x6082f9ec4573 in CCoinsViewCache::Flush() /home/admin/actions-runner/_work/_temp/build/src/./coins.cpp:265:22
#3 0x6082f97cb77f in TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0::operator()() const /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:77:48
#4 0x6082f97cb77f in unsigned long CallOneOf<TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_10>(FuzzedDataProvider&, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)::$_10) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/util.h:42:27
#5 0x6082f97cb77f in TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:56:9
#6 0x6082f97cfc0c in coins_view_db_fuzz_target(std::span<unsigned char const, 18446744073709551615ul>) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/./test/fuzz/coins_view.cpp:323:5
#7 0x6082f9e6efae in std::function<void (std::span<unsigned char const, 18446744073709551615ul>)>::operator()(std::span<unsigned char const, 18446744073709551615ul>) const /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
#8 0x6082f9e6efae in test_one_input(std::span<unsigned char const, 18446744073709551615ul>) /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/util/./test/fuzz/fuzz.cpp:88:5
#9 0x6082f9e6efae in LLVMFuzzerTestOneInput /home/admin/actions-runner/_work/_temp/build/src/test/fuzz/util/./test/fuzz/fuzz.cpp:216:5
#10 0x6082f947d66f in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a8366f) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#11 0x6082f947cc79 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a82c79) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#12 0x6082f947e9e2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a849e2) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#13 0x6082f947ef00 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a84f00) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#14 0x6082f946b585 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a71585) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#15 0x6082f9497946 in main (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a9d946) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
#16 0x756d565fa1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
#17 0x756d565fa28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
#18 0x6082f945fb54 in _start (/home/admin/actions-runner/_work/_temp/build/bin/fuzz+0x1a65b54) (BuildId: c46e5f78c23c1d0b2ce0fb85b0817bc7fae97dd8)
SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow /home/admin/actions-runner/_work/_temp/src/coins.h:288:23
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0x1,0xff,0xff,0xff,0xf2,0xfc,0xff,0x2f,0x1f,0xa9,0x5d,0x30,0x30,0x90,0x0,0x0,0x0,0x14,0x69,0x0,0x0,0x0,0x0,0x0,0x0,0xc1,0xc1,0xc1,0xc1,0xc1,0xc1,0xb7,0xc1,0xc1,0x5c,0x91,0x3b,0x72,0x0,0xff,0xfa,0x21,
\001\377\377\377\362\374\377/\037\251]00\220\000\000\000\024i\000\000\000\000\000\000\301\301\301\301\301\301\267\301\301\\\221;r\000\377\372!
artifact_prefix='./'; Test unit written to ./crash-ebc99ca6d40a3c7a37202cdba9a8b36a2cb6007b
Base64: Af////L8/y8fqV0wMJAAAAAUaQAAAAAAAMHBwcHBwbfBwVyRO3IA//oh
⚠️ Failure generated from target with exit code 1: ['/home/admin/actions-runner/_work/_temp/build/bin/fuzz', '-runs=1', PosixPath('/home/admin/actions-runner/_work/_temp/ci/scratch/qa-assets/fuzz_corpora/coins_view_db')] |
4b31781 to
cb270b0
Compare
|
@fanquake the fuzz test is fixed in a separate PR, I have reverted a similar attempt here |
cb270b0 to
b9ce8f3
Compare
b9ce8f3 to
bedb063
Compare
bedb063 to
1435099
Compare
78f1f46 to
0fe4ba2
Compare
0fe4ba2 to
62cb2a2
Compare
|
Rebased after #32313, the PR is simpler and better tested this way. |
54151d6 to
62cb2a2
Compare
62cb2a2 to
2327159
Compare
|
A general observation: keeping a count like the dirty count is very efficient, but prone to going out of sync, e.g with future incorrect code changes. For eg., a missing decrement can result in the value creeping higher, a missing increment can result the value going lower, event to underflow -- |
d850b49 to
3631ca7
Compare
l0rinc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the comments @optout21, since I wasn't invalidating any ACKs, I have applied your suggestions (with rebase) - let me know if this is what you meant.
keeping a count like the dirty count is very efficient, but prone to going out of sync
yeah, we have a few of these - but the sanitizers and sanity checks and fuzzers usually reveal the problems (sanitizers break on underflow and the sanity checks recalculate everything during testing).
3631ca7 to
d119cdf
Compare
|
Clean rebase, reviews are welcome again. |
d119cdf to
eac63b2
Compare
Adds test coverage by randomly calling `EmplaceCoinInternalDANGER` in `SimulationTest` to verify it remains correct as we modify it in a future commit. Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Adds `m_dirty_count` member to track the running count of dirty cache entries as follows: * Incremented when entries are marked dirty via `CCoinsCacheEntry::SetDirty` * Decremented when dirty entries are removed or cleaned * Passed through `CoinsViewCacheCursor` and updated during iteration * Validated in `SanityCheck()` by recomputing from scratch The dirty count is needed because after non-wiping flushes (introduced in bitcoin#28280 and bitcoin#28233), the percentage of dirty entries in the cache may be far below 100%. Using total cache size for flush warnings and disk space checks is therefore misleading. Updates all test code to properly initialize and maintain the dirty count. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
…ecks Changes flush warnings to use the actual number of dirty entries being written rather than total cache size or memory usage: * Moves warning from `FlushStateToDisk` to `CCoinsViewDB::BatchWrite` so it applies to both regular flushes and `AssumeUTXO` snapshot writes * Changes threshold from `WARN_FLUSH_COINS_SIZE` (1 GiB) to `WARN_FLUSH_COINS_COUNT` (10M entries), approximately equivalent - this also helps with the confusion caused by UTXO size difference on-disk vs in-memory * Moves benchmark logging to `BatchWrite` where the actual disk I/O occurs to make sure AssumeUTXO also warns * Uses dirty count for disk space check (48 bytes per entry estimate) * Removes redundant `changed` counter since `dirty_count` is now tracked This ensures users are warned appropriately even when only a fraction of the cache is dirty, and provides accurate warnings during `AssumeUTXO` loads. Co-authored-by: l0rinc <pap.lorinc@gmail.com>
eac63b2 to
0ac4cb6
Compare
|
Rebased after #33866
Ready for review again! |
| ReallocateCache(); | ||
| } | ||
| cachedCoinsUsage = 0; | ||
| m_dirty_count = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We call Assume(m_dirty_count == 0); in ReallocateCache(); above, but only set m_dirty_count = 0; here. We should just Assume(m_dirty_count == 0); after BatchWrite like we do with Sync.
andrewtoth
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 0ac4cb6
This removes redundant logging of the large flush warning when running in pruned mode or in steady state, the latter of which happens more often now that we're flushing every hour.
This also does a more accurate disk space check, which currently might needlessly shutdown nodes even though the flush is actually small and can fit in the available disk space.
Revival of #31703 with all outstanding review feedback addressed.
Changes compared to the original
EmplaceCoinInternalDANGERis also fuzz tested nowFlushStateToDisktoCCoinsViewDB::BatchWrite, ensuringAssumeUTXOsnapshot writes also display progress warningsBatchWritesince we now trackdirty_countdirectlydirty_count(not num_dirty) and parameter alignment is corrected