Skip to content

vinyl: fix duplicate multikey stmt accounting with deferred deletes#10807

Merged
locker merged 2 commits intotarantool:masterfrom
locker:vy-duplicate-multikey-stmt-accounting-fix
Nov 12, 2024
Merged

vinyl: fix duplicate multikey stmt accounting with deferred deletes#10807
locker merged 2 commits intotarantool:masterfrom
locker:vy-duplicate-multikey-stmt-accounting-fix

Conversation

@locker
Copy link
Member

@locker locker commented Nov 11, 2024

vinyl-luatest/select_consistency_test.lua revealed a problem with accounting of duplicate multikey statements in memory in case the defer_deletes option is enabled. This PR fixes the problem and also makes the test more thorough and robust.

Closes #10751
Closes #10752

@locker locker requested a review from a team as a code owner November 11, 2024 15:57
`vy_mem_insert()` and `vy_mem_insert_upsert()` increment the row count
statistic of `vy_mem` only if no statement is replaced, which is
correct, while `vy_lsm_commit()` increments the row count of `vy_lsm`
unconditionally. As a result, `vy_lsm` may report a non-zero statement
count (via `index.stat()` or `index.len()`) after a dump. This may
happen only with a non-unique multikey index, when the statement has
duplicates in the indexed array, and only if the `deferred_deletes`
option is enabled, because otherwise we drop duplicates when we form
the transaction write set, see `vy_tx_set()`. With `deferred_deletes`,
we may create a `txv` for each multikey entry at the time when we
prepare to commit the transaction, see `vy_tx_handle_deferred_delete()`.

Another problem is that `vy_mem_rollback_stmt()` always decrements
the row count, even if it didn't find the rolled back statement in
the tree. As a result, if the transaction with duplicate multikey
entries is rolled back on WAL error, we'll decrement the row count
of `vy_mem` more times than necessary.

To fix this issue, let's make the `vy_mem` methods update the in-memory
statistic of `vy_lsm`. This way they should always stay in-sync. Also,
we make `vy_mem_rollback_stmt()` skip updating the statistics in case
the rolled back statement isn't present in the tree.

This issue results in `vinyl-luatest/select_consistency_test.lua`
flakiness when checking `index.len()` after compaction. Let's make
the test more thorough and also check that `index.len()` equals
`index.count()`.

Closes tarantool#10751
Part of tarantool#10752

NO_DOC=bug fix
The test expects at least 10 dumps to be created in 60 seconds. It
usually works but sometimes, when the host is heavy loaded, Vinyl
doesn't produce enough dumps in time and fails the test. On CI the test
usually fails with 7-9 dumps. To avoid flaky failures, let's reduce the
expected dump count down to 5.

Closes tarantool#10752

NO_DOC=test fix
NO_CHANGELOG=test fix
@locker locker force-pushed the vy-duplicate-multikey-stmt-accounting-fix branch from a66a8a9 to 6d6b1a0 Compare November 11, 2024 15:59
@locker locker requested a review from nshy November 11, 2024 16:13
@coveralls
Copy link

Coverage Status

coverage: 87.298% (+0.002%) from 87.296%
when pulling 6d6b1a0 on locker:vy-duplicate-multikey-stmt-accounting-fix
into f69e2ae
on tarantool:master
.

Copy link
Contributor

@nshy nshy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@locker locker assigned locker and unassigned nshy Nov 12, 2024
@locker locker added the full-ci Enables all tests for a pull request label Nov 12, 2024
@locker locker merged commit 5325abd into tarantool:master Nov 12, 2024
@locker locker deleted the vy-duplicate-multikey-stmt-accounting-fix branch November 12, 2024 10:44
@locker
Copy link
Member Author

locker commented Nov 12, 2024

Cherry-picked to 2.11 and 3.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Enables all tests for a pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vinyl-luatest/select_consistency_test.lua is flaky Invalid memory row count for multikey index with deferred deletes

4 participants