Skip to content

Fix missing WAL in new manifest by rolling over the WAL deletion record from prev manifest#10892

Closed
hx235 wants to merge 4 commits intofacebook:mainfrom
hx235:fix_missing_wal
Closed

Fix missing WAL in new manifest by rolling over the WAL deletion record from prev manifest#10892
hx235 wants to merge 4 commits intofacebook:mainfrom
hx235:fix_missing_wal

Conversation

@hx235
Copy link
Copy Markdown
Contributor

@hx235 hx235 commented Oct 27, 2022 •
edited
Loading

Context
Options::track_and_verify_wals_in_manifest = true verifies each of the WALs tracked in manifest indeed presents in the WAL folder. If not, a corruption "Missing WAL with log number" will be thrown.

DB::SyncWAL() called at a specific timing (i.e, at the TEST_SYNC_POINT("FindObsoleteFiles::PostMutexUnlock")) can record in a new manifest the WAL addition of a WAL file that already had a WAL deletion recorded in the previous manifest.
And the WAL deletion record is not rollover-ed to the new manifest. So the new manifest creates the illusion of such WAL never gets deleted and should presents at db re/open.

  • Such WAL deletion record can be caused by flushing the memtable associated with that WAL and such WAL deletion can actually happen in PurgeObsoleteFiles().

As a consequence, upon DB::Reopen(), this WAL file can be deleted while manifest still has its WAL addition record , which causes a false alarm of corruption "Missing WAL with log number" to be thrown.

Summary
This PR fixes this false alarm by rolling over the WAL deletion record from prev manifest to the new manifest by adding the WAL deletion record to the new manifest.

Test

  • Make check
  • Added new unit test TEST_F(DBWALTest, FixSyncWalOnObseletedWalWithNewManifestCausingMissingWAL) that failed before the fix and passed after
  • [Ongoing]CI stress test + aggressive value as in [CI only]Aggressive crash test value #10761 , which is how this false alarm was first surfaced, to confirm such false alarm disappears
  • [Ongoing]Regular CI stress test to confirm such fix didn't harm anything

@hx235 hx235 added the WIP Work in progress label Oct 27, 2022
@hx235 hx235 changed the title Delete WAL of number before version_set's min_log_number_to_keep in CheckIterationResult to fix missing WAL Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ Oct 27, 2022
@hx235 hx235 changed the title Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ in VersionEditHandler::CheckIterationResult Oct 27, 2022
@hx235 hx235 changed the title Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ in VersionEditHandler::CheckIterationResult Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ in VersionEditHandler::CheckIterationResult() Oct 27, 2022
@hx235 hx235 changed the title Fix missing WAL by deleting WAL before VersionSet::min_log_number_to_keep_ in VersionEditHandler::CheckIterationResult() Fix missing WAL by removing WAL before min log number to keep from manifest in VersionEditHandler::CheckIterationResult() Oct 27, 2022
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Oct 27, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Oct 27, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Oct 27, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Oct 28, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Oct 28, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@hx235 hx235 requested a review from ajkr October 31, 2022 18:24
ajkr
ajkr reviewed Oct 31, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@ajkr ajkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure yet. Isn't it odd that we write a WalAddition for 4.log when it's already obsolete, and then ignore it later?

ajkr
ajkr reviewed Oct 31, 2022
View reviewed changes
HISTORY.md Outdated
Copy link
Copy Markdown
Contributor

@ajkr ajkr Oct 31, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much implementation detail IMO. How about: "For users of track_and_verify_wals_in_manifest, fixed a race condition in MANIFEST rollover (see max_manifest_file_size) leading to false positive DB::Open() failures"?

edit: Or just delete the "obsoleted WAL and recording such WAL addition of an obsoleted WAL" part; I think that's the only part in yours that is too low level

Copy link
Copy Markdown
Contributor Author

@hx235 hx235 Nov 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

ajkr
ajkr suggested changes Nov 1, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@ajkr ajkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a straightforward thing we can add is change WriteCurrentStateToManifest() to call DeleteWalsBefore()

Ideally we would not have WalAdditions following WalDeletions affecting the same WAL, but that is a bigger problem unrelated to rollover or not - it just works fine in case no rollover happened.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@hx235 hx235 changed the title Fix missing WAL by removing WAL before min log number to keep from manifest in VersionEditHandler::CheckIterationResult() Fix missing WAL by DeleteWalsBefore() in ProcessManifestWrites to respect min log number to keep Nov 8, 2022
@hx235 hx235 changed the title Fix missing WAL by DeleteWalsBefore() in ProcessManifestWrites to respect min log number to keep Fix missing WAL by DeleteWalsBefore() in ProcessManifestWrites() to respect min log number to keep Nov 8, 2022
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@hx235
Copy link
Copy Markdown
Contributor Author

hx235 commented Nov 8, 2022 •
edited
Loading

Update: addressed feedback and going thru stress test now.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

1 similar comment
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 8, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

hx235
hx235 commented Nov 28, 2022
View reviewed changes
db/version_set.cc Outdated
Copy link
Copy Markdown
Contributor Author

@hx235 hx235 Nov 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto format change.

@hx235 hx235 changed the title Fix missing WAL by DeleteWalsBefore() in ProcessManifestWrites() to respect min log number to keep Fix missing WAL in new manifest by rolling over the WAL deletion record from prev manifest Nov 28, 2022
ajkr
ajkr approved these changes Nov 28, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@ajkr ajkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 28, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 29, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 29, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 29, 2022

@hx235 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Nov 29, 2022

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@hx235 hx235 mentioned this pull request Dec 1, 2022
Closed
facebook-github-bot pushed a commit that referenced this pull request Dec 7, 2022
@hx235 @facebook-github-bot
Move two history entries mistake out of 7.9 section (#11013)
5f52dbc 
Summary:
#10892 and #10955 mistakenly added two entries under sealed 7.9.history section. This PR fixes these two.

No need to update 7.9 branch (https://github.com/facebook/rocksdb/blob/7.9.fb/HISTORY.md) cuz it's cut before these two PRs landed

Pull Request resolved: #11013

Reviewed By: cbi42

Differential Revision: D41666514

Pulled By: hx235

fbshipit-source-id: c4bc7a29ff663664bf0be1ba1c7eab4d00a61528
@hx235 hx235 mentioned this pull request Feb 6, 2023
Closed
facebook-github-bot pushed a commit that referenced this pull request Feb 7, 2023
@hx235 @facebook-github-bot
Simplify TEST_F(DBWALTest, FixSyncWalOnObseletedWalWithNewManifestCau…
9b66331 
…singMissingWAL) (#11186)

Summary:
**Context/Summary**:
Simplify `TEST_F(DBWALTest, FixSyncWalOnObseletedWalWithNewManifestCausingMissingWAL)` based on #11016 (review) and delete unused sync points.

Pull Request resolved: #11186

Test Plan:
- UT failed before fix in #10892 and passes after
- Check UT not flaky when running with https://app.circleci.com/pipelines/github/facebook/rocksdb/21985/workflows/5f6cc355-78c1-46d8-89ee-0fd679725a8a/jobs/540878

Reviewed By: ajkr

Differential Revision: D43034723

Pulled By: hx235

fbshipit-source-id: f503774987b8f3718505f99e95080a7fad28ac66
@igorcanadi igorcanadi mentioned this pull request Jan 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@ajkr ajkr ajkr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hx235 @facebook-github-bot @ajkr