Add troubleshooting docs about data corruption#88760
Merged
DaveCTurner merged 7 commits intoelastic:mainfrom Jul 28, 2022
Merged
Add troubleshooting docs about data corruption#88760DaveCTurner merged 7 commits intoelastic:mainfrom
DaveCTurner merged 7 commits intoelastic:mainfrom
Conversation
Adds some docs giving more detailed background about what data corruption really means and some suggestions about how to narrow down the root cause.
Collaborator
|
Pinging @elastic/es-docs (Team:Docs) |
Collaborator
|
Pinging @elastic/es-core-infra (Team:Core/Infra) |
DaveCTurner
commented
Jul 25, 2022
Comment on lines
+77
to
+81
| than {es} and look for data integrity errors. On Linux the `fio` and | ||
| `stress-ng` tools can both generate challenging I/O workloads and verify the | ||
| integrity of the data they write. You can check that durable writes persist | ||
| across power outages using a script such as | ||
| [`diskchecker.pl`]https://gist.github.com/bradfitz/3172656. Try different |
Member
Author
There was a problem hiding this comment.
Unclear if we should mention these tools like this. Maybe we should give example invocations for fio and stress-ng? diskchecker.pl looks janky but it's pretty much the best way to find fsync() bugs (it's in the PostgreSQL docs).
Contributor
henningandersen
left a comment
There was a problem hiding this comment.
This looks good, left a couple of mostly optional comments.
| apart from the data corruption, but data corruption itself is a very strong | ||
| indicator that your storage subsystem is not working correctly. | ||
|
|
||
| To narrow down the source of the corruptions, systematically change components |
Contributor
There was a problem hiding this comment.
I think I'd like to reword this a bit to be more of a non-exhaustive list of suggestion of ways to help diagnosing corruptions.
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
arteam
approved these changes
Jul 28, 2022
Contributor
arteam
left a comment
There was a problem hiding this comment.
This looks awesome! Thank you David!
|
|
||
| Verifying a checksum is expensive since it involves reading every byte of the | ||
| file which takes significant effort and might evict more useful data from the | ||
| filesystem cache, so systems typically doesn't verify the checksum on a file |
|
|
||
| The files that make up a Lucene index are written in full before they are used. | ||
| If a file is needed to recover an index after a restart then your storage | ||
| system will previously have confirmed to {es} that this file was durably synced |
Contributor
There was a problem hiding this comment.
Did you mean had previously confirmed here? will previously have confirmed sounds a bit strange to me.
Member
Author
There was a problem hiding this comment.
This one is legitimate English - if you want the technical details, will have confirmed is the future perfect tense, and that's acceptable to use in the first conditional form as I'm doing here. But I think it's fair to say that it could be confusing too, so I simplified it in 85a8ee2.
DaveCTurner
added a commit
that referenced
this pull request
Jul 28, 2022
Adds some docs giving more detailed background about what data corruption really means and some suggestions about how to narrow down the root cause. Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
DaveCTurner
added a commit
that referenced
this pull request
Jul 28, 2022
Adds some docs giving more detailed background about what data corruption really means and some suggestions about how to narrow down the root cause. Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Jul 29, 2022
* upstream/main: Add 8.5 migration docs (elastic#88923) Script: Reindex & UpdateByQuery Metadata (elastic#88665) Remove unused plugins dir var from server CLI (elastic#88917) Use tracing API in TaskManager (elastic#88885) Add source fallback for keyword fields using operation (elastic#88735) Prune changelogs after 8.3.3 release Bump versions after 8.3.3 release Add a test for checking for misspelled "dry_run" parameters for Desired Nodes API (elastic#88898) Speedup BalanceUnbalancedClusterTests (elastic#88794) Preventing exceptions on node shutdown in integration tests (elastic#88827) Do not trigger check part3 for test mute and docs PRs (elastic#88895) Add troubleshooting docs about data corruption (elastic#88760) Mute RollupActionSingleNodeTests#testRollupDatastream (elastic#88891) [DOCS] Domain splitting impacts API keys (elastic#88677) Fix SqlSearchIT testAllTypesWithRequestToOldNodes (elastic#88866) (elastic#88883) Update synthetic-source.asciidoc (elastic#88880) Log more details in TaskAssertions (elastic#88864) Make Tuple a record (elastic#88280)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds some docs giving more detailed background about what data
corruption really means and some suggestions about how to narrow down
the root cause.