Cherry-pick #24424 to 7.x: Add tests and support for truncated files in filestream input#24959
Merged
kvch merged 1 commit intoelastic:7.xfrom Apr 7, 2021
Merged
Cherry-pick #24424 to 7.x: Add tests and support for truncated files in filestream input#24959kvch merged 1 commit intoelastic:7.xfrom
kvch merged 1 commit intoelastic:7.xfrom
Conversation
…c#24424) ## What does this PR do? Add support for truncated files and adds migrates related tests from `test_harvester.py`. It also adds more tests to cover the following cases: * truncation is detected only by the `Prospector` * truncation is detected first by the `Harvester` then by the `Prospector` * truncation is detected first by the `Prospector` then by the `Harvester` * file gets truncated when the output is not able to accept events ## Why is it important? The support for stopping reading from truncated files was already implemented. However, `filestream` input could not start reading it from the beginning. A new file system event is added called `OpTruncate`. When the size of a file has shrinked compared to the last time the scanner has encountered it, an `OpTruncate` event is emitted. When the prospector gets this event, the `HarvesterGroup` is restarting the `Harvester` of the file. Restarting basically means that the new `Harvester` cancels the previous reader and starts (cherry picked from commit 2875faf)
Contributor
|
Pinging @elastic/agent (Team:Agent) |
Contributor
Author
|
Failing test is unrelated. |
Contributor
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪Test errors
Expand to view the tests failures
|
| Test | Results |
|---|---|
| Failed | 1 |
| Passed | 12494 |
| Skipped | 2137 |
| Total | 14632 |
Genuine test errors 
💔 There are test failures but not known flaky tests, most likely a genuine test failure.
- Name:
Build&Test / filebeat-windows-8-windows-8 / test_tcp_with_custom_word_delimiter – filebeat.tests.system.test_tcp.Test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #24424 to 7.x branch. Original message:
What does this PR do?
Add support for truncated files and adds migrates related tests from
test_harvester.py. It also adds more tests to cover the following cases:ProspectorHarvesterthen by theProspectorProspectorthen by theHarvesterWhy is it important?
The support for stopping reading from truncated files was already implemented. However,
filestreaminput could not start reading it from the beginning.A new file system event is added called
OpTruncate. When the size of a file has shrinked compared to the last time the scanner has encountered it, anOpTruncateevent is emitted. When the prospector gets this event, theHarvesterGroupis restarting theHarvesterof the file. Restarting basically means that the newHarvestercancels the previous reader and starts a new one.Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.