Skip to content

[8.1](backport #30717) filebeat: ID for filestream is required#30998

Merged
rdner merged 2 commits into8.1from
mergify/bp/8.1/pr-30717
Mar 28, 2022
Merged

[8.1](backport #30717) filebeat: ID for filestream is required#30998
rdner merged 2 commits into8.1from
mergify/bp/8.1/pr-30717

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Mar 24, 2022 •
edited by zube bot
Loading

This is an automatic backport of pull request #30717 done by Mergify.
Cherry-pick of c61b219 has failed:

On branch mergify/bp/8.1/pr-30717
Your branch is up to date with 'origin/8.1'.

You are currently cherry-picking commit c61b21920e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   filebeat/input/filestream/environment_test.go
	modified:   filebeat/input/filestream/input_integration_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/harvester_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/manager_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/prospector.go
	modified:   filebeat/input/filestream/internal/input-logfile/store.go
	modified:   filebeat/input/filestream/internal/input-logfile/store_test.go
	modified:   filebeat/input/filestream/prospector.go
	modified:   filebeat/input/filestream/prospector_test.go
	new file:   filebeat/tests/system/config/filestream-fixup-id.yml.j2
	modified:   filebeat/tests/system/filebeat.py
	new file:   filebeat/tests/system/input/filestream-fix-registry-global-id.j2
	new file:   filebeat/tests/system/input/registry-meta.json
	modified:   filebeat/tests/system/test_input.py
	modified:   testing/environments/local.yml

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .golangci.yml
	deleted by us:   dev-tools/templates/.golangci.yml
	both modified:   filebeat/docs/inputs/input-filestream-file-options.asciidoc
	both modified:   filebeat/input/filestream/internal/input-logfile/manager.go
	both modified:   filebeat/input/filestream/parsers_integration_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

@belimawr
filebeat: Migrates registry entries from filestream input without ID (#…
df09eb6 
…30717)

When a filestream input does not have an ID, we set the default
.global ID at runtime and this ID is used to identify which files in
the registry belong to each input. But if there are more than one
input without an ID, then data is duplicated when Filebeat is
restarted because the it's not possible to identify which input owns
each file.

We are adding a way to migrate those entries in the registry to the
newly set ID. Ideally this will happen only once when the users update
to this new version of Filebeat.

They way we do it is by looking the registry for files that have their
path matching the input but have the input ID as .global, those
entries are then copied with the new key (which uses the input ID) so
Filebeat can continue its operation without duplicating data.

Most of the logic is the same of the already existing method
UpdateIdentifiers from the sourceStore but for this PR it was also
needed to update the in-memory store right away. Also borrowing from
UpdateIdentifiers usage, this is done during the file prospector
initialisation.

Log errors are written when we detect filestream inputs without an ID
or when more than one input is set with the same ID.

Some files are refactored to pass the new linter, some test code also
had some small refactoring.

Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
(cherry picked from commit c61b219)

# Conflicts:
#	.golangci.yml
#	dev-tools/templates/.golangci.yml
#	filebeat/docs/inputs/input-filestream-file-options.asciidoc
#	filebeat/input/filestream/internal/input-logfile/manager.go
#	filebeat/input/filestream/parsers_integration_test.go
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Mar 24, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 24, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-03-24T18:55:07.256+0000

  • Duration: 125 min 29 sec

Test stats 🧪

Test Results
Failed 0
Passed 25672
Skipped 2192
Total 27864

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@belimawr belimawr force-pushed the mergify/bp/8.1/pr-30717 branch from 6ed2293 to 60625b0 Compare March 24, 2022 18:50
@belimawr belimawr force-pushed the mergify/bp/8.1/pr-30717 branch from 60625b0 to ed81d3e Compare March 24, 2022 18:54
@belimawr belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 25, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 25, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 25, 2022
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Mar 28, 2022

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

@rdner rdner merged commit 6118f25 into 8.1 Mar 28, 2022
@rdner rdner deleted the mergify/bp/8.1/pr-30717 branch March 28, 2022 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@belimawr belimawr belimawr approved these changes

Assignees

@belimawr belimawr

Labels

backport conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@elasticmachine @belimawr @rdner @jlind23