Skip to content

[7.17](backport #30717) filebeat: ID for filestream is required#30996

Merged
belimawr merged 2 commits into7.17from
mergify/bp/7.17/pr-30717
Mar 25, 2022
Merged

[7.17](backport #30717) filebeat: ID for filestream is required#30996
belimawr merged 2 commits into7.17from
mergify/bp/7.17/pr-30717

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Mar 24, 2022 •
edited by zube bot
Loading

This is an automatic backport of pull request #30717 done by Mergify.
Cherry-pick of c61b219 has failed:

On branch mergify/bp/7.17/pr-30717
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit c61b21920e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   filebeat/input/filestream/environment_test.go
	modified:   filebeat/input/filestream/input_integration_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/harvester_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/manager_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/prospector.go
	modified:   filebeat/input/filestream/internal/input-logfile/store.go
	modified:   filebeat/input/filestream/internal/input-logfile/store_test.go
	modified:   filebeat/input/filestream/prospector.go
	modified:   filebeat/input/filestream/prospector_test.go
	new file:   filebeat/tests/system/config/filestream-fixup-id.yml.j2
	modified:   filebeat/tests/system/filebeat.py
	new file:   filebeat/tests/system/input/filestream-fix-registry-global-id.j2
	new file:   filebeat/tests/system/input/registry-meta.json
	modified:   filebeat/tests/system/test_input.py
	modified:   testing/environments/local.yml

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .golangci.yml
	deleted by us:   dev-tools/templates/.golangci.yml
	both modified:   filebeat/docs/inputs/input-filestream-file-options.asciidoc
	both modified:   filebeat/input/filestream/internal/input-logfile/manager.go
	both modified:   filebeat/input/filestream/parsers_integration_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

@belimawr
filebeat: Migrates registry entries from filestream input without ID (#…
d4b1cc5 
…30717)

When a filestream input does not have an ID, we set the default
.global ID at runtime and this ID is used to identify which files in
the registry belong to each input. But if there are more than one
input without an ID, then data is duplicated when Filebeat is
restarted because the it's not possible to identify which input owns
each file.

We are adding a way to migrate those entries in the registry to the
newly set ID. Ideally this will happen only once when the users update
to this new version of Filebeat.

They way we do it is by looking the registry for files that have their
path matching the input but have the input ID as .global, those
entries are then copied with the new key (which uses the input ID) so
Filebeat can continue its operation without duplicating data.

Most of the logic is the same of the already existing method
UpdateIdentifiers from the sourceStore but for this PR it was also
needed to update the in-memory store right away. Also borrowing from
UpdateIdentifiers usage, this is done during the file prospector
initialisation.

Log errors are written when we detect filestream inputs without an ID
or when more than one input is set with the same ID.

Some files are refactored to pass the new linter, some test code also
had some small refactoring.

Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
(cherry picked from commit c61b219)

# Conflicts:
#	.golangci.yml
#	dev-tools/templates/.golangci.yml
#	filebeat/docs/inputs/input-filestream-file-options.asciidoc
#	filebeat/input/filestream/internal/input-logfile/manager.go
#	filebeat/input/filestream/parsers_integration_test.go
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Mar 24, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2022
@botelastic
Copy link
Copy Markdown

botelastic bot commented Mar 24, 2022

This pull request doesn't have a Team:<team> label.

@belimawr belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 24, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 24, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@belimawr belimawr force-pushed the mergify/bp/7.17/pr-30717 branch 2 times, most recently from db05291 to 7a5211c Compare March 24, 2022 18:40
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 24, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-03-24T18:59:00.197+0000

  • Duration: 85 min 39 sec

Test stats 🧪

Test Results
Failed 0
Passed 21628
Skipped 1549
Total 23177

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@belimawr belimawr force-pushed the mergify/bp/7.17/pr-30717 branch from 7a5211c to f69cab2 Compare March 24, 2022 18:58
@belimawr belimawr merged commit 0e69326 into 7.17 Mar 25, 2022
@belimawr belimawr deleted the mergify/bp/7.17/pr-30717 branch March 25, 2022 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@belimawr belimawr belimawr approved these changes

Assignees

@belimawr belimawr

Labels

backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elasticmachine @belimawr