Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs by matschaffer · Pull Request #30440 · elastic/beats

matschaffer · 2022-02-17T05:20:18Z

What does this PR do?

Adds a "drop" to the elasticsearch pipeline as well as explicit "mixed" test logs to ensure we won't attempt to ingest logs across mismatched pipelines.

Why is it important?

Without it, the elasticsearch slowlog pipeline will attempt to ingest all 5 file sets shipped by the elasticsearch filebeat module.

Additionally the test cases help guard against removal of the drop processors.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

See https://www.elastic.co/guide/en/beats/devguide/master/filebeat-modules-devguide.html#_test for setup instructions. Specify TESTING_FILEBEAT_MODULES=elasticsearch to test the elasticsearch module directly.

Related issues

Fixes #30428
Related #30164
Related #16540

Use cases

In Elasticsearch log4j2.properties config is defined that server, deprecation, slowlog and audit logs are written to Console:

grep -R "Console" log4j2.properties
appender.rolling.type = Console
appender.deprecation_rolling.type = Console
appender.index_search_slowlog_rolling.type = Console
appender.index_indexing_slowlog_rolling.type = Console
appender.audit_rolling.type = Console

On the kubernetes node, where elasticsearch container is running, all those logs will be in one file:

/var/log/containers/__elasticsearch-.log

Filebeat pod has the whole folder /var/log mounted and reads log files from /var/log/containers/.

The elasticsearch module has 5 filesets which will lead to reading the kubernetes log 5 times and shipping every message to each pipeline.

This works around the issue of duplicate log storage by dropping at the top of the pipeline.

This pipeline already contained a drop to avoid duplicate logging.

This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging.

test-audit-docker.log also contains a case but it was overlooked in the expected file until elastic#30164 added the appropriate drop statements.

elasticmachine · 2022-02-17T05:20:21Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2022-02-17T05:20:21Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2022-02-17T07:04:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-21T01:30:45.265+0000
Duration: 99 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8673
Skipped	1122
Total	9795

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

matschaffer · 2022-02-17T07:04:46Z

/test

matschaffer · 2022-02-17T07:15:04Z

The docs failure seems to be unrelated:

14:33:31 INFO:build_docs:To github.com:elastic/built-docs
14:33:31 INFO:build_docs: ! [rejected]                  beats_30440 -> beats_30440 (fetch first)

If the above test doesn't fix it, I'll merge main on Monday.

dedemorton · 2022-02-17T07:23:43Z

@elasticmachine run elasticsearch-ci/docs

klacabane

Nice set of tests :)

tetianakravchenko

Thank you for adding tests!
I've tried those pipeline adjustment as well as audit pipeline changes from #30164 on k8s environment - all works, no duplication

mergify · 2022-02-17T14:02:04Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 30428-drop-non-matching-es-logs upstream/30428-drop-non-matching-es-logs
git merge upstream/main
git push upstream 30428-drop-non-matching-es-logs

…hing-es-logs

…h logs (#30440) * Ensure we drop server logs that show up in deprecation pipeline * Add note about deprecation dataset normalization * Add test for mixed es server logs This pipeline already contained a drop to avoid duplicate logging. * Ensure we drop server logs that show up in slowlog pipeline This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging. * Add "mixed" test for elasticsearch audit logs test-audit-docker.log also contains a case but it was overlooked in the expected file until #30164 added the appropriate drop statements. * Changelog entry * Remove duplicatd filebeat header (cherry picked from commit 7b67384)

…nd-k8s-env * upstream/main: fix typos and improve sentences (elastic#30432) Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (elastic#30440) {,x-pack/}auditbeat: replace uses of github.com/pkg/errors with stdlib equivalents (elastic#30321) Spelling fix (elastic#30439) packetbeat/beater: make sure Npcap installation runs before interfaces are needed in all cases (elastic#30438) Add BC about Homebrew no longer being available in 8.0 (elastic#30419) Install gawk as a replacement for mawk in Docker containers. (elastic#30452) Clean up python-related system tests (elastic#30415) Fix TestNewModuleRegistry flakiness (elastic#30453) [Filebeat] [auditd]: Support EXECVE events with truncated argument list (elastic#30382) Set `log.offset` to the start of the reported line in filestream (elastic#30445) clarify SelectedPackageTypes meaning and improve its usage (elastic#30142) [elasticsearch module] serialize shards properties (elastic#30408) Add docs about hints and templates autodiscovery priority (elastic#30343)

… ingest of elasticsearch logs (#30487) Co-authored-by: Mat Schaffer <mat@elastic.co>

…h logs (#30440) (#30488) * Ensure we drop server logs that show up in deprecation pipeline * Add note about deprecation dataset normalization * Add test for mixed es server logs This pipeline already contained a drop to avoid duplicate logging. * Ensure we drop server logs that show up in slowlog pipeline This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging. * Add "mixed" test for elasticsearch audit logs test-audit-docker.log also contains a case but it was overlooked in the expected file until #30164 added the appropriate drop statements. * Changelog entry * Remove duplicatd filebeat header (cherry picked from commit 7b67384) Co-authored-by: Mat Schaffer <mat@elastic.co>

…ckaging-docker * upstream/main: (26 commits) Update docker/distribution to 2.8.0 (elastic#30462) Add `parsers` examples to `filestream` reference configuration (elastic#30529) extend documentation about setting orchestrator.cluster fields (elastic#30518) Forward-port 8.0.1 changelog to main (elastic#30522) Switch skip to use `CI` (elastic#30512) packetbeat/beater: don't attempt to install npcap when already installed (elastic#30509) Fix Docker module: rename fields on dashboards (elastic#30500) fix typos and improve sentences (elastic#30432) Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (elastic#30440) {,x-pack/}auditbeat: replace uses of github.com/pkg/errors with stdlib equivalents (elastic#30321) Spelling fix (elastic#30439) packetbeat/beater: make sure Npcap installation runs before interfaces are needed in all cases (elastic#30438) Add BC about Homebrew no longer being available in 8.0 (elastic#30419) Install gawk as a replacement for mawk in Docker containers. (elastic#30452) Clean up python-related system tests (elastic#30415) Fix TestNewModuleRegistry flakiness (elastic#30453) [Filebeat] [auditd]: Support EXECVE events with truncated argument list (elastic#30382) Set `log.offset` to the start of the reported line in filestream (elastic#30445) clarify SelectedPackageTypes meaning and improve its usage (elastic#30142) [elasticsearch module] serialize shards properties (elastic#30408) ...

…-29710 * '8.1' of github.com:elastic/beats: (51 commits) refactor pushDockerImages (#30414) (#30624) ci: add windows-2022 in the extended meta-stage (#30528) (#30630) Curate k8s testing versions to only keep the actively maintained (#30619) (#30625) [8.1](backport #30355) Add Beats upgrade docs for 8.0 (#30612) Remove references to gcp from the Functionbeat docs (#30579) (#30609) x-pack/auditbeat/module/system/socket: defend against exec with zero arguments (#30586) (#30597) [MySQL Enterprise] Adding default paths values to manifest.yml (#30598) (#30604) metricbeat - fix elasticsearch and kibana integration tests failures in 8.0 (#30566) (#30594) Install gawk as a replacement for mawk in Docker containers. (#30452) (#30465) [Filebeat] Remove RecordedFuture dataset from Threat Intel module (#30564) (#30568) Adjust the documentation of `backoff` options in filestream input (#30552) (#30557) packetbeat/beater: help the GC clean up the Npcap installer if it's not used (#30513) (#30546) Osquerybeat: Add install verification for osquerybeat (#30388) (#30404) Update docker/distribution to 2.8.0 (#30462) (#30540) Add `parsers` examples to `filestream` reference configuration (#30529) (#30537) [8.1](backport #30068) ZooKeeper module: Adapt to ZooKeeper 3.6+ `mntr` response fields' changes. (#30360) [8.1](backport #30512) Switch skip to use `CI` (#30525) Forward-port 8.0.1 changelog to 8.1 (#30517) packetbeat/beater: don't attempt to install npcap when already installed (#30509) (#30511) Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (#30440) (#30488) ...

matschaffer added 5 commits February 17, 2022 14:11

Ensure we drop server logs that show up in deprecation pipeline

485fedf

Add note about deprecation dataset normalization

22f2bc7

Add test for mixed es server logs

9c1acc1

This pipeline already contained a drop to avoid duplicate logging.

Ensure we drop server logs that show up in slowlog pipeline

ae2a682

This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging.

Add "mixed" test for elasticsearch audit logs

537d51a

test-audit-docker.log also contains a case but it was overlooked in the expected file until elastic#30164 added the appropriate drop statements.

matschaffer self-assigned this Feb 17, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 17, 2022

Changelog entry

d50579e

matschaffer requested review from klacabane, neptunian, pgomulka, ruflin and tetianakravchenko February 17, 2022 05:24

klacabane approved these changes Feb 17, 2022

View reviewed changes

tetianakravchenko approved these changes Feb 17, 2022

View reviewed changes

matschaffer added 2 commits February 21, 2022 10:29

Merge remote-tracking branch 'upstream/main' into 30428-drop-non-matc…

a89209d

…hing-es-logs

Remove duplicatd filebeat header

0bbab8d

matschaffer merged commit 7b67384 into elastic:main Feb 21, 2022

This was referenced Feb 21, 2022

[8.0](backport #30440) Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs #30487

Merged

[8.1](backport #30440) Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs #30488

Merged

matschaffer deleted the 30428-drop-non-matching-es-logs branch February 21, 2022 04:18

matschaffer added a commit that referenced this pull request Feb 21, 2022

[8.0](backport #30440) Add drop and explicit tests to avoid duplicate…

de30c54

… ingest of elasticsearch logs (#30487) Co-authored-by: Mat Schaffer <mat@elastic.co>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs#30440

Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs#30440
matschaffer merged 8 commits intoelastic:mainfrom
matschaffer:30428-drop-non-matching-es-logs

matschaffer commented Feb 17, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Feb 17, 2022

Uh oh!

elasticmachine commented Feb 17, 2022

Uh oh!

elasticmachine commented Feb 17, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

matschaffer commented Feb 17, 2022

Uh oh!

matschaffer commented Feb 17, 2022

Uh oh!

dedemorton commented Feb 17, 2022

Uh oh!

klacabane left a comment

Uh oh!

tetianakravchenko left a comment

Uh oh!

mergify bot commented Feb 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

matschaffer commented Feb 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Use cases

Uh oh!

elasticmachine commented Feb 17, 2022

Uh oh!

elasticmachine commented Feb 17, 2022

Uh oh!

elasticmachine commented Feb 17, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

matschaffer commented Feb 17, 2022

Uh oh!

matschaffer commented Feb 17, 2022

Uh oh!

dedemorton commented Feb 17, 2022

Uh oh!

klacabane left a comment

Choose a reason for hiding this comment

Uh oh!

tetianakravchenko left a comment

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Feb 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

matschaffer commented Feb 17, 2022 •

edited

Loading

elasticmachine commented Feb 17, 2022 •

edited by jenkins-beats-ci bot

Loading