Don't include full ES index template in errors by andrewkroh · Pull Request #25743 · elastic/beats

andrewkroh · 2021-05-17T19:32:28Z

What does this PR do?

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use <beatname> export template --es.version=1.2.3.

Fixes #25540

Why is it important?

~1MB log lines are difficult to work with.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Fixes libbeat 7.12.1 default logging logs excessive amount of data when failing to create index template #25540

elasticmachine · 2021-05-17T19:32:55Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-05-17T19:35:26Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-05-17T20:43:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: andrewkroh commented: run tests
Start Time: 2021-06-14T18:07:52.435+0000
Duration: 64 min 18 sec
Commit: b1865f9

Test stats 🧪

Test	Results
Failed	0
Passed	243
Skipped	24
Total	267

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	243
Skipped	24
Total	267

mergify · 2021-05-19T17:42:19Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b bugfix/libbeat/omit-index-template-from-err upstream/bugfix/libbeat/omit-index-template-from-err
git merge upstream/master
git push upstream bugfix/libbeat/omit-index-template-from-err

ph

lgtm, nice addition

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`. Fixes elastic#25540

mergify · 2021-06-14T12:29:58Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b bugfix/libbeat/omit-index-template-from-err upstream/bugfix/libbeat/omit-index-template-from-err
git merge upstream/master
git push upstream bugfix/libbeat/omit-index-template-from-err

andrewkroh · 2021-06-14T18:02:59Z

The build was green, but uploading failed.

[2021-06-14T15:42:14.912Z] + gsutil -m -q cp -a public-read XX gs://beats-ci-temp/ci/cache/
[2021-06-14T15:42:16.354Z] ServiceException: 401 Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.
[2021-06-14T15:42:16.354Z] CommandException: 1 file/object could not be transferred.
script returned exit code 1

andrewkroh · 2021-06-14T18:07:39Z

run tests

* Don't include full ES index template in errors Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`. Fixes #25540 * Update expected log messages in tests (cherry picked from commit 766e303) # Conflicts: # libbeat/template/load.go

* Don't include full ES index template in errors Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`. Fixes elastic#25540 * Update expected log messages in tests

* master: (25 commits) Fix UBI source URL (elastic#26384) Skip test_rotating_file in osx and windows (elastic#26379) Remove outdated k8s manifests for managed elastic-agent (elastic#26368) Enable agent to send custom headers to kibana/ES (elastic#26275) [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354) Make the Syslog input GA (elastic#26293) Move Kerberos FAST config flag to shared kerberos config (elastic#26141) Add k8s cluster identifiers (elastic#26056) Store message from MongoDB json logs in message field (elastic#26338) update threatintel ECS version (elastic#26274) update envoyproxy ECS version (elastic#26277) [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774) Update go-structform to 0.0.9 (elastic#26251) Forward port 7.13.2 changelog to master (elastic#26323) Updated filter expression for filtering 86 artifacts (elastic#26313) Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324) [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999) Change link to snapshots in README (elastic#26317) Don't include full ES index template in errors (elastic#25743) First refactor of the system module - system/cpu and system/core (elastic#25771) ...

…#26318) * Don't include full ES index template in errors (#25743) Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`. Fixes #25540 * Update expected log messages in tests (cherry picked from commit 766e303) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 17, 2021

andrewkroh added bug libbeat Team:Elastic-Agent Label for the Agent team labels May 17, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 17, 2021

andrewkroh added the review label May 17, 2021

andrewkroh force-pushed the bugfix/libbeat/omit-index-template-from-err branch from 7ac6e07 to 54cbd0a Compare May 17, 2021 19:34

andrewkroh added the Team:Security-External Integrations label May 17, 2021

andrewkroh requested a review from a team June 9, 2021 23:53

ph approved these changes Jun 10, 2021

View reviewed changes

andrewkroh added 2 commits June 14, 2021 08:26

Update expected log messages in tests

017d150

andrewkroh force-pushed the bugfix/libbeat/omit-index-template-from-err branch from 5e2d42e to 017d150 Compare June 14, 2021 12:26

Merge branch 'master' into bugfix/libbeat/omit-index-template-from-err

b1865f9

andrewkroh added the backport-v7.14.0 Automated backport with mergify label Jun 14, 2021

andrewkroh merged commit 766e303 into elastic:master Jun 15, 2021

mergify bot mentioned this pull request Jun 15, 2021

[7.x](backport #25743) Don't include full ES index template in errors #26318

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't include full ES index template in errors#25743

Don't include full ES index template in errors#25743
andrewkroh merged 3 commits intoelastic:masterfrom
andrewkroh:bugfix/libbeat/omit-index-template-from-err

andrewkroh commented May 17, 2021 •

edited

Loading

Uh oh!

elasticmachine commented May 17, 2021

Uh oh!

elasticmachine commented May 17, 2021

Uh oh!

elasticmachine commented May 17, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

mergify bot commented May 19, 2021

Uh oh!

ph left a comment

Uh oh!

mergify bot commented Jun 14, 2021

Uh oh!

andrewkroh commented Jun 14, 2021 •

edited

Loading

Uh oh!

andrewkroh commented Jun 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andrewkroh commented May 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

elasticmachine commented May 17, 2021

Uh oh!

elasticmachine commented May 17, 2021

Uh oh!

elasticmachine commented May 17, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

mergify bot commented May 19, 2021

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jun 14, 2021

Uh oh!

andrewkroh commented Jun 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewkroh commented Jun 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewkroh commented May 17, 2021 •

edited

Loading

elasticmachine commented May 17, 2021 •

edited by jenkins-beats-ci bot

Loading

andrewkroh commented Jun 14, 2021 •

edited

Loading