Fix `decode_json_fields` processor to always add error key by belimawr · Pull Request #29107 · elastic/beats

belimawr · 2021-11-23T18:22:06Z

What does this PR do?

When the decode_json_fields processor encountered an error while
decoding the JSON it was not always respecting the add_error_key
configuration.

This commit fixes it.

Why is it important?

Errors from decode_json_fields processor are now clearly reported

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Add the following to you filebeat configuration:

processors:
  - decode_json_fields:
      fields: ["my_json_key"]
      process_array: false
      max_depth: 1
      target: "json_object"
      add_error_key: true

Make sure the my_json_key field of the event is an invalid JSON.

Related issues

Use cases

Screenshots

Logs

libbeat/processors/actions/decode_json_fields.go

elasticmachine · 2021-11-23T20:22:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-25T10:52:26.941+0000
Duration: 100 min 39 sec
Commit: c469e56

Test stats 🧪

Test	Results
Failed	0
Passed	48530
Skipped	4286
Total	52816

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2021-11-24T17:51:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-add-error-key upstream/fix-add-error-key
git merge upstream/master
git push upstream fix-add-error-key

When the `decode_json_fields` processor encountered an error while decoding the JSON it was not always respecting the `add_error_key` configuration. This commit fixes it.

elasticmachine · 2021-11-25T10:52:44Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

When the `decode_json_fields` processor encountered an error while decoding the JSON it was not always respecting the `add_error_key` configuration. This commit fixes it. (cherry picked from commit 37c6229)

…29178) When the `decode_json_fields` processor encountered an error while decoding the JSON it was not always respecting the `add_error_key` configuration. This commit fixes it. (cherry picked from commit 37c6229) Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…29177) When the `decode_json_fields` processor encountered an error while decoding the JSON it was not always respecting the `add_error_key` configuration. This commit fixes it. (cherry picked from commit 37c6229) Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

* upstream/master: (577 commits) Remove Journalbeat (elastic#29131) Add note that there is no warranty or support for generator code (elastic#28797) packetbeat: preparation for npcap addition (elastic#29017) Use the generic helper for opening file to read in filestream (elastic#29180) Workflow for macos (elastic#29174) Fix `decode_json_fields` processor to always add error key (elastic#29107) Workflow for macos (elastic#29156) Fix agent download timeout values in yaml files (elastic#29039) Added workflow file for builds with macos (elastic#29148) CI: enable AWS cloud testing on a PR basis if changes in certain files (elastic#29047) Remove links to Journalbeat (elastic#29134) Fix rds metadata in cloudwatch metricset (elastic#29106) [mergify]: notify conflicts in PRs that are still open (elastic#29122) Use NamedWatcher in Agent's k8s provider (elastic#29095) override host on statsd metricset (elastic#29103) Skip config check in autodiscover for duplicated configurations (elastic#29048) Change "filebeat.config.modules.enabled" to "true" (elastic#28769) Remove deprecated spool queue from Beats (elastic#28869) Add `beat` field back to beat.stats (elastic#29094) Revert "Move labels and annotations under kubernetes.namespace. (elastic#27917)" (elastic#29069) ...

* upstream/master: [libbeat] Fix add_labels flattening of arrays values (elastic#29211) Change elastic-agent pprof default to false (elastic#29155) elastic#28472 fix flaky tests in libbeat fmtstr to use time.UTC instead of time.Local (elastic#28473) Adopt `parsers` in Filebeat's journald input (elastic#29070) [Elastic Agent] Add process error handling guidelines (elastic#29152) winlogbeat/sys/winevent: use reflect IsZero method (elastic#29190) Remove Journalbeat (elastic#29131) Add note that there is no warranty or support for generator code (elastic#28797) packetbeat: preparation for npcap addition (elastic#29017) Use the generic helper for opening file to read in filestream (elastic#29180) Workflow for macos (elastic#29174) Fix `decode_json_fields` processor to always add error key (elastic#29107)

tetianakravchenko · 2022-03-30T13:11:22Z

@belimawr this PR should close this issue #27067, right?

belimawr added bug backport-v8.0.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Nov 23, 2021

belimawr requested a review from kvch November 23, 2021 18:22

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 23, 2021

belimawr commented Nov 23, 2021

View reviewed changes

libbeat/processors/actions/decode_json_fields.go Outdated Show resolved Hide resolved

belimawr added 4 commits November 25, 2021 11:51

Fix decode_json_fields processor to always add error key

6ba9d04

When the `decode_json_fields` processor encountered an error while decoding the JSON it was not always respecting the `add_error_key` configuration. This commit fixes it.

Add field to error data, input is renamed to data

e5fce6c

changelog entry

a83565c

Add test

c469e56

belimawr force-pushed the fix-add-error-key branch from 775fa1b to c469e56 Compare November 25, 2021 10:52

belimawr added the review label Nov 25, 2021

belimawr marked this pull request as ready for review November 25, 2021 10:52

kvch approved these changes Nov 25, 2021

View reviewed changes

belimawr merged commit 37c6229 into elastic:master Nov 29, 2021

belimawr deleted the fix-add-error-key branch November 29, 2021 10:31

mergify bot mentioned this pull request Nov 29, 2021

[7.16](backport #29107) Fix decode_json_fields processor to always add error key #29177

Merged

mergify bot mentioned this pull request Nov 29, 2021

[8.0](backport #29107) Fix decode_json_fields processor to always add error key #29178

Merged

kvch mentioned this pull request Mar 30, 2022

Filebeat: decode_json_fields processor ignores add_error_key option #27067

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `decode_json_fields` processor to always add error key#29107

Fix `decode_json_fields` processor to always add error key#29107
belimawr merged 4 commits intoelastic:masterfrom
belimawr:fix-add-error-key

belimawr commented Nov 23, 2021 •

edited

Loading

Uh oh!

Uh oh!

elasticmachine commented Nov 23, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

mergify bot commented Nov 24, 2021

Uh oh!

elasticmachine commented Nov 25, 2021

Uh oh!

tetianakravchenko commented Mar 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

belimawr commented Nov 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

Uh oh!

elasticmachine commented Nov 23, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

mergify bot commented Nov 24, 2021

Uh oh!

elasticmachine commented Nov 25, 2021

Uh oh!

tetianakravchenko commented Mar 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

belimawr commented Nov 23, 2021 •

edited

Loading

elasticmachine commented Nov 23, 2021 •

edited by jenkins-beats-ci bot

Loading