Skip to content

[libbeat] Fix add_labels flattening of arrays values#29211

Merged
andrewkroh merged 2 commits intoelastic:masterfrom
andrewkroh:bugfix/libbeat/add-labels-flatten
Dec 1, 2021
Merged

[libbeat] Fix add_labels flattening of arrays values#29211
andrewkroh merged 2 commits intoelastic:masterfrom
andrewkroh:bugfix/libbeat/add-labels-flatten

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Dec 1, 2021

What does this PR do?

The processor was not working as described in the docs. For example:

# Input config
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
# Observed
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
# Expected
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}

Why is it important?

The generated values to not align with the docs or how labels is described in ECS.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 1, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 1, 2021

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Dec 1, 2021
@andrewkroh
Fix add_labels flattening of arrays values
4a62f82 
The processor was not working as described in the docs. For example:

```yaml
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
```

```json
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
```

```json
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}
```
@andrewkroh andrewkroh force-pushed the bugfix/libbeat/add-labels-flatten branch from e1dfc55 to 4a62f82 Compare December 1, 2021 00:10
@andrewkroh andrewkroh added backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify labels Dec 1, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 1, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-12-01T00:16:15.552+0000

  • Duration: 102 min 26 sec

  • Commit: 14ad7cd

Test stats 🧪

Test Results
Failed 0
Passed 48632
Skipped 4286
Total 52918

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@andrewkroh andrewkroh merged commit 1dd9714 into elastic:master Dec 1, 2021
mergify bot pushed a commit that referenced this pull request Dec 1, 2021
@andrewkroh
[libbeat] Fix add_labels flattening of arrays values (#29211)
9c2e181 
* Fix add_labels flattening of arrays values

The processor was not working as described in the docs. For example:

```yaml
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
```

```json
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
```

```json
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}
```

* Expose ucfg FlattenedKeys() in common.Config

(cherry picked from commit 1dd9714)
@mergify mergify bot mentioned this pull request Dec 1, 2021
Merged
mergify bot pushed a commit that referenced this pull request Dec 1, 2021
@andrewkroh
[libbeat] Fix add_labels flattening of arrays values (#29211)
660053c 
* Fix add_labels flattening of arrays values

The processor was not working as described in the docs. For example:

```yaml
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
```

```json
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
```

```json
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}
```

* Expose ucfg FlattenedKeys() in common.Config

(cherry picked from commit 1dd9714)
@mergify mergify bot mentioned this pull request Dec 1, 2021
Merged
elasticmachine pushed a commit to nxei/beats that referenced this pull request Dec 1, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into github-actions-build
99c50fb 
* upstream/master:
  [libbeat] Fix add_labels flattening of arrays values (elastic#29211)
  Change elastic-agent pprof default to false (elastic#29155)
  elastic#28472 fix flaky tests in libbeat fmtstr to use time.UTC instead of time.Local (elastic#28473)
  Adopt `parsers` in Filebeat's journald input (elastic#29070)
  [Elastic Agent] Add process error handling guidelines (elastic#29152)
  winlogbeat/sys/winevent: use reflect IsZero method (elastic#29190)
  Remove Journalbeat (elastic#29131)
  Add note that there is no warranty or support for generator code (elastic#28797)
  packetbeat: preparation for npcap addition (elastic#29017)
  Use the generic helper for opening file to read in filestream (elastic#29180)
  Workflow for macos (elastic#29174)
  Fix `decode_json_fields` processor to always add error key (elastic#29107)
andrewkroh added a commit that referenced this pull request Dec 1, 2021
@mergify @andrewkroh
[libbeat] Fix add_labels flattening of arrays values (#29211) (#29213)
66cbe0e 
* Fix add_labels flattening of arrays values

The processor was not working as described in the docs. For example:

```yaml
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
```

```json
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
```

```json
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}
```

* Expose ucfg FlattenedKeys() in common.Config

(cherry picked from commit 1dd9714)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
andrewkroh added a commit that referenced this pull request Dec 1, 2021
@mergify @andrewkroh
[libbeat] Fix add_labels flattening of arrays values (#29211) (#29212)
0a1289e 
* Fix add_labels flattening of arrays values

The processor was not working as described in the docs. For example:

```yaml
processors:
  - add_labels:
      labels:
        number: 1
        with.dots: test
        nested:
          with.dots: nested
        array:
          - do
          - re
          - with.field: mi
```

```json
{
  "labels": {
    "array": [
      "do",
      "re",
      {
        "with": {
          "field": "mi"
        }
      }
    ],
    "nested.with.dots": "nested",
    "number": 1,
    "with.dots": "test"
  }
```

```json
{
  "labels": {
    "number": 1,
    "with.dots": "test",
    "nested.with.dots": "nested",
    "array.0": "do",
    "array.1": "re",
    "array.2.with.field": "mi"
  }
}
```

* Expose ucfg FlattenedKeys() in common.Config

(cherry picked from commit 1dd9714)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

Assignees

No one assigned

Labels

backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify bug libbeat :Processors Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewkroh @elasticmachine @efd6