Skip to content

[linux] Avoid using dynamic templates for flattened objects#12575

Merged
mrodm merged 6 commits intoelastic:mainfrom
mrodm:update_linux_mappings
Feb 14, 2025
Merged

[linux] Avoid using dynamic templates for flattened objects#12575
mrodm merged 6 commits intoelastic:mainfrom
mrodm:update_linux_mappings

Conversation

@mrodm
Copy link
Copy Markdown
Collaborator

@mrodm mrodm commented Feb 3, 2025 •
edited
Loading

Proposed commit message

Update flattened field definitions to not create dynamic templates.

As they were previously, this dynamic template is created

  "dynamic_templates": [
    {
      "linux.memory.vmstat.*": {
        "path_match": "linux.memory.vmstat.*",
        "match_mapping_type": "object",
        "mapping": {
          "type": "flattened"
        }
      }
    },
  ],

but this definition does not match the fields under linux.memory.vmstat.

With these changes:

        "linux": {
          "properties": {
            "memory": {
              "properties": {
                "vmstat": {
                  "type": "flattened"
                }

Example of errors found (buildkite link):

test case failed: one or more errors found in mappings in metrics-linux.memory index template:
[0] field "linux.memory.vmstat.allocstall_device" is undefined: field definition not found
[1] field "linux.memory.vmstat.allocstall_dma" is undefined: field definition not found
[2] field "linux.memory.vmstat.allocstall_dma32" is undefined: field definition not found
[3] field "linux.memory.vmstat.allocstall_movable" is undefined: field definition not found
[4] field "linux.memory.vmstat.allocstall_normal" is undefined: field definition not found
...

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

How to test this PR locally

Run elastic-package with mappings validation enabled and run system tests:

cd packages/linux
elastic-package stack up -v -d --version <kibana_constraint>
export ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings
# Add defer-cleanup to check the data ingested in the corresponding indices/data streams
elastic-package test system -v --data-streams memory --defer-cleanup 900s

elastic-package stack down -v

Related issues

@mrodm mrodm self-assigned this Feb 3, 2025
@andrewkroh andrewkroh added the Integration:linux Linux Metrics label Feb 3, 2025
@andrewkroh andrewkroh added Integration:1password 1Password (Partner supported) Integration:abnormal_security Abnormal AI New Integration Issue or pull request for creating a new integration package. labels Feb 4, 2025
@mrodm mrodm force-pushed the update_linux_mappings branch from ec0a5c4 to 4b4628e Compare February 4, 2025 10:10
@mrodm mrodm removed Integration:1password 1Password (Partner supported) Integration:abnormal_security Abnormal AI New Integration Issue or pull request for creating a new integration package. labels Feb 4, 2025
@mrodm mrodm force-pushed the update_linux_mappings branch 4 times, most recently from fa4ee98 to 91c1977 Compare February 5, 2025 18:18
@mrodm mrodm force-pushed the update_linux_mappings branch from 91c1977 to 163c8ff Compare February 6, 2025 12:55
mrodm
mrodm commented Feb 6, 2025
View reviewed changes
packages/linux/data_stream/memory/fields/fields.yml
description: |
The percentage of used swap memory.
- name: vmstat.*
- name: vmstat
Copy link
Copy Markdown
Collaborator Author

@mrodm mrodm Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documents ingested would be like:

    "linux": {
      "memory": {
        "vmstat": {
          "workingset_refault_file": 2138589,
          "direct_map_level3_splits": 44,
          "direct_map_level2_splits": 1491,
          "thp_swpout_fallback": 0,
          "htlb_buddy_alloc_success": 0,
          ...
        }
      }
    }

And the content of that field is directly a JSON (as expected for flattened fields):

{
  "workingset_refault_file": 2138589,
  "direct_map_level3_splits": 44,
  "direct_map_level2_splits": 1491,
  "thp_swpout_fallback": 0,
  "htlb_buddy_alloc_success": 0,
  ...
}

@mrodm mrodm marked this pull request as ready for review February 11, 2025 10:11
@mrodm mrodm requested review from a team as code owners February 11, 2025 10:11
mrodm
mrodm commented Feb 11, 2025
View reviewed changes
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] label Feb 11, 2025
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 11, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

leehinman
leehinman approved these changes Feb 13, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about the buildkite changes but the rest of it looks fine to me.

@mrodm
Copy link
Copy Markdown
Collaborator Author

mrodm commented Feb 14, 2025

I'm not sure about the buildkite changes but the rest of it looks fine to me.

Thanks!
Buildkite changes are going to be reverted before merging

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Feb 14, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 14, 2025

💚 Build Succeeded

History

cc @mrodm

@mrodm mrodm merged commit de12c57 into elastic:main Feb 14, 2025
@mrodm mrodm deleted the update_linux_mappings branch February 14, 2025 09:38
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Feb 14, 2025

Package linux - 0.7.1 containing this change is available at https://epr.elastic.co/package/linux/0.7.1/

flexitrev pushed a commit that referenced this pull request Mar 20, 2025
@mrodm @flexitrev
[linux] Avoid using dynamic templates for flattened objects (#12575)
f9fd457 
Update flattened field definitions to not create dynamic templates
for "linux.memory.vmstat.*" fields.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman approved these changes

@khushijain21 khushijain21 Awaiting requested review from khushijain21 khushijain21 is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@mrodm mrodm

Labels

Integration:linux Linux Metrics Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrodm @elasticmachine @leehinman @pierrehilbert @andrewkroh