[system] - Cleanup fields.yml files by andrewkroh · Pull Request #8100 · elastic/integrations

andrewkroh · 2023-10-05T02:11:12Z

Proposed commit message

This cleans up the field definitions for the package.

Remove unnecessary attributes like required, level,format, etc.

Use external: ecs for any field that exists in ECS.

Remove description from field groups. The description does not get used by anything (not in docs, mappings, fleet).

Remove duplicate field declarations.

Replace dotted YAML key kibana.version in manifest.yml.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

[git-generate] go run github.com/andrewkroh/fydler@eddf66dada2 -fix packages/system/**/fields/*yml cd packages/system elastic-package format

packages/system/data_stream/core/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/cpu/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/diskio/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/fsstat/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/load/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/memory/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/network/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/network/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/network/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/network/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/network/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process/fields/ecs.yml:1:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process/fields/ecs.yml:14:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process/fields/ecs.yml:18:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process_summary/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process_summary/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process_summary/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process_summary/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/process_summary/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/socket_summary/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/socket_summary/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/socket_summary/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/socket_summary/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact) packages/system/data_stream/socket_summary/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)

[git-generate] cd packages/system elastic-package format

[git-generate] cd packages/system elastic-package changelog add --link '#8100' --next minor --type enhancement --description 'Modified the field definitions to reference ECS where possible and remove invalid field attributes.'

elasticmachine · 2023-10-05T02:21:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-05T09:25:30.968+0000
Duration: 16 min 0 sec

Test stats 🧪

Test	Results
Failed	0
Passed	151
Skipped	0
Total	151

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Add 'type' to workaround validation issues. See elastic/package-spec#634

elasticmachine · 2023-10-05T02:54:29Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	63.415% (`52/82`)	👎 -36.585
Lines	99.863% (`2924/2928`)	👍 7.581
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-10-05T03:08:02Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

muthu-mps · 2023-10-05T05:15:19Z

@andrewkroh - As these changes are common across integrations, Do you recommend creating separate issues to make this change for the rest of the integrations by keeping this PR as a base ?

efd6 · 2023-10-05T05:21:17Z

@muthu-mps What do you mean by "keeping this PR as a base"?

muthu-mps · 2023-10-05T05:27:27Z

@muthu-mps What do you mean by "keeping this PR as a base"?

I mean applying this changes to the rest of the integrations which is not referencing the ECS fields and having descriptions in agent.yml group fields.

efd6 · 2023-10-05T05:33:14Z

The change is for the most part entirely mechanical, so it should be easy to do the same to other packages.

packages/system/data_stream/auth/fields/base-fields.yml

packages/system/data_stream/diskio/fields/agent.yml

@timestamp

Move @timestamp to base-fields.yml

muthu-mps

LGTM!

elasticmachine · 2023-10-05T13:40:57Z

Package system - 1.41.0 containing this change is available at https://epr.elastic.co/search?package=system

andrewkroh added 5 commits October 4, 2023 21:45

system - use ecs, remove unused attributes

9c47b05

[git-generate] go run github.com/andrewkroh/fydler@eddf66dada2 -fix packages/system/**/fields/*yml cd packages/system elastic-package format

system - remove duplicate field definitions

8e0b974

system - generate readme

f7877bf

[git-generate] cd packages/system elastic-package format

system - remove dotted YAML key

63c3e3b

andrewkroh added the Integration:system System label Oct 5, 2023

system - add changelog

44e6fcf

[git-generate] cd packages/system elastic-package changelog add --link '#8100' --next minor --type enhancement --description 'Modified the field definitions to reference ECS where possible and remove invalid field attributes.'

andrewkroh added Team:Security-External Integrations Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Oct 5, 2023

system - workaround ecs with metric_type issue

a57d4ad

Add 'type' to workaround validation issues. See elastic/package-spec#634

andrewkroh marked this pull request as ready for review October 5, 2023 03:08

andrewkroh requested review from a team as code owners October 5, 2023 03:08

efd6 approved these changes Oct 5, 2023

View reviewed changes

muthu-mps reviewed Oct 5, 2023

View reviewed changes

packages/system/data_stream/auth/fields/base-fields.yml Show resolved Hide resolved

packages/system/data_stream/diskio/fields/agent.yml Show resolved Hide resolved

address review comment

44284b6

Move @timestamp to base-fields.yml

muthu-mps self-requested a review October 5, 2023 09:42

muthu-mps approved these changes Oct 5, 2023

View reviewed changes

andrewkroh merged commit bb4fbfa into elastic:main Oct 5, 2023

ishleenk17 mentioned this pull request Oct 20, 2023

Dimension setting missing in host.name in system package #8256

Closed

This was referenced Oct 24, 2023

Dimension setting added to host.name in system package #8261

Merged

[Kubernetes] Add state namespace datastream #8329

Merged

constanca-m mentioned this pull request Dec 12, 2023

[Kubernetes] Reference ECS fields and add agent.id field #8697

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[system] - Cleanup fields.yml files#8100

[system] - Cleanup fields.yml files#8100
andrewkroh merged 8 commits intoelastic:mainfrom
andrewkroh:system/feature/fields-yml-cleanup

andrewkroh commented Oct 5, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Oct 5, 2023 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Oct 5, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Oct 5, 2023

Uh oh!

muthu-mps commented Oct 5, 2023

Uh oh!

efd6 commented Oct 5, 2023

Uh oh!

muthu-mps commented Oct 5, 2023

Uh oh!

efd6 commented Oct 5, 2023

Uh oh!

Uh oh!

Uh oh!

muthu-mps left a comment

Uh oh!

elasticmachine commented Oct 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andrewkroh commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Uh oh!

elasticmachine commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

elasticmachine commented Oct 5, 2023

Uh oh!

muthu-mps commented Oct 5, 2023

Uh oh!

efd6 commented Oct 5, 2023

Uh oh!

muthu-mps commented Oct 5, 2023

Uh oh!

efd6 commented Oct 5, 2023

Uh oh!

Uh oh!

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andrewkroh commented Oct 5, 2023 •

edited

Loading

elasticmachine commented Oct 5, 2023 •

edited

Loading

elasticmachine commented Oct 5, 2023 •

edited

Loading