Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 by tommyers-elastic · Pull Request #8041 · elastic/integrations

tommyers-elastic · 2023-10-02T12:25:55Z

This PR updates all obs-cloud-monitoring owned packages, except azure and azure_metrics, to package spec v3.

These changes are mostly autogenerated in three stages.

Firstly we run elastic-package format on everything. This commit can be excluded from review to make the diff cleaner.
Then the following yq commands are run across all the affected packages:

find $1/data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find $1/data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml

Next the ecs-update tool is run as follows:

go run github.com/andrewkroh/go-examples/ecs-update@main -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-cloudnative-monitoring packages/*

Any remaining issues were fixed manually. Even though most of the changes here were generated by tooling, this still needs careful review (hide whitespace changes to make your life easier).

There should be no functional changes to any of the packages here.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

…ckages

…ed packages

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/aws_logs

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfargate

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfirehose

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_application_insights

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_billing

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/gcp_metrics

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_metrics

packages/azure/data_stream/activitylogs/fields/ecs.yml

packages/azure_metrics/data_stream/container_instance/fields/package-fields.yml

packages/azure_metrics/data_stream/container_registry/fields/agent.yml

packages/azure_metrics/data_stream/storage_account/fields/agent.yml

elasticmachine · 2023-10-02T13:20:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-13T15:44:02.024+0000
Duration: 25 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	37
Skipped	0
Total	37

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-10-02T13:20:20Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	66.667% (`14/21`)	👍 3.252
Lines	100.0% (`135/135`)	💚 0.137
Conditionals	100.0% (`0/0`)	💚

zmoog

LGTM.

@tommyers-elastic, do you plan to address the platformlogs' _dev/test/pipeline/test-platformlogs-invalid-raw.log malformed JSON issues on in a different PR?

IIRC, the test fails on stack versions > 8.6 because the json processor in Elasticsearch has become stricter by default starting from 8.7.

More on this issue at zmoog/public-notes#54

tommyers-elastic · 2023-10-05T09:00:11Z

hey @zmoog yeh i have another PR just for azure because it contains quite a few more changes. i haven't addressed the malformed log issue there yet, but i will get round to it soon.

zmoog · 2023-10-05T11:48:09Z

@tommyers-elastic, I mentioned it so we do not forget it. Happy to help with a review!

packages/awsfirehose/data_stream/logs/routing_rules.yml

…n yaml files.

packages/awsfargate/data_stream/task_stats/fields/fields.yml

…compatibility

...s/azure_application_insights/data_stream/app_state/elasticsearch/ingest_pipeline/default.yml

…o this PR by mistake

elasticmachine · 2023-10-16T08:39:18Z

Package aws_logs - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=aws_logs

elasticmachine · 2023-10-16T08:45:58Z

Package awsfargate - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=awsfargate

elasticmachine · 2023-10-16T08:56:01Z

Package awsfirehose - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=awsfirehose

elasticmachine · 2023-10-16T08:56:03Z

Package azure_application_insights - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=azure_application_insights

elasticmachine · 2023-10-16T09:00:49Z

Package azure_billing - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_billing

elasticmachine · 2023-10-16T09:10:21Z

Package gcp_metrics - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=gcp_metrics

tommyers-elastic added 10 commits October 2, 2023 11:41

run 'elastic-package format' across all obs-cloud-monitoring owned pa…

7157a85

…ckages

remove duplicate and unsupported fields from obs-cloud-monitoring own…

28e0f77

…ed packages

additional manual fixes for 'azure_metrics' package

ba71602

tommyers-elastic added the enhancement New feature or request label Oct 2, 2023

tommyers-elastic requested a review from a team as a code owner October 2, 2023 12:25

update PR number on affected packages

61893bc

tommyers-elastic commented Oct 2, 2023

View reviewed changes

packages/azure/data_stream/activitylogs/fields/ecs.yml Show resolved Hide resolved

tommyers-elastic commented Oct 2, 2023

View reviewed changes

packages/azure_metrics/data_stream/container_instance/fields/package-fields.yml Show resolved Hide resolved

tommyers-elastic commented Oct 2, 2023

View reviewed changes

packages/azure_metrics/data_stream/container_registry/fields/agent.yml Show resolved Hide resolved

tommyers-elastic commented Oct 2, 2023

View reviewed changes

packages/azure_metrics/data_stream/storage_account/fields/agent.yml Show resolved Hide resolved

tommyers-elastic requested a review from zmoog October 2, 2023 12:30

tommyers-elastic changed the title ~~Update (most) obs-cloud-monitoring owned packages to format version 3~~ Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 Oct 2, 2023

zmoog approved these changes Oct 5, 2023

View reviewed changes

kaiyan-sheng approved these changes Oct 9, 2023

View reviewed changes

tommyers-elastic added 2 commits October 10, 2023 09:26

Merge branch 'main' into update-cloud-packages-to-format-version-3

24a634a

revert changes to 'azure' package - these will be done in a sepearate PR

8c334db

tommyers-elastic commented Oct 10, 2023

View reviewed changes

packages/awsfirehose/data_stream/logs/routing_rules.yml Outdated Show resolved Hide resolved

tommyers-elastic added 2 commits October 10, 2023 09:48

fix weird auto-formatting introduced into embedded painless scripts i…

9215357

…n yaml files.

Merge branch 'main' into update-cloud-packages-to-format-version-3

7b76cdd

tommyers-elastic commented Oct 13, 2023

View reviewed changes

packages/awsfargate/data_stream/task_stats/fields/fields.yml Outdated Show resolved Hide resolved

tommyers-elastic mentioned this pull request Oct 13, 2023

migrate obs-cloud-monitoring owned packages to format_version: 3.0.0 #8187

Closed

22 tasks

tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch 4 times, most recently from 462fc6a to ed1b1fc Compare October 13, 2023 13:50

tommyers-elastic added 5 commits October 13, 2023 15:13

update awsfargate mappings for package-spec v3 compatibility

113fc87

update awsfirehose mappings for package-spec v3 compatibility

23d2e42

update azure_application_insights validation.yml for package-spec v3 …

4531fd5

…compatibility

update azure_billing validation.yml for package-spec v3 compatibility

0021f03

revert changes to azure_metrics

f98d4e6

tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch from ed1b1fc to f98d4e6 Compare October 13, 2023 14:13

tommyers-elastic commented Oct 13, 2023

View reviewed changes

...s/azure_application_insights/data_stream/app_state/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

tommyers-elastic added 2 commits October 13, 2023 15:44

Merge branch 'main' into update-cloud-packages-to-format-version-3

af25e66

remove ingest pipeline that seems to have somehow creeped its way int…

9b95188

…o this PR by mistake

kaiyan-sheng approved these changes Oct 13, 2023

View reviewed changes

shmsr mentioned this pull request Oct 16, 2023

Bulk package-spec v3 migration of packages for serverless #8147

Closed

tommyers-elastic merged commit c9e4188 into elastic:main Oct 16, 2023

Conversation

tommyers-elastic commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic commented Oct 5, 2023

Uh oh!

zmoog commented Oct 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

elasticmachine commented Oct 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tommyers-elastic commented Oct 2, 2023 •

edited

Loading

elasticmachine commented Oct 2, 2023 •

edited

Loading

elasticmachine commented Oct 2, 2023 •

edited

Loading