Move add-cloud-metadata to ECS by ruflin · Pull Request #9265 · elastic/beats

ruflin · 2018-11-28T09:30:32Z

Fields renamed:

meta.cloud.instance_id -> cloud.instance.id
meta.cloud.instance_name -> cloud.instance.name
meta.cloud.machine_type -> cloud.machine.type
meta.cloud.availability_zone -> cloud.availability_zone
meta.cloud.project_id -> cloud.project.id
meta.cloud.region -> cloud.region

Further changes:

Added alias for old fields
Update asset generation to include ECS assets in libbeat fields.yml. This is needed for testing as the aliases fields are in there.

elasticmachine · 2018-12-10T09:08:46Z

Pinging @elastic/infrastructure

webmat

Almost there. The only small problem I see is that the meta.cloud.provider has been deleted instead of being made an alias to cloud.provider (both in migration file and actual common fields).

Once that alias is back in, I'm good with this.

Oh and I discovered another name clash with ECS ;-)

webmat · 2018-12-12T18:17:17Z

libbeat/processors/add_cloud_metadata/_meta/fields.yml

meta.cloud.provider seems to be directly deleted, instead of becoming an alias to cloud.provider. Please add it back as an alias

webmat · 2018-12-12T18:18:43Z

dev-tools/ecs-migration.yml

Make sure you add the cloud.provider alias here as well

Done, not sure how that slipped through above :-(

webmat · 2018-12-12T18:43:33Z

libbeat/processors/add_cloud_metadata/provider_alibaba_cloud_test.go

Oh jeez. Another thing called ECS. After our schema and AWS' container service (yes, in that order). Moreover, Alibaba's thing is named "Elastic ..." as well.
💯 🤦‍♂️ 😆

yep, that one we knew ;-)

webmat · 2018-12-12T19:30:52Z

Restarted the single Travis test that was failing, and it passed. Likely a transient issue.

Not familiar enough with the provider tests to understand the failure in the Alibaba test:

10:44:45 ok  	github.com/elastic/beats/libbeat/outputs/transport/transptest	1.309s	coverage: 47.5% of statements
10:45:06 ok  	github.com/elastic/beats/libbeat/processors/actions	19.241s	coverage: 58.3% of statements
10:45:06 ok  	github.com/elastic/beats/libbeat/processors	19.828s	coverage: 68.3% of statements
10:45:06 command [go test -race -cover -coverprofile /tmp/gotestcover-323955738 github.com/elastic/beats/libbeat/processors/add_cloud_metadata]: exit status 1
10:45:06 --- FAIL: TestRetrieveAlibabaCloudMetadata (19.20s)
10:45:06     <autogenerated>:1: 
10:45:06         	Error Trace:	provider_alibaba_cloud_test.go:85
10:45:06         	Error:      	Not equal: 
10:45:06         	            	expected: common.MapStr{"cloud":common.MapStr{"provider":"ecs", "instance":common.MapStr{"id":"i-wz9g2hqiikg0aliyun2b"}, "region":"cn-shenzhen", "availability_zone":"cn-shenzhen-a"}}
10:45:06         	            	actual  : common.MapStr{}
10:45:06         	            	
10:45:06         	            	Diff:
10:45:06         	            	--- Expected
10:45:06         	            	+++ Actual
10:45:06         	            	@@ -1,2 +1,2 @@
10:45:06         	            	-(common.MapStr) (len=1) {"cloud":{"availability_zone":"cn-shenzhen-a","instance":{"id":"i-wz9g2hqiikg0aliyun2b"},"provider":"ecs","region":"cn-shenzhen"}}
10:45:06         	            	+(common.MapStr) {}
10:45:06         	            	 
10:45:06         	Test:       	TestRetrieveAlibabaCloudMetadata
10:45:06 FAIL
10:45:06 coverage: 85.2% of statements
10:45:06 FAIL	github.com/elastic/beats/libbeat/processors/add_cloud_metadata	19.331s
10:45:10 ok  	github.com/elastic/beats/libbeat/processors/add_host_metadata	1.025s	coverage: 83.1% of statements

Test run

ruflin · 2018-12-12T19:53:13Z

@webmat Already tried to reproduce the above failing test locally but couldn't so far. Not sure how it's possible that it fails especially as it's based on local data. Will need to further investigate.

webmat · 2018-12-12T19:57:55Z

About the failing test: is there an actual API call being performed? Or is this based on a static fixture file of the API response?

May just be a network hiccup if there's an API call. I guess we'll see after this test run.

webmat

If tests pass, this looks good to me

Fields renamed: ``` * meta.cloud.instance_id -> cloud.instance.id * meta.cloud.instance_name -> cloud.instance.name * meta.cloud.machine_type -> cloud.machine.type * meta.cloud.availability_zone -> cloud.availability_zone * meta.cloud.project_id -> cloud.project.id * meta.cloud.region -> cloud.region ``` Further changes: * Added alias for old fields * Update asset generation to include ECS assets in libbeat fields.yml. This is needed for testing as the aliases fields are in there.

ruflin added review libbeat :Processors ecs labels Nov 28, 2018

ruflin force-pushed the add_cloud_metadata-to-ecs branch from d6458ec to 6ad40c3 Compare November 28, 2018 09:33

ruflin mentioned this pull request Nov 28, 2018

Transition Beats to ECS #8655

Closed

ruflin force-pushed the add_cloud_metadata-to-ecs branch 4 times, most recently from 0b6df57 to 14d4495 Compare December 7, 2018 12:32

ruflin added the Team:Integrations Label for the Integrations team label Dec 10, 2018

ruflin self-assigned this Dec 10, 2018

ruflin force-pushed the add_cloud_metadata-to-ecs branch from 14d4495 to e81a3f4 Compare December 10, 2018 09:11

ruflin requested a review from webmat December 10, 2018 09:11

ruflin force-pushed the add_cloud_metadata-to-ecs branch 3 times, most recently from 49e274d to 7dc4ef8 Compare December 12, 2018 15:29

webmat suggested changes Dec 12, 2018

View reviewed changes

webmat approved these changes Dec 12, 2018

View reviewed changes

ruflin force-pushed the add_cloud_metadata-to-ecs branch from 5af93d1 to b7251b7 Compare December 13, 2018 08:55

ruflin merged commit 9354aa0 into elastic:master Dec 13, 2018

ruflin deleted the add_cloud_metadata-to-ecs branch December 13, 2018 11:56

exekias mentioned this pull request May 30, 2019

[libbeat]add EC2 fields to yaml for add_cloud_metadata #12335

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move add-cloud-metadata to ECS#9265

Move add-cloud-metadata to ECS#9265
ruflin merged 1 commit intoelastic:masterfrom
ruflin:add_cloud_metadata-to-ecs

ruflin commented Nov 28, 2018 •

edited

Loading

Uh oh!

elasticmachine commented Dec 10, 2018

Uh oh!

webmat left a comment •

edited

Loading

Uh oh!

webmat Dec 12, 2018

Uh oh!

ruflin Dec 12, 2018

Uh oh!

webmat Dec 12, 2018

Uh oh!

ruflin Dec 12, 2018

Uh oh!

webmat Dec 12, 2018

Uh oh!

ruflin Dec 12, 2018

Uh oh!

webmat commented Dec 12, 2018 •

edited

Loading

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

webmat commented Dec 12, 2018

Uh oh!

webmat left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ruflin commented Nov 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Dec 10, 2018

Uh oh!

webmat left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

webmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

webmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

webmat commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

webmat commented Dec 12, 2018

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ruflin commented Nov 28, 2018 •

edited

Loading

webmat left a comment •

edited

Loading

webmat commented Dec 12, 2018 •

edited

Loading