Add cloud provider metadata by mikker · Pull Request #871 · elastic/apm-agent-ruby

mikker · 2020-09-15T14:04:33Z

ghost · 2020-09-15T14:42:40Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #871 updated]
Start Time: 2020-10-07T10:10:56.850+0000
Duration: 25 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	67565
Skipped	129
Total	67694

Steps errors

Expand to view the steps failures

Name: Shell Script
- Description: ./spec/scripts/spec.sh jruby:9.2 grape-1.3
- Duration: 4 min 36 sec
- Start Time: 2020-10-07T10:16:57.355+0000
- log
Name: Shell Script
- Description: [2020-10-07T10:18:12.398Z] + ./spec/scripts/spec.sh docker.elastic.co/observability-ci/jruby:9.2-13-
- Duration: 7 min 30 sec
- Start Time: 2020-10-07T10:18:12.096+0000
- log
Name: Shell Script
- Description: [2020-10-07T10:17:54.028Z] + ./spec/scripts/spec.sh docker.elastic.co/observability-ci/jruby:9.2-8-j
- Duration: 7 min 32 sec
- Start Time: 2020-10-07T10:17:53.737+0000
- log
Name: Shell Script
- Description: [2020-10-07T10:17:47.030Z] + ./spec/scripts/spec.sh docker.elastic.co/observability-ci/jruby:9.2-8-j
- Duration: 4 min 34 sec
- Start Time: 2020-10-07T10:17:46.746+0000
- log
Name: Shell Script
- Description: [2020-10-07T10:17:47.063Z] + ./spec/scripts/spec.sh docker.elastic.co/observability-ci/jruby:9.2-8-j
- Duration: 8 min 54 sec
- Start Time: 2020-10-07T10:17:46.777+0000
- log

estolfo · 2020-10-05T10:41:03Z

Overall, it looks good but why do we have flat values for some fields while other agents (for ex python) and the documentation has embedded values? I might be missing something in the issue comments...

For example, this shows id and name being embedded under account. And python uses an embedded value here. But we use account_id and account_name.

mikker · 2020-10-05T12:13:04Z

Good catch! I just didn't want to add nested objects when storing them internally. I need to add the nested objects again when turning into json.

mikker · 2020-10-05T14:27:08Z

AWS confirmed:

{
  "metadata": {
    "service": {
      "name": "App",
      "environment": "development",
      "version": null,
      "agent": { "name": "ruby", "version": "3.10.1" },
      "framework": { "name": "Sinatra", "version": "2.1.0" },
      "language": { "name": "ruby", "version": "2.7.2" },
      "runtime": { "name": "ruby", "version": "2.7.2" }
    },
    "process": {
      "pid": 18346,
      "title": "/home/ec2-user/.rvm/gems/ruby-2.7.2/bin/rackup",
      "argv": []
    },
    "system": {
      "hostname": "ip-172-31-4-63.eu-west-1.compute.internal",
      "architecture": "x86_64",
      "platform": "linux",
      "kubernetes": {
        "namespace": null,
        "node": { "name": null },
        "pod": { "name": null, "uid": null }
      }
    },
    "labels": null,
    "cloud": {
      "provider": "aws",
      "account": { "id": "907075276364", "name": null },
      "availability_zone": "eu-west-1b",
      "instance": { "id": "i-0f0165f84e8d411be", "name": null },
      "machine": { "type": "t2.micro" },
      "project": { "id": null, "name": null },
      "region": "eu-west-1"
    }
  }
}

mikker · 2020-10-06T07:06:11Z

GCE:

{
  "metadata": {
    "service": {
      "name": "App",
      "environment": "development",
      "version": null,
      "agent": { "name": "ruby", "version": "3.10.1" },
      "framework": { "name": "Sinatra", "version": "2.1.0" },
      "language": { "name": "ruby", "version": "2.7.0" },
      "runtime": { "name": "ruby", "version": "2.7.0" }
    },
    "process": {
      "pid": 31623,
      "title": "/home/mikkelmalmberg/.rvm/gems/ruby-2.7.0/bin/rackup",
      "argv": []
    },
    "system": {
      "hostname": "instance-1",
      "architecture": "x86_64",
      "platform": "linux",
      "kubernetes": {
        "namespace": null,
        "node": "{:name=>nil}",
        "pod": "{:name=>nil, :uid=>nil}"
      }
    },
    "labels": null,
    "cloud": {
      "provider": "gcp",
      "account": { "id": null, "name": null },
      "availability_zone": "us-central1-a",
      "instance": { "id": "1558417203220877769", "name": "instance-1" },
      "machine": { "type": "projects/944210798271/machineTypes/e2-micro" },
      "project": { "id": "944210798271", "name": "brnbw-10er" },
      "region": "us-central1"
    }
  }
}

mikker · 2020-10-06T07:45:50Z

Azure:

{
  "metadata": {
    "service": {
      "name": "App",
      "environment": "development",
      "version": null,
      "agent": { "name": "ruby", "version": "3.10.1" },
      "framework": { "name": "Sinatra", "version": "2.1.0" },
      "language": { "name": "ruby", "version": "2.7.0" },
      "runtime": { "name": "ruby", "version": "2.7.0" }
    },
    "process": {
      "pid": 2209,
      "title": "/home/azureuser/.rvm/gems/ruby-2.7.0/bin/rackup",
      "argv": []
    },
    "system": {
      "hostname": "apm-test",
      "architecture": "x86_64",
      "platform": "linux",
      "kubernetes": {
        "namespace": null,
        "node": "{:name=>nil}",
        "pod": "{:name=>nil, :uid=>nil}"
      }
    },
    "labels": null,
    "cloud": {
      "provider": "azure",
      "account": { "id": "81cf7b1d-04ab-440e-9802-5d0fb07d31ce", "name": null },
      "availability_zone": "",
      "instance": {
        "id": "b06fe3ff-4294-4422-aefa-71bccbbf8a51",
        "name": "apm-test"
      },
      "machine": { "type": "Standard_B2s" },
      "project": { "id": null, "name": "apm-test_group" },
      "region": "westeurope"
    }
  }
}

estolfo

lgtm!

lib/elastic_apm/util.rb

estolfo · 2020-10-07T09:34:43Z

spec/elastic_apm/metadata/cloud_info_spec.rb

+          it 'tries all three' do
+            WebMock.stub_request(:get, Metadata::CloudInfo::AWS_URI).to_timeout
+            WebMock.stub_request(:get, Metadata::CloudInfo::GCP_URI).to_timeout
+            WebMock.stub_request(:get, Metadata::CloudInfo::AZURE_URI).to_timeout


cool, didn't know about the to_timeout expectation :)

Me neither! Found it when writing this PR.

mikker self-assigned this Sep 15, 2020

mikker added this to the 7.10 milestone Sep 15, 2020

mikker added the [zube]: In Progress label Sep 16, 2020

mikker marked this pull request as ready for review October 1, 2020 12:52

felixbarny linked an issue Oct 2, 2020 that may be closed by this pull request

Collect cloud identification metadata #841

Closed

mikker requested a review from estolfo October 6, 2020 12:04

mikker added [zube]: In Review and removed [zube]: In Progress labels Oct 7, 2020

estolfo approved these changes Oct 7, 2020

View reviewed changes

Mikkel Malmberg added 15 commits October 7, 2020 12:09

Add cloud provider metadata

fa8ef49

Add azure, auto

bcaef2a

Default to 'none' in test

319bded

Maybe fix specs, who knows

7b79360

..

94b60c7

Add a few MockIntake stubs

0a5d150

🔨

9e31271

Catch connection errors

678219b

Don't manually stop in Rails logger int spec

2a955af

Serialize cloud info

9729f8c

Expect 200s

241a7a0

Convert to string when truncating keyword fields

803cde8

🔨

5f25ed7

Use same JSON conversion method

d20aa48

Remove a bunch of leftover Webmock resets

6a155b2

Update changelog

4db6e37

mikker merged commit ad13072 into elastic:master Oct 7, 2020

mikker deleted the fetch-cloud-info branch October 7, 2020 11:14

zube bot added [zube]: Done and removed [zube]: In Review labels Oct 7, 2020

bmorelli25 mentioned this pull request Feb 10, 2021

docs: Add cloud_provider documentation #961

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cloud provider metadata#871

Add cloud provider metadata#871
mikker merged 16 commits intoelastic:masterfrom
mikker:fetch-cloud-info

mikker commented Sep 15, 2020 •

edited

Loading

Uh oh!

ghost commented Sep 15, 2020 •

edited by ghost

Loading

Build stats

Test stats 🧪

Uh oh!

estolfo commented Oct 5, 2020

Uh oh!

mikker commented Oct 5, 2020

Uh oh!

mikker commented Oct 5, 2020 •

edited

Loading

Uh oh!

mikker commented Oct 6, 2020 •

edited

Loading

Uh oh!

mikker commented Oct 6, 2020 •

edited

Loading

Uh oh!

estolfo left a comment

Uh oh!

Uh oh!

estolfo Oct 7, 2020

Uh oh!

mikker Oct 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mikker commented Sep 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Sep 15, 2020 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Uh oh!

estolfo commented Oct 5, 2020

Uh oh!

mikker commented Oct 5, 2020

Uh oh!

mikker commented Oct 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikker commented Oct 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikker commented Oct 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

estolfo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

estolfo Oct 7, 2020

Choose a reason for hiding this comment

Uh oh!

mikker Oct 7, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mikker commented Sep 15, 2020 •

edited

Loading

ghost commented Sep 15, 2020 •

edited by ghost

Loading

mikker commented Oct 5, 2020 •

edited

Loading

mikker commented Oct 6, 2020 •

edited

Loading

mikker commented Oct 6, 2020 •

edited

Loading