Skip to content

Add metrics collection from cost explorer into aws/billing metricset#20527

Merged
kaiyan-sheng merged 23 commits intoelastic:masterfrom
kaiyan-sheng:aws_cost_metricset
Sep 14, 2020
Merged

Add metrics collection from cost explorer into aws/billing metricset#20527
kaiyan-sheng merged 23 commits intoelastic:masterfrom
kaiyan-sheng:aws_cost_metricset

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Aug 10, 2020 •
edited
Loading

What does this PR do?

This PR is to add metrics collection from cost explorer into aws/billing metricset.

Why is it important?

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Enable aws module with ./metricbeat modules enable aws
  2. Edit aws config file modules.d/aws.yml to:
- module: aws
  period: 24h
  metricsets:
    - billing
  credential_profile_name: elastic-beats
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"
  1. Run metricbeat ./metricbeat -e
  2. You should be able to see data in billing dashboard

Screenshots

Screen Shot 2020-08-25 at 3 22 31 PM (2)

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 10, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Aug 10, 2020
@kaiyan-sheng kaiyan-sheng added in progress Pull request is currently in progress. Team:Platforms Label for the Integrations - Platforms team labels Aug 10, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 10, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 10, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20527 updated]

  • Start Time: 2020-09-14T16:26:35.959+0000

  • Duration: 65 min 52 sec

Test stats 🧪

Test Results
Failed 0
Passed 3590
Skipped 795
Total 4385

@kaiyan-sheng kaiyan-sheng changed the title Add cost metricset for aws module Add metrics collection from cost explorer into aws/billing metricset Aug 12, 2020
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review August 12, 2020 20:42
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 12, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@stevewritescode
Copy link
Copy Markdown

stevewritescode commented Aug 24, 2020

This looks like it directly addresses the use case I had in mind with the original ticket. Looks great! I'm happy to try it out after the PR is approved / merged.

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Sep 1, 2020

One concern here to discuss: With this change, metric name for billing metric EstimatedCharges from CloudWatch changed from aws.billing.metrics.EstimatedCharges.max to aws.billing.EstimatedCharges.

I could keep the old name but it doesn't align with the new metrics from Cost Explorer, such as aws.billing.UnblendedCost.amount.

@kaiyan-sheng kaiyan-sheng removed the in progress Pull request is currently in progress. label Sep 2, 2020
@kaiyan-sheng kaiyan-sheng requested a review from exekias September 4, 2020 02:43
ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Left some minors.

ChrsMark
ChrsMark approved these changes Sep 14, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kaiyan-sheng kaiyan-sheng merged commit a0f8041 into elastic:master Sep 14, 2020
@kaiyan-sheng kaiyan-sheng deleted the aws_cost_metricset branch September 14, 2020 18:45
@kaiyan-sheng kaiyan-sheng mentioned this pull request Sep 14, 2020
Merged
6 tasks
kaiyan-sheng pushed a commit that referenced this pull request Sep 15, 2020
Add metrics collection from cost explorer into aws/billing metricset (#…
0d0a8cb 
…20527) (#21080)

* Add cost metricset for aws module
* Add cost_explorer_config for billing metricset with group_by_dimension_keys and group_by_tag_keys

(cherry picked from commit a0f8041)
@jsoriano jsoriano mentioned this pull request Sep 18, 2020
Merged
jsoriano added a commit that referenced this pull request Sep 18, 2020
@jsoriano
Remove billing from the list of AWS light metricsets (#21164)
43f9bbc 
On tests, loading any metricset from the AWS module is trying
to load the billing metricset as light metricset, what fails. This
shouldn't happen after #15011, but it is probably happening
because on tests, not all metricsets are registered.

billing metricset was refactored to a native implementation
recently, in #20527.

By now, remove billing from the list so tests can be executed.
@jsoriano jsoriano mentioned this pull request Sep 18, 2020
Merged
jsoriano added a commit to jsoriano/beats that referenced this pull request Sep 18, 2020
@jsoriano
Remove billing from the list of AWS light metricsets (elastic#21164)
ec684a0 
On tests, loading any metricset from the AWS module is trying
to load the billing metricset as light metricset, what fails. This
shouldn't happen after elastic#15011, but it is probably happening
because on tests, not all metricsets are registered.

billing metricset was refactored to a native implementation
recently, in elastic#20527.

By now, remove billing from the list so tests can be executed.

(cherry picked from commit 43f9bbc)
jsoriano added a commit that referenced this pull request Sep 18, 2020
@jsoriano
Remove billing from the list of AWS light metricsets (#21164) (#21170)
91a312f 
On tests, loading any metricset from the AWS module is trying
to load the billing metricset as light metricset, what fails. This
shouldn't happen after #15011, but it is probably happening
because on tests, not all metricsets are registered.

billing metricset was refactored to a native implementation
recently, in #20527.

By now, remove billing from the list so tests can be executed.

(cherry picked from commit 43f9bbc)
v1v added a commit to v1v/beats that referenced this pull request Sep 18, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
4c7e8da 
…ne-2.0

* upstream/master: (44 commits)
  Update users.asciidoc (elastic#20802) (elastic#21108)
  Fix docker provider builder. (elastic#21118)
  [Elastic Agent] Add docker composable dynamic provider. (elastic#20842)
  Add new modules/filesets from rsa2elk for 7.10 (elastic#20820)
  Fix broken links to external websites (elastic#21061)
  [docs] typo in the command line (elastic#20799)
  [Filebeat] add panos type and sub_type (elastic#20912)
  Move the `compute_vm_scalset` to  a light metricset and map the cloud metadata (elastic#21038)
  [Filebeat] Add support for Cloudtrail digest files (elastic#21086)
  Add metrics collection from cost explorer into aws/billing metricset (elastic#20527)
  Add vendoring to Google Cloud Functions again (elastic#21070)
  [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042)
  Do not need Google credentials before using it (elastic#21072)
  [Filebeat][New Module] Zoom webhook module (elastic#20414)
  Add support for GMT timezone offset in decode_cef (elastic#20993)
  Filebeat: Fix random error on harvester close (elastic#21048)
  Add ingress controller dashboards (elastic#21052)
  Fix loggers in composable module. (elastic#21047)
  [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037)
  Add changelog. (elastic#21041)
  ...
@legoguy1000 legoguy1000 mentioned this pull request Sep 8, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

@exekias exekias Awaiting requested review from exekias

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

review Team:Platforms Label for the Integrations - Platforms team v7.10.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add AWS Cost and Usage reports to Metricbeat

4 participants

@kaiyan-sheng @elasticmachine @stevewritescode @ChrsMark