Skip to content

[Metricbeat] Add billing metricset for aws module#14801

Merged
kaiyan-sheng merged 8 commits intoelastic:masterfrom
kaiyan-sheng:add_billing
Dec 6, 2019
Merged

[Metricbeat] Add billing metricset for aws module#14801
kaiyan-sheng merged 8 commits intoelastic:masterfrom
kaiyan-sheng:add_billing

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Nov 26, 2019 •
edited
Loading

AWS stores estimate billing charges in Cloudwatch us-east-1 region for users to see estimate charges for total, or by each service. The values are estimates, not predictions. The estimate approximates the cost of AWS usage to date within the current billing cycle and will increase as users continue to consume resources.

Screen Shot 2019-11-26 at 9 00 33 AM

This will be a good use case for alerting so users get alerted when AWS charges are higher than expected

One overview dashboard is built for this metricset:
localhost_5601_app_kibana (1)

How to test this

Enable aws module and change aws.yml to:

- module: aws
  period: 12h
  metricsets:
    - billing
  credential_profile_name: test-mb

This takes a while because estimated billing data are sent to Cloudwatch every several hours.

closes #14934

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner November 26, 2019 19:46
@kaiyan-sheng kaiyan-sheng self-assigned this Nov 26, 2019
@kaiyan-sheng kaiyan-sheng added Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan [zube]: In Review review labels Dec 2, 2019
@mtojek
Copy link
Copy Markdown
Contributor

mtojek commented Dec 3, 2019

Out of curiosity - can I easily jump to another region just by clicking the dashboard?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Dec 3, 2019

Out of curiosity - can I easily jump to another region just by clicking the dashboard?

@mtojek Great question, usually in other aws metricsets, I have a filter for regions so user can jump to different regions by selecting it using the filter. In this case for the estimated billing data, there is no need to add the region filter. Because AWS stores estimate billing charges only in us-east-1 region. For this metricset, it's limited to only query cloudwatch for us-east-1 region for the same reason: https://github.com/elastic/beats/pull/14801/files#diff-e1445680f9751a62f6ede3036549b515R7

fearful-symmetry
fearful-symmetry reviewed Dec 3, 2019
View reviewed changes
x-pack/metricbeat/module/aws/billing/_meta/fields.yml
description: >
`billing` contains the estimated charges for your AWS account in Cloudwatch.
release: beta
fields:
Copy link
Copy Markdown
Contributor

@fearful-symmetry fearful-symmetry Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be empty?

Copy link
Copy Markdown
Contributor Author

@kaiyan-sheng kaiyan-sheng Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I believe so. Because this metricset is created using light module with name aws.metrics.EstimatedCharges.max so it's not gonna be aws.billing.*.

@kaiyan-sheng kaiyan-sheng merged commit bbb8f1f into elastic:master Dec 6, 2019
@kaiyan-sheng kaiyan-sheng deleted the add_billing branch December 6, 2019 14:03
@kaiyan-sheng kaiyan-sheng mentioned this pull request Dec 6, 2019
Merged
@kaiyan-sheng kaiyan-sheng added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 6, 2019
kaiyan-sheng pushed a commit that referenced this pull request Dec 6, 2019
Cherry-pick #14801 to 7.x: [Metricbeat] Add billing metricset for aws…
cf4411c 
… module (#14986)

* [Metricbeat] Add billing metricset for aws module (#14801)

* Add billing metricset for aws module

* add dashboard

* Update changelog

(cherry picked from commit bbb8f1f)

* fix cherrypick
@ChrsMark ChrsMark self-assigned this Jan 16, 2020
@ChrsMark
Copy link
Copy Markdown
Member

ChrsMark commented Jan 17, 2020 •
edited
Loading

BC1: Tested manually and it seems that aws.dimensions.ServiceName : * query of Total Estimated Charges [Metricbeat AWS] visualization is wrong. cc: @kaiyan-sheng

@ChrsMark ChrsMark added the test-plan-regression Manually testing this PR found a regression label Jan 17, 2020
This was referenced Jan 17, 2020
Merged
Merged
Merged
@ChrsMark ChrsMark mentioned this pull request Jan 30, 2020
Merged
5 tasks
@ChrsMark
Copy link
Copy Markdown
Member

ChrsMark commented Jan 30, 2020

Tested with BC4. Works perfect @kaiyan-sheng !

@ChrsMark ChrsMark added test-plan-ok This PR passed manual testing and removed test-plan-regression Manually testing this PR found a regression labels Jan 30, 2020
@cxadmin
Copy link
Copy Markdown

cxadmin commented Aug 5, 2020

Hellow.
Is it possible to monitor multiple billing accounts?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Aug 5, 2020

Hellow.
Is it possible to monitor multiple billing accounts?

Hi @cxadmin, this billing metricset for now is only getting the estimated charges from CloudWatch. If you want to get multiple billing accounts, you can enable aws module with different sets of account credentials. For example:

- module: aws
  period: 12h
  credential_profile_name: test-account-1
  metricsets:
    - billing
- module: aws
  period: 12h
  credential_profile_name: test-account-2
  metricsets:
    - billing
- module: aws
  period: 12h
  credential_profile_name: test-account-3
  metricsets:
    - billing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fearful-symmetry fearful-symmetry fearful-symmetry left review comments

+1 more reviewer

@mtojek mtojek mtojek approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ChrsMark ChrsMark

Labels

Metricbeat Metricbeat review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan test-plan-ok This PR passed manual testing v7.6.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Metricbeat] AWS Module: billing metricset

6 participants

@kaiyan-sheng @mtojek @ChrsMark @cxadmin @fearful-symmetry @andresrc