[Metricbeat] Change Account ID to Project ID in `gcp.billing` module by legoguy1000 · Pull Request #26412 · elastic/beats

legoguy1000 · 2021-06-22T15:08:52Z

What does this PR do?

Currently the GCP Project ID is being ingested as cloud.account.id instead of cloud.project.id. This PR fixes that.

Why is it important?

See above

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes GCP billing reports wrong account ID #26357

Use cases

Screenshots

Logs

elasticmachine · 2021-06-22T15:13:43Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: kaiyan-sheng commented: /test
Start Time: 2021-06-25T03:49:50.593+0000
Duration: 78 min 36 sec
Commit: 2352c7f

Test stats 🧪

Test	Results
Failed	0
Passed	1969
Skipped	192
Total	2161

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1969
Skipped	192
Total	2161

elasticmachine · 2021-06-23T03:05:55Z

Pinging @elastic/integrations (Team:Integrations)

kaiyan-sheng

Hi @legoguy1000 thank you for your contribution!! Do you know if we can get the account ID and account name in GCP from the given project ID? I would like to report both account ID/Name and project ID in gcp.billing.

legoguy1000 · 2021-06-23T11:55:59Z

@kaiyan-sheng based on this, https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema, I made additional changes to the query to pull the account info. I don't have a GCP account with any actual billing so I don't have a good way to validate the query besides just looking at the schema in the above link. I was able to add the project.name and what I believe is the account.id. According to the schema, there isn't a field in the table for account.name.

legoguy1000 · 2021-06-23T22:24:19Z

@kaiyan-sheng can you run the tests when you get a chance.

kaiyan-sheng

Thanks for the change. I tested it locally and here is what I saw:

2021-06-23T15:33:35.456-0600    ERROR   [billing]       billing/billing.go:242  bigquery Wait failed: googleapi: Error 400: SELECT list expression references column billing_account_id which is neither grouped nor aggregated at [6:33], invalidQuery

2021-06-23T15:54:33.736-0600    ERROR   [billing]       billing/billing.go:241  bigquery Wait failed: googleapi: Error 400: SELECT list expression references column cost_type which is neither grouped nor aggregated at [6:33], invalidQuery

I will look into it a bit more tomorrow :)

kaiyan-sheng · 2021-06-24T02:45:02Z

/test

x-pack/metricbeat/module/gcp/billing/billing.go

kaiyan-sheng

OK, adding GROUP BY and ORDER BY fixed the issue:

query := fmt.Sprintf(`
			SELECT
				invoice.month,
				project.id,
				project.name,
				billing_account_id,
				cost_type,
			  (SUM(CAST(cost * 1000000 AS int64))
				+ SUM(IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0))) / 1000000
				AS total_exact
			FROM %s
			WHERE project.id IS NOT NULL
			AND invoice.month = '%s'
			AND cost_type = '%s'
			GROUP BY 1, 2, 3, 4, 5
			ORDER BY 1 ASC, 2 ASC, 3 ASC, 4 ASC, 5 ASC;`, tableMeta.tableFullID, month, costType)

legoguy1000 · 2021-06-24T23:12:27Z

@kaiyan-sheng Query fixed. Should be good for CI tests now.

kaiyan-sheng · 2021-06-25T03:49:37Z

/test

legoguy1000 · 2021-06-25T12:18:43Z

@kaiyan-sheng Looks good to merge now.

* master: (32 commits) [Metricbeat] Change Account ID to Project ID in `gcp.billing` module (elastic#26412) update libbeat fields.ecs.yml file and ecsVersion to 1.10.0 (elastic#26121) [Filebeat] Update AWS ELB ingest pipeline (elastic#26441) [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module (elastic#26158) Fix the irregular and typo on prometheus module. (elastic#25726) [Filebeat] Parse additonal debug data fields for Okta module (elastic#25818) fix: update MSSQL Server linux image's Docker registry (elastic#26440) Update indexing.go godocs (elastic#26408) Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled (elastic#26411) Add support for copytruncate method when rotating input logs with an external tool in `filestream` input (elastic#23457) Allow fields with ip_range datatype (elastic#26444) Add Anomali ThreatStream support to threatintel module (elastic#26350) fix: use the right param type (elastic#26469) [Automation] Update elastic stack version to 8.0.0-7640093f for testing (elastic#26460) Set SM Filebeat modules as GA (elastic#26226) Fix rfc5464 date parsing in the syslog input (elastic#26419) Add linked account information into billing metricset (elastic#26285) [Filebeat] Update HA Proxy log grok patterns (elastic#25835) disable metricbeat logstash test_node_stats (elastic#26436) chore: pass BEAT_VERSION when running E2E tests (elastic#26291) ...

…26412) (#26505) (cherry picked from commit 076e0a6) Co-authored-by: Alex Resnick <adr8292@gmail.com>

elastic#26357: Change Account ID to Project ID

b88364f

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 22, 2021

Update changelog

23b51d7

legoguy1000 marked this pull request as ready for review June 22, 2021 15:13

kaiyan-sheng self-assigned this Jun 23, 2021

kaiyan-sheng added the Team:Integrations Label for the Integrations team label Jun 23, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 23, 2021

kaiyan-sheng reviewed Jun 23, 2021

View reviewed changes

query additional items

5f2f9bf

kaiyan-sheng reviewed Jun 24, 2021

View reviewed changes

legoguy1000 commented Jun 24, 2021

View reviewed changes

x-pack/metricbeat/module/gcp/billing/billing.go Outdated Show resolved Hide resolved

fix typo

b62ff80

kaiyan-sheng reviewed Jun 24, 2021

View reviewed changes

fix Big Query per comments

2352c7f

kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Jun 25, 2021

kaiyan-sheng approved these changes Jun 25, 2021

View reviewed changes

kaiyan-sheng merged commit 076e0a6 into elastic:master Jun 25, 2021

legoguy1000 deleted the 26357-gcp-billing branch June 25, 2021 15:35

kaiyan-sheng mentioned this pull request Jun 25, 2021

Cherry-pick #26412 to 7.x: [Metricbeat] Change Account ID to Project ID in gcp.billing module #26505

Merged

6 tasks

kaiyan-sheng added v7.14.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jun 25, 2021

kaiyan-sheng mentioned this pull request Jun 25, 2021

update data.json for gcp billing #26506

Merged

kaiyan-sheng pushed a commit that referenced this pull request Jun 28, 2021

[Metricbeat] Change Account ID to Project ID in gcp.billing module (#…

bedc535

…26412) (#26505) (cherry picked from commit 076e0a6) Co-authored-by: Alex Resnick <adr8292@gmail.com>

kaiyan-sheng mentioned this pull request Jun 28, 2021

Cherry-pick #26506 to 7.x: update data.json for gcp billing #26542

Merged

Conversation

legoguy1000 commented Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Jun 22, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Jun 23, 2021

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

legoguy1000 commented Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

legoguy1000 commented Jun 23, 2021

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng commented Jun 24, 2021

Uh oh!

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

legoguy1000 commented Jun 24, 2021

Uh oh!

kaiyan-sheng commented Jun 25, 2021

Uh oh!

legoguy1000 commented Jun 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

legoguy1000 commented Jun 22, 2021 •

edited

Loading

elasticmachine commented Jun 22, 2021 •

edited by jenkins-beats-ci bot

Loading

legoguy1000 commented Jun 23, 2021 •

edited

Loading