Skip to content

[gcp/billing] always quote table name identifier#26870

Merged
endorama merged 3 commits intoelastic:masterfrom
endorama:endorama/gcp-billing-quoted-identifier
Jul 16, 2021
Merged

[gcp/billing] always quote table name identifier#26870
endorama merged 3 commits intoelastic:masterfrom
endorama:endorama/gcp-billing-quoted-identifier

Conversation

@endorama
Copy link
Copy Markdown
Contributor

@endorama endorama commented Jul 13, 2021 •
edited
Loading

What does this PR do?

Extract query generation and use quoted identifiers when querying BigQuery from GCP Billing module.

Why is it important?

When users specify a table name containing special characters (not A-Z a-z _) the query would fail.

This is because the code is using an Unquoted identifier for the table name that may contain only a limited characters set.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • This change wrap the entire identifier, this is the expected usage from my understanding. Is it?

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@endorama endorama added the Team:Integrations Label for the Integrations team label Jul 13, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 13, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 13, 2021

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 13, 2021
@botelastic
Copy link
Copy Markdown

botelastic bot commented Jul 13, 2021

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 13, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-15T08:45:58.782+0000

  • Duration: 71 min 35 sec

  • Commit: e26be1c

Test stats 🧪

Test Results
Failed 0
Passed 2512
Skipped 288
Total 2800

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 2512
Skipped 288
Total 2800

@endorama
Copy link
Copy Markdown
Contributor Author

endorama commented Jul 14, 2021

@kaiyan-sheng does this change require an entry in the CHANGELOG? (Feels like yes but worth checking)

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Jul 14, 2021

@kaiyan-sheng does this change require an entry in the CHANGELOG? (Feels like yes but worth checking)

Yep I think this deserves a changelog entry under bugs.

@endorama
Copy link
Copy Markdown
Contributor Author

endorama commented Jul 14, 2021

@kaiyan-sheng May you guide me in doing that? Which file should I edit? Which version will this bugfix be part of?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Jul 14, 2021

@endorama Of course! So changelog for new features and bug fixes you think should be exposed to users will go into beats/CHANGELOG.next.asciidoc. Since this is a bug fix in Metricbeat, the changelog entry will go in here: https://github.com/elastic/beats/blame/master/CHANGELOG.next.asciidoc#L412

kaiyan-sheng
kaiyan-sheng approved these changes Jul 14, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this PR and it works with my gcp billing account. But the table name doesn't contain special characters.

@endorama
Copy link
Copy Markdown
Contributor Author

endorama commented Jul 15, 2021

@kaiyan-sheng CHANGELOG updated. I was wondering if this change should be backported to 7.x?

@kaiyan-sheng kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Jul 15, 2021
@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Jul 15, 2021

@endorama LGTM. Thanks for the fix! Yes this will need to be backported into 7.x and also 7.14 branch since it's a bug fix.

@endorama endorama merged commit f731832 into elastic:master Jul 16, 2021
@endorama endorama deleted the endorama/gcp-billing-quoted-identifier branch July 16, 2021 09:32
@endorama endorama added the backport-v7.14.0 Automated backport with mergify label Jul 16, 2021
mergify bot pushed a commit that referenced this pull request Jul 16, 2021
@endorama
[gcp/billing] always quote table name identifier (#26870)
675c477 
(cherry picked from commit f731832)
@mergify mergify bot mentioned this pull request Jul 16, 2021
Merged
@endorama endorama added the backport-v7.15.0 Automated backport with mergify label Jul 16, 2021
mergify bot pushed a commit that referenced this pull request Jul 16, 2021
@endorama
[gcp/billing] always quote table name identifier (#26870)
5e7f937 
(cherry picked from commit f731832)
@mergify mergify bot mentioned this pull request Jul 16, 2021
Merged
endorama added a commit that referenced this pull request Jul 16, 2021
@mergify @endorama
[7.14](backport #26870) [gcp/billing] always quote table name identif…
cfe6eb6 
…ier (#26931)

(cherry picked from commit f731832)

Co-authored-by: endorama <526307+endorama@users.noreply.github.com>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jul 19, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
78d203f 
* master: (61 commits)
  Add disk queue unit tests based on the queuetest package
  [Heartbeat] redact authorization headers from logger (elastic#26892)
  Expose custom process metrics  (elastic#26912)
  [gcp/billing] always quote table name identifier (elastic#26870)
  Add Beats central management removal to BCs (elastic#26400)
  Add custom suffix to identifiers in filestream input when needed (elastic#26669)
  Update asa-ftd-pipeline.yml (elastic#26265)
  Use common host parser in vsphere module (elastic#26904)
  [automation] Update go release version 1.16.6 (elastic#26860)
  Skip flaky test: filestream and harvester group (elastic#26728)
  [Filebeat] Remove alias fields from Suricata and Traefik module mappings (elastic#26627)
  docs: apm-server.auth (elastic#26831)
  [Automation] Update elastic stack version to 8.0.0-2f008f4a for testing (elastic#26881)
  Clarify the scope of start/end multiline example (elastic#26786)
  [Heartbeat]: update Node.js version for synthetics (elastic#26867)
  [fix][httpjson] Fix incorrect key for template data (elastic#26848)
  [httpjson] Add value_type parameter to httpjson transforms (elastic#26847)
  [Heartbeat]: capture error from journey/end events (elastic#26781)
  [Winlogbeat] Fixes for wineventlog experimental api (elastic#26826)
  Set agent.id to Fleet Agent ID for each metric/log monitoring input (elastic#26776)
  ...
cachedout pushed a commit that referenced this pull request Jul 19, 2021
@endorama
[gcp/billing] always quote table name identifier (#26870)
97f5728 
(cherry picked from commit f731832)
endorama added a commit that referenced this pull request Jul 21, 2021
@mergify @endorama
[7.x](backport #26870) [gcp/billing] always quote table name identifi…
9f0cc3a 
…er (#26932)

(cherry picked from commit f731832)

Co-authored-by: endorama <526307+endorama@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport-v7.14.0 Automated backport with mergify backport-v7.15.0 Automated backport with mergify needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Metricbeat GCP billing metricset does not work with alphanumerical project ids

3 participants

@endorama @elasticmachine @kaiyan-sheng