Skip to content

[8.3](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32298

Merged
zmoog merged 3 commits into8.3from
mergify/bp/8.3/pr-31970
Jul 20, 2022
Merged

[8.3](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32298
zmoog merged 3 commits into8.3from
mergify/bp/8.3/pr-31970

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Jul 11, 2022

This is an automatic backport of pull request #31970 done by Mergify.


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)
@mergify mergify bot requested review from a team as code owners July 11, 2022 10:21
@mergify mergify bot requested review from faec and fearful-symmetry and removed request for a team July 11, 2022 10:21
@mergify mergify bot added the backport label Jul 11, 2022
@mergify mergify bot assigned zmoog Jul 11, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 11, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 11, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-19T16:03:01.034+0000

  • Duration: 131 min 19 sec

Test stats 🧪

Test Results
Failed 0
Passed 22138
Skipped 1910
Total 24048

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Jul 18, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@mtojek mtojek added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Jul 18, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 18, 2022
zmoog added 2 commits July 19, 2022 17:53
Preallocating events slice using the number of usage details as
starting size.
Remove unrelated entries.
@zmoog zmoog merged commit 38e84fa into 8.3 Jul 20, 2022
@zmoog zmoog deleted the mergify/bp/8.3/pr-31970 branch July 20, 2022 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants