[8.3](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32298
Merged
[8.3](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32298
Conversation
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc)
Contributor
Contributor
Author
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
Preallocating events slice using the number of usage details as starting size.
Remove unrelated entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automatic backport of pull request #31970 done by Mergify.
Mergify commands and options
More conditions and actions can be found in the documentation.
You can also trigger Mergify actions by commenting on this pull request:
@Mergifyio refreshwill re-evaluate the rules@Mergifyio rebasewill rebase this PR on its base branch@Mergifyio updatewill merge the base branch into this PR@Mergifyio backport <destination>will backport this PR on<destination>branchAdditionally, on Mergify dashboard you can:
Finally, you can contact us on https://mergify.com