[7.17](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32297
[7.17](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32297
Conversation
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
|
Backporting this change to 7.17 requires extra work due to recent changes in shared libraries. For example, the move from I'm closing this PR due to the unexpected cost of porting. We will reevaluate this position in the future if there is an appetite for this change in 7.17.x series. |
|
Backporting to 7.17 is back in progress. |
99683f7 to
e9d7197
Compare
|
/test |
a866f0f to
a8ea6fa
Compare
|
/package |
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
e9c062e to
a3485a4
Compare
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
|
This pull request is now in conflicts. Could you fix it? 🙏 |
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
|
I believe elastic/e2e-testing#3048 will help to get this PR through. |
|
/test |
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
|
/test |
2 similar comments
|
/test |
|
/test |
|
/test |
1 similar comment
|
/test |
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
d4ee905 to
1e27a62
Compare
|
/test |
1e27a62 to
fdffd3b
Compare
|
/test |
5103bc6 to
4161ab1
Compare
|
This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏 |
0f5e3b7 to
79d03b8
Compare
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
79d03b8 to
010ceb3
Compare
|
/test |
This is an automatic backport of pull request #31970 done by Mergify.
Cherry-pick of 1f232dc has failed:
To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Mergify commands and options
More conditions and actions can be found in the documentation.
You can also trigger Mergify actions by commenting on this pull request:
@Mergifyio refreshwill re-evaluate the rules@Mergifyio rebasewill rebase this PR on its base branch@Mergifyio updatewill merge the base branch into this PR@Mergifyio backport <destination>will backport this PR on<destination>branchAdditionally, on Mergify dashboard you can:
Finally, you can contact us on https://mergify.com