[Azure Billing] Upgrade Usage Details API to version 2019-10-01#31970
Merged
tommyers-elastic merged 35 commits intoelastic:mainfrom Jul 11, 2022
Merged
Conversation
|
This pull request doesn't have a |
Contributor
31c290b to
4d876f2
Compare
Contributor
Author
|
/test |
rdner
reviewed
Jun 23, 2022
rdner
approved these changes
Jun 23, 2022
tommyers-elastic
approved these changes
Jun 23, 2022
Contributor
tommyers-elastic
left a comment
There was a problem hiding this comment.
awesome work - no blockers from me!
kaiyan-sheng
approved these changes
Jun 27, 2022
471e9bc to
d49342b
Compare
Contributor
Author
|
/package |
Contributor
|
This pull request is now in conflicts. Could you fix it? 🙏 |
…pi-upgrade-2019-10-01
…pi-upgrade-2019-10-01
…pi-upgrade-2019-10-01
mergify bot
pushed a commit
that referenced
this pull request
Jul 11, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
mergify bot
pushed a commit
that referenced
this pull request
Jul 11, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc)
|
In what version should this be fixed? |
Contributor
Author
|
@willemdh It will be available in 8.4. I'm currently checking to backport it to the next 8.3.x and 7.17.x releases. |
zmoog
added a commit
that referenced
this pull request
Jul 20, 2022
…ersion 2019-10-01 (#32298) * [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970) * Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
Contributor
Author
|
Quick update about the availability of the fix across currently planned releases of Beats (Metricbeat) and Elastic Agent:
|
zmoog
added a commit
that referenced
this pull request
Aug 24, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Aug 25, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Aug 31, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Sep 11, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Oct 10, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Oct 11, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Oct 13, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Oct 18, 2022
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go
zmoog
added a commit
that referenced
this pull request
Oct 18, 2022
…version 2019-10-01 (#32297) * [Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970) * Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com> (cherry picked from commit 1f232dc) # Conflicts: # go.sum # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go * Fix conflicts * Clean up * Remove unrelated changelog entries * go mod tidy Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>
chrisberkhout
pushed a commit
that referenced
this pull request
Jun 1, 2023
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR upgrades the Usage Details API client to version 2019-10-01 to add support for the existing "legacy" and the new "modern" 1 Azure account types.
MIgration
In the migration from the original (2019-01-01) version to the new (2019-10-01) version, we kept the same field naming tried to maintain the semantics as close as possible to the original.
In addition, we added a few new fields that were available on both "legacy" and "modern" account types:
usage_dateaccount_idunit_pricequantityThere are lots of other fields available for both legacy and modern account types, but we plan to introduce them with different PRs and according to provided value and users' interests.
Patch Management
To accomplish this goal, we had to patch the existing Usage Details API client to make it work with modern customers with a Microsoft Customer Agreement.
Solution A (currently implemented)
To host the patch, we created a fork of the original Azure/azure-sdk-for-go repo.
We applied the patch on top of the azure/azure-sdk-for-go@v59.0.0; see https://github.com/elastic/azure-sdk-for-go/releases/tag/v59.0.0-elastic-1 for more details about branch and tag.
As a long-term solution, we are also getting in touch with Microsoft to update the OpenAPI spec used to generate the clients, so we don't have to maintain any patch anymore.
Solution B (pivoted)
Since the Usage Details API and other clients are generated from OpenAPI specs, we could generate and patch our own API clients using the AutoRest, the same toolchain used by Microsoft.
Pros:
I tested this approach in the zmoog/azure-sdk-clients and it works.
Why is it important?
Right now, a few people using modern 1 Azure accounts cannot use the Azure Billing metrics module due to errors. For more information on this problem, check the following issues:
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs
Footnotes
In 2021, Azure introduced new plans creating a distinction between "legacy" and "modern" account types. ↩ ↩2