Skip to content

[7.17](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32297

Merged
zmoog merged 5 commits into7.17from
mergify/bp/7.17/pr-31970
Oct 18, 2022
Merged

[7.17](backport #31970) [Azure Billing] Upgrade Usage Details API to version 2019-10-01#32297
zmoog merged 5 commits into7.17from
mergify/bp/7.17/pr-31970

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Jul 11, 2022

This is an automatic backport of pull request #31970 done by Mergify.
Cherry-pick of 1f232dc has failed:

On branch mergify/bp/7.17/pr-31970
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit 1f232dc343.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   metricbeat/docs/fields.asciidoc
	modified:   x-pack/metricbeat/module/azure/_meta/fields.yml
	modified:   x-pack/metricbeat/module/azure/billing/_meta/fields.yml
	new file:   x-pack/metricbeat/module/azure/billing/billing_test.go
	modified:   x-pack/metricbeat/module/azure/billing/client_test.go
	modified:   x-pack/metricbeat/module/azure/billing/data_test.go
	modified:   x-pack/metricbeat/module/azure/fields.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   go.sum
	both modified:   x-pack/metricbeat/module/azure/billing/billing.go
	both modified:   x-pack/metricbeat/module/azure/billing/client.go
	both modified:   x-pack/metricbeat/module/azure/billing/data.go
	both modified:   x-pack/metricbeat/module/azure/billing/mock_service.go
	both modified:   x-pack/metricbeat/module/azure/billing/service.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

@mergify mergify bot requested review from a team as code owners July 11, 2022 10:21
@mergify mergify bot requested review from cmacknz and leehinman and removed request for a team July 11, 2022 10:21
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Jul 11, 2022
@mergify mergify bot assigned zmoog Jul 11, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 11, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 11, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-18T13:56:52.617+0000

  • Duration: 132 min 53 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Jul 18, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@mtojek mtojek added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Jul 18, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 18, 2022
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Jul 20, 2022

Backporting this change to 7.17 requires extra work due to recent changes in shared libraries. For example, the move from common.MapStr to mapstr.M has several ramifications in the codebase.

I'm closing this PR due to the unexpected cost of porting. We will reevaluate this position in the future if there is an appetite for this change in 7.17.x series.

@zmoog zmoog closed this Jul 20, 2022
@mergify mergify bot deleted the mergify/bp/7.17/pr-31970 branch July 20, 2022 21:20
@zmoog zmoog restored the mergify/bp/7.17/pr-31970 branch August 23, 2022 08:44
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Aug 23, 2022

Backporting to 7.17 is back in progress.

@zmoog zmoog reopened this Aug 23, 2022
@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch from 99683f7 to e9d7197 Compare August 24, 2022 19:33
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Aug 25, 2022

/test

@zmoog zmoog removed the conflicts There is a conflict in the backported pull request label Aug 25, 2022
@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch from a866f0f to a8ea6fa Compare August 25, 2022 09:36
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Aug 25, 2022

/package

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Aug 29, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch 3 times, most recently from e9c062e to a3485a4 Compare August 31, 2022 08:17
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Sep 5, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Sep 8, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.17/pr-31970 upstream/mergify/bp/7.17/pr-31970
git merge upstream/7.17
git push upstream mergify/bp/7.17/pr-31970

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Sep 26, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Sep 28, 2022

I believe elastic/e2e-testing#3048 will help to get this PR through.

@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Sep 29, 2022

/test

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Oct 3, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 3, 2022

/test

2 similar comments
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 3, 2022

/test

@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 4, 2022

/test

@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 5, 2022

/test

1 similar comment
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 5, 2022

/test

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Oct 10, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch from d4ee905 to 1e27a62 Compare October 10, 2022 20:00
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 11, 2022

/test

@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch from 1e27a62 to fdffd3b Compare October 11, 2022 11:41
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 11, 2022

/test

@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch 2 times, most recently from 5103bc6 to 4161ab1 Compare October 13, 2022 10:27
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Oct 17, 2022

This pull request has not been merged yet. Could you please review and merge it @zmoog? 🙏

@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch 2 times, most recently from 0f5e3b7 to 79d03b8 Compare October 17, 2022 11:35
zmoog added 5 commits October 18, 2022 11:15
* Add Usage Details API with patches

This is a stripped down version of the 2019-10-01 version of Usage
Details API and Forecasts API with small changes required to data
range data filtering.

The standard version of the API does non support the required date
filtering required to handle MCA customers.

I hope in the future Microsoft will release a new version of this API
with support for MCA, so we can get rid of this code and just use the
upstream version.

* Update billing module to use API 2019-10-01

Usage Details is now using API version 2019-10-01, and now the module
can handle both the legacy and modern data formats.

Due to restrictions for modern customers with a Microsoft Customer
Agreement [1], the start and end dates are passed to the `List()`
method in both the `$filter` and `start/endDate` query string
parameters.

[1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range

* Fix linter objections

* Fix tests

* Fix all linter complaints

* Switch to forked github.com/Azure/azure-sdk-for-go

We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go
with the patches needed to run Usage Details API with modern Azure
accounts.

[1]: https://github.com/elastic/azure-sdk-for-go

* Add logger

* Properly initialize shared event fields

* Remove the embedded azure clients

We no longer need this proof-of-concept code; we are going to leverage
the external fork [1].

[1]: https://github.com/elastic/azure-sdk-for-go

* I will run the linter before committing, I will...

* Do not fail on unsupported subscriptions

Forecasts API 2019-10-01 supports Enterprise customers only [1], so it
returns a 404 error when used by MCA customers.

Check for 404s and log a warning, so at least the Usage Details are
successfully collected and sent to Elasticsearch.

[1]: "Provides operations to get usage forecasts for Enterprise
Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/
fore more.

* Extract resource name from path for modern data

The resource name is not available as a standalone field, but can
be extracted from the instance name field.

* Fix linter objections and failing tests

* Update Go mod/sum

* Finalize the list of supported fields

* Cleanup

* Update generated docs

* Remove duplicated field

* Update generated docs (yes, again)

* Fix typos and leftovers

* Go.mod replace: use version instead of ts+hash

It's better looking and more meaningful

* Use spaces as logger name separator

Other logger names in the azure package are using this convention.

* Fix a HUGE mistake in start time and add a comment

* Switch to structured logging

* Handle unsupported data format as an error

* Remove unclear detail

We'll add it back when we can explain the context.

* Move time window calculation to a testable func

Now it's asier to read and test.

* Add missing file header

* Update x-pack/metricbeat/module/azure/billing/billing_test.go

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

* Update CHANGELOG

* Fix typo in a function name

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 1f232dc)

# Conflicts:
#	go.sum
#	x-pack/metricbeat/module/azure/billing/billing.go
#	x-pack/metricbeat/module/azure/billing/client.go
#	x-pack/metricbeat/module/azure/billing/data.go
#	x-pack/metricbeat/module/azure/billing/mock_service.go
#	x-pack/metricbeat/module/azure/billing/service.go
@zmoog zmoog force-pushed the mergify/bp/7.17/pr-31970 branch from 79d03b8 to 010ceb3 Compare October 18, 2022 09:15
@zmoog
Copy link
Copy Markdown
Contributor

zmoog commented Oct 18, 2022

/test

@zmoog zmoog merged commit 8d7046e into 7.17 Oct 18, 2022
@zmoog zmoog deleted the mergify/bp/7.17/pr-31970 branch October 18, 2022 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants