Skip to content

Commit 1f232dc

Browse files
zmoogkaiyan-shengtommyers-elastic
authored
[Azure Billing] Upgrade Usage Details API to version 2019-10-01 (#31970)
* Add Usage Details API with patches This is a stripped down version of the 2019-10-01 version of Usage Details API and Forecasts API with small changes required to data range data filtering. The standard version of the API does non support the required date filtering required to handle MCA customers. I hope in the future Microsoft will release a new version of this API with support for MCA, so we can get rid of this code and just use the upstream version. * Update billing module to use API 2019-10-01 Usage Details is now using API version 2019-10-01, and now the module can handle both the legacy and modern data formats. Due to restrictions for modern customers with a Microsoft Customer Agreement [1], the start and end dates are passed to the `List()` method in both the `$filter` and `start/endDate` query string parameters. [1]: https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/manage-automation#get-usage-details-for-a-scope-during-specific-date-range * Fix linter objections * Fix tests * Fix all linter complaints * Switch to forked github.com/Azure/azure-sdk-for-go We created a fork [1] of the stream github.com/Azure/azure-sdk-for-go with the patches needed to run Usage Details API with modern Azure accounts. [1]: https://github.com/elastic/azure-sdk-for-go * Add logger * Properly initialize shared event fields * Remove the embedded azure clients We no longer need this proof-of-concept code; we are going to leverage the external fork [1]. [1]: https://github.com/elastic/azure-sdk-for-go * I will run the linter before committing, I will... * Do not fail on unsupported subscriptions Forecasts API 2019-10-01 supports Enterprise customers only [1], so it returns a 404 error when used by MCA customers. Check for 404s and log a warning, so at least the Usage Details are successfully collected and sent to Elasticsearch. [1]: "Provides operations to get usage forecasts for Enterprise Subscriptions"; see https://docs.microsoft.com/en-us/rest/api/consumption/ fore more. * Extract resource name from path for modern data The resource name is not available as a standalone field, but can be extracted from the instance name field. * Fix linter objections and failing tests * Update Go mod/sum * Finalize the list of supported fields * Cleanup * Update generated docs * Remove duplicated field * Update generated docs (yes, again) * Fix typos and leftovers * Go.mod replace: use version instead of ts+hash It's better looking and more meaningful * Use spaces as logger name separator Other logger names in the azure package are using this convention. * Fix a HUGE mistake in start time and add a comment * Switch to structured logging * Handle unsupported data format as an error * Remove unclear detail We'll add it back when we can explain the context. * Move time window calculation to a testable func Now it's asier to read and test. * Add missing file header * Update x-pack/metricbeat/module/azure/billing/billing_test.go Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update CHANGELOG * Fix typo in a function name Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
1 parent 1f2895e commit 1f232dc

16 files changed

Lines changed: 382 additions & 158 deletions

File tree

‎CHANGELOG.next.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,8 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
114114
- Enhance Oracle Module: Connection string for Oracle does not handle special characters properly {issue}24609[24609] {pull}31368[#31368]
115115
- Enhance Oracle Module: New sysmetric metricset {issue}30946[30946] {pull}31462[#31462]
116116
- Upgrade Mongodb library in Beats to v5 {pull}31185[31185]
117+
- Azure Billing: upgrade Usage Details API to version 2019-10-01 {pull}31970[31970]
118+
* Differentiate between actual idle CPU states and an uninterruptible disk sleep. https://github.com/elastic/elastic-agent-system-metrics/pull/32[system-metrics#32]
117119

118120
*Packetbeat*
119121

‎NOTICE.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -862,12 +862,12 @@ Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-event-hubs
862862

863863

864864
--------------------------------------------------------------------------------
865-
Dependency : github.com/Azure/azure-sdk-for-go
866-
Version: v59.0.0+incompatible
865+
Dependency : github.com/elastic/azure-sdk-for-go
866+
Version: v59.0.0-elastic-1+incompatible
867867
Licence type (autodetected): MIT
868868
--------------------------------------------------------------------------------
869869

870-
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go@v59.0.0+incompatible/LICENSE.txt:
870+
Contents of probable licence file $GOMODCACHE/github.com/elastic/azure-sdk-for-go@v59.0.0-elastic-1+incompatible/LICENSE.txt:
871871

872872
The MIT License (MIT)
873873

‎go.mod‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,7 @@ require (
308308
)
309309

310310
replace (
311+
github.com/Azure/azure-sdk-for-go => github.com/elastic/azure-sdk-for-go v59.0.0-elastic-1+incompatible
311312
github.com/Microsoft/go-winio => github.com/bi-zone/go-winio v0.4.15
312313
github.com/Shopify/sarama => github.com/elastic/sarama v1.19.1-0.20220310193331-ebc2b0d8eef3
313314
github.com/apoydence/eachers => github.com/poy/eachers v0.0.0-20181020210610-23942921fe77 //indirect, see https://github.com/elastic/beats/pull/29780 for details.

‎go.sum‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -77,12 +77,6 @@ github.com/Azure/azure-pipeline-go v0.1.8/go.mod h1:XA1kFWRVhSK+KNFiOhfv83Fv8L9a
7777
github.com/Azure/azure-pipeline-go v0.1.9/go.mod h1:XA1kFWRVhSK+KNFiOhfv83Fv8L9achrP7OxIzeTn1Yg=
7878
github.com/Azure/azure-pipeline-go v0.2.1 h1:OLBdZJ3yvOn2MezlWvbrBMTEUQC72zAftRZOMdj5HYo=
7979
github.com/Azure/azure-pipeline-go v0.2.1/go.mod h1:UGSo8XybXnIGZ3epmeBw7Jdz+HiUVpqIlpz/HKHylF4=
80-
github.com/Azure/azure-sdk-for-go v16.2.1+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
81-
github.com/Azure/azure-sdk-for-go v41.3.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
82-
github.com/Azure/azure-sdk-for-go v51.1.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
83-
github.com/Azure/azure-sdk-for-go v55.2.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
84-
github.com/Azure/azure-sdk-for-go v59.0.0+incompatible h1:I1ULJqny1qQhUBFy11yDXHhW3pLvbhwV0PTn7mjp9V0=
85-
github.com/Azure/azure-sdk-for-go v59.0.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
8680
github.com/Azure/azure-storage-blob-go v0.6.0/go.mod h1:oGfmITT1V6x//CswqY2gtAHND+xIP64/qL7a5QJix0Y=
8781
github.com/Azure/azure-storage-blob-go v0.8.0 h1:53qhf0Oxa0nOjgbDeeYPUeyiNmafAFEY95rZLK0Tj6o=
8882
github.com/Azure/azure-storage-blob-go v0.8.0/go.mod h1:lPI3aLPpuLTeUwh1sViKXFxwl2B6teiRqI0deQUvsw0=
@@ -528,6 +522,8 @@ github.com/eclipse/paho.mqtt.golang v1.2.0/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7j
528522
github.com/eclipse/paho.mqtt.golang v1.3.5 h1:sWtmgNxYM9P2sP+xEItMozsR3w0cqZFlqnNN1bdl41Y=
529523
github.com/eclipse/paho.mqtt.golang v1.3.5/go.mod h1:eTzb4gxwwyWpqBUHGQZ4ABAV7+Jgm1PklsYT/eo8Hcc=
530524
github.com/edsrzf/mmap-go v1.0.0/go.mod h1:YO35OhQPt3KJa3ryjFM5Bs14WD66h8eGKpfaBNrHW5M=
525+
github.com/elastic/azure-sdk-for-go v59.0.0-elastic-1+incompatible h1:jlUO91EFZuvAO+2Zg+WdV0iTWe/x1X8maTxdYIKCWu4=
526+
github.com/elastic/azure-sdk-for-go v59.0.0-elastic-1+incompatible/go.mod h1:4zuQekLQi489ShcqTmS1Zj1ta0qrcNBlSuGa+ziu2vM=
531527
github.com/elastic/bayeux v1.0.5 h1:UceFq01ipmT3S8DzFK+uVAkbCdiPR0Bqei8qIGmUeY0=
532528
github.com/elastic/bayeux v1.0.5/go.mod h1:CSI4iP7qeo5MMlkznGvYKftp8M7qqP/3nzmVZoXHY68=
533529
github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 h1:lnDkqiRFKm0rxdljqrj3lotWinO9+jFmeDXIC4gvIQs=

‎metricbeat/docs/fields.asciidoc‎

Lines changed: 45 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5580,11 +5580,6 @@ azure module
55805580

55815581

55825582

5583-
[float]
5584-
=== azure
5585-
5586-
5587-
55885583

55895584
*`azure.timegrain`*::
55905585
+
@@ -5659,6 +5654,16 @@ type: keyword
56595654
The subscription ID
56605655

56615656

5657+
type: keyword
5658+
5659+
--
5660+
5661+
*`azure.subscription_name`*::
5662+
+
5663+
--
5664+
The subscription name
5665+
5666+
56625667
type: keyword
56635668

56645669
--
@@ -5947,7 +5952,7 @@ billing and usage details
59475952
*`azure.billing.currency`*::
59485953
+
59495954
--
5950-
The currency
5955+
Billing Currency.
59515956

59525957

59535958
type: keyword
@@ -5957,7 +5962,27 @@ type: keyword
59575962
*`azure.billing.pretax_cost`*::
59585963
+
59595964
--
5960-
Cost
5965+
The amount of cost before tax.
5966+
5967+
5968+
type: float
5969+
5970+
--
5971+
5972+
*`azure.billing.unit_price`*::
5973+
+
5974+
--
5975+
Unit Price is the price applicable to you. (your EA or other contract price).
5976+
5977+
5978+
type: float
5979+
5980+
--
5981+
5982+
*`azure.billing.quantity`*::
5983+
+
5984+
--
5985+
Measure the quantity purchased or consumed. The amount of the meter used during the billing period.
59615986

59625987

59635988
type: float
@@ -5977,7 +6002,7 @@ type: keyword
59776002
*`azure.billing.product`*::
59786003
+
59796004
--
5980-
The product type
6005+
Product name for the consumed service or purchase.
59816006

59826007

59836008
type: keyword
@@ -6007,7 +6032,7 @@ type: date
60076032
*`azure.billing.billing_period_id`*::
60086033
+
60096034
--
6010-
The billing period id
6035+
The billing period id.
60116036

60126037

60136038
type: keyword
@@ -6017,7 +6042,17 @@ type: keyword
60176042
*`azure.billing.account_name`*::
60186043
+
60196044
--
6020-
The billing account name
6045+
Name of the Billing Account.
6046+
6047+
6048+
type: keyword
6049+
6050+
--
6051+
6052+
*`azure.billing.account_id`*::
6053+
+
6054+
--
6055+
Billing Account identifier.
60216056

60226057

60236058
type: keyword

‎x-pack/metricbeat/module/azure/_meta/fields.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
fields:
77
- name: azure
88
type: group
9-
description: >
109
fields:
1110
- name: timegrain
1211
type: keyword
@@ -43,6 +42,10 @@
4342
type: keyword
4443
description: >
4544
The subscription ID
45+
- name: subscription_name
46+
type: keyword
47+
description: >
48+
The subscription name
4649
- name: application_id
4750
type: keyword
4851
description: >

‎x-pack/metricbeat/module/azure/billing/_meta/fields.yml‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,27 @@
77
- name: currency
88
type: keyword
99
description: >
10-
The currency
10+
Billing Currency.
1111
- name: pretax_cost
1212
type: float
1313
description: >
14-
Cost
14+
The amount of cost before tax.
15+
- name: unit_price
16+
type: float
17+
description: >
18+
Unit Price is the price applicable to you. (your EA or other contract price).
19+
- name: quantity
20+
type: float
21+
description: >
22+
Measure the quantity purchased or consumed. The amount of the meter used during the billing period.
1523
- name: department_name
1624
type: keyword
1725
description: >
1826
The department name
1927
- name: product
2028
type: keyword
2129
description: >
22-
The product type
30+
Product name for the consumed service or purchase.
2331
- name: usage_start
2432
type: date
2533
description: >
@@ -31,11 +39,15 @@
3139
- name: billing_period_id
3240
type: keyword
3341
description: >
34-
The billing period id
42+
The billing period id.
3543
- name: account_name
3644
type: keyword
3745
description: >
38-
The billing account name
46+
Name of the Billing Account.
47+
- name: account_id
48+
type: keyword
49+
description: >
50+
Billing Account identifier.
3951
- name: actual_cost
4052
type: float
4153
description: >

‎x-pack/metricbeat/module/azure/billing/billing.go‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,12 @@
55
package billing
66

77
import (
8-
"github.com/pkg/errors"
9-
10-
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
8+
"fmt"
9+
"time"
1110

1211
"github.com/elastic/beats/v7/metricbeat/mb"
1312
"github.com/elastic/beats/v7/metricbeat/mb/parse"
13+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
1414
"github.com/elastic/elastic-agent-libs/logp"
1515
)
1616

@@ -24,7 +24,7 @@ func init() {
2424

2525
// MetricSet holds any configuration or state information. It must implement
2626
// the mb.MetricSet interface. And this is best achieved by embedding
27-
// mb.BaseMetricSet because it implements all of the required mb.MetricSet
27+
// mb.BaseMetricSet because it implements all the required mb.MetricSet
2828
// interface methods except for Fetch.
2929
type MetricSet struct {
3030
mb.BaseMetricSet
@@ -38,31 +38,45 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
3838
var config azure.Config
3939
err := base.Module().UnpackConfig(&config)
4040
if err != nil {
41-
return nil, errors.Wrap(err, "error unpack raw module config using UnpackConfig")
41+
return nil, fmt.Errorf("error unpack raw module config using UnpackConfig: %w", err)
4242
}
4343
if err != nil {
4444
return nil, err
4545
}
4646
// instantiate monitor client
4747
billingClient, err := NewClient(config)
4848
if err != nil {
49-
return nil, errors.Wrap(err, "error initializing the billing client: module azure - billing metricset")
49+
return nil, fmt.Errorf("error initializing the billing client: module azure - billing metricset: %w", err)
5050
}
5151
return &MetricSet{
5252
BaseMetricSet: base,
5353
client: billingClient,
54+
log: logp.NewLogger("azure billing"),
5455
}, nil
5556
}
5657

5758
// Fetch methods implements the data gathering and data conversion to the right metricset
5859
// It publishes the event which is then forwarded to the output. In case
5960
// of an error set the Error field of mb.Event or simply call report.Error().
6061
func (m *MetricSet) Fetch(report mb.ReporterV2) error {
61-
results, err := m.client.GetMetrics()
62+
// The time interval is yesterday (00:00:00->23:59:59) in UTC.
63+
startTime, endTime := previousDayFrom(time.Now())
64+
65+
m.log.
66+
With("billing.start_time", startTime).
67+
With("billing.end_time", endTime).
68+
Infow("Fetching billing data")
69+
70+
results, err := m.client.GetMetrics(startTime, endTime)
6271
if err != nil {
63-
return errors.Wrap(err, "error retrieving usage information")
72+
return fmt.Errorf("error retrieving usage information: %w", err)
6473
}
65-
events := EventsMapping(m.client.Config.SubscriptionId, results)
74+
75+
events, err := EventsMapping(m.client.Config.SubscriptionId, results, startTime, endTime)
76+
if err != nil {
77+
return fmt.Errorf("error mapping events: %w", err)
78+
}
79+
6680
for _, event := range events {
6781
isOpen := report.Event(event)
6882
if !isOpen {
@@ -72,3 +86,10 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error {
7286

7387
return nil
7488
}
89+
90+
// previousDayFrom returns the start/end times (00:00:00->23:59:59 UTC) of the day before, given the `reference` time.
91+
func previousDayFrom(reference time.Time) (time.Time, time.Time) {
92+
startTime := reference.UTC().Truncate(24 * time.Hour).Add((-24) * time.Hour)
93+
endTime := startTime.Add(time.Hour * 24).Add(time.Second * (-1))
94+
return startTime, endTime
95+
}

‎x-pack/metricbeat/module/azure/billing/billing_test.go‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
2+
// or more contributor license agreements. Licensed under the Elastic License;
3+
// you may not use this file except in compliance with the Elastic License.
4+
5+
package billing
6+
7+
import (
8+
"testing"
9+
"time"
10+
11+
"github.com/stretchr/testify/assert"
12+
)
13+
14+
func TestPreviousDayFrom(t *testing.T) {
15+
t.Run("returns the previous day as time interval to collect metrics", func(t *testing.T) {
16+
referenceTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-09 09:41:00")
17+
assert.NoError(t, err)
18+
expectedStartTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 00:00:00")
19+
assert.NoError(t, err)
20+
expectedEndTime, err := time.Parse("2006-01-02 15:04:05", "2007-01-08 23:59:59")
21+
assert.NoError(t, err)
22+
23+
actualStartTime, actualEndTime := previousDayFrom(referenceTime)
24+
25+
assert.Equal(t, expectedStartTime, actualStartTime)
26+
assert.Equal(t, expectedEndTime, actualEndTime)
27+
})
28+
}

0 commit comments

Comments
 (0)