Skip to content

Commit 554d564

Browse files
narphnarph
authored
Add support for different Azure Cloud environments in the metricbeat azure module (elastic#21044)
* mofidy doc * work * changelog * fix url * work * fmt update * err
1 parent 713a503 commit 554d564

11 files changed

Lines changed: 116 additions & 49 deletions

File tree

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -769,6 +769,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
769769
- Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]
770770
- Move `compute_vm_scaleset` to light metricset. {pull}21038[21038] {issue}20985[20985]
771771
- Sanitize `event.host`. {pull}21022[21022]
772+
- Add support for different Azure Cloud environments in the metricbeat azure module. {pull}21044[21044] {issue}20988[20988]
772773
- Add overview and platform health dashboards to Cloud Foundry module. {pull}21124[21124]
773774
- Release lambda metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255]
774775
- Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]

‎metricbeat/docs/modules/azure.asciidoc‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,26 @@ Required credentials for the `azure` module:
7070
`tenant_id`:: The unique identifier of the Azure Active Directory instance
7171

7272

73-
Users can use the azure credentials keys if configured `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`
73+
The azure credentials keys can be used if configured `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`
74+
75+
`resource_manager_endpoint` ::
76+
_string_
77+
Optional, by default the azure public environment will be used, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
78+
Ex:
79+
https://management.chinacloudapi.cn for azure ChinaCloud
80+
https://management.microsoftazure.de for azure GermanCloud
81+
https://management.azure.com for azure PublicCloud
82+
https://management.usgovcloudapi.net for azure USGovernmentCloud
83+
84+
`active_directory_endpoint` ::
85+
_string_
86+
Optional, by default the associated active directory endpoint to the resource manager endpoint will be used, to override, users can provide a specific active directory endpoint in order to use a different azure environment.
87+
Ex:
88+
https://login.microsoftonline.com for azure ChinaCloud
89+
https://login.microsoftonline.us for azure GermanCloud
90+
https://login.chinacloudapi.cn for azure PublicCloud
91+
https://login.microsoftonline.de for azure USGovernmentCloud
92+
7493

7594
[float]
7695
== Metricsets

‎x-pack/metricbeat/module/azure/_meta/docs.asciidoc‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,26 @@ Required credentials for the `azure` module:
6262
`tenant_id`:: The unique identifier of the Azure Active Directory instance
6363

6464

65-
Users can use the azure credentials keys if configured `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`
65+
The azure credentials keys can be used if configured `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`
66+
67+
`resource_manager_endpoint` ::
68+
_string_
69+
Optional, by default the azure public environment will be used, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
70+
Ex:
71+
https://management.chinacloudapi.cn for azure ChinaCloud
72+
https://management.microsoftazure.de for azure GermanCloud
73+
https://management.azure.com for azure PublicCloud
74+
https://management.usgovcloudapi.net for azure USGovernmentCloud
75+
76+
`active_directory_endpoint` ::
77+
_string_
78+
Optional, by default the associated active directory endpoint to the resource manager endpoint will be used, to override, users can provide a specific active directory endpoint in order to use a different azure environment.
79+
Ex:
80+
https://login.microsoftonline.com for azure ChinaCloud
81+
https://login.microsoftonline.us for azure GermanCloud
82+
https://login.chinacloudapi.cn for azure PublicCloud
83+
https://login.microsoftonline.de for azure USGovernmentCloud
84+
6685

6786
[float]
6887
== Metricsets

‎x-pack/metricbeat/module/azure/billing/billing.go‎

Lines changed: 3 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@
55
package billing
66

77
import (
8-
"time"
9-
108
"github.com/pkg/errors"
119

10+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
11+
1212
"github.com/elastic/beats/v7/libbeat/logp"
1313
"github.com/elastic/beats/v7/metricbeat/mb"
1414
"github.com/elastic/beats/v7/metricbeat/mb/parse"
@@ -32,19 +32,10 @@ type MetricSet struct {
3232
log *logp.Logger
3333
}
3434

35-
// Config options
36-
type Config struct {
37-
ClientId string `config:"client_id" validate:"required"`
38-
ClientSecret string `config:"client_secret" validate:"required"`
39-
TenantId string `config:"tenant_id" validate:"required"`
40-
SubscriptionId string `config:"subscription_id" validate:"required"`
41-
Period time.Duration `config:"period" validate:"nonzero,required"`
42-
}
43-
4435
// New creates a new instance of the MetricSet. New is responsible for unpacking
4536
// any MetricSet specific configuration options if there are any.
4637
func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
47-
var config Config
38+
var config azure.Config
4839
err := base.Module().UnpackConfig(&config)
4940
if err != nil {
5041
return nil, errors.Wrap(err, "error unpack raw module config using UnpackConfig")

‎x-pack/metricbeat/module/azure/billing/client.go‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ import (
88
"fmt"
99
"time"
1010

11+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
12+
1113
"github.com/pkg/errors"
1214

1315
"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"
@@ -18,7 +20,7 @@ import (
1820
// Client represents the azure client which will make use of the azure sdk go metrics related clients
1921
type Client struct {
2022
BillingService Service
21-
Config Config
23+
Config azure.Config
2224
Log *logp.Logger
2325
}
2426

@@ -29,8 +31,8 @@ type Usage struct {
2931
}
3032

3133
// NewClient instantiates the an Azure monitoring client
32-
func NewClient(config Config) (*Client, error) {
33-
usageService, err := NewService(config.ClientId, config.ClientSecret, config.TenantId, config.SubscriptionId)
34+
func NewClient(config azure.Config) (*Client, error) {
35+
usageService, err := NewService(config)
3436
if err != nil {
3537
return nil, err
3638
}

‎x-pack/metricbeat/module/azure/billing/client_test.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,12 @@ import (
1111
"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"
1212
"github.com/stretchr/testify/assert"
1313
"github.com/stretchr/testify/mock"
14+
15+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
1416
)
1517

1618
var (
17-
config = Config{}
19+
config = azure.Config{}
1820
)
1921

2022
func TestClient(t *testing.T) {

‎x-pack/metricbeat/module/azure/billing/mock_service.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ package billing
77
import (
88
"github.com/stretchr/testify/mock"
99

10+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
11+
1012
"github.com/elastic/beats/v7/libbeat/logp"
1113

1214
"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"
@@ -27,7 +29,7 @@ type MockService struct {
2729
func NewMockClient() *Client {
2830
return &Client{
2931
new(MockService),
30-
Config{},
32+
azure.Config{},
3133
logp.NewLogger("test azure monitor"),
3234
}
3335
}

‎x-pack/metricbeat/module/azure/billing/service.go‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ package billing
77
import (
88
"context"
99

10+
"github.com/elastic/beats/v7/x-pack/metricbeat/module/azure"
11+
1012
"github.com/Azure/azure-sdk-for-go/services/consumption/mgmt/2019-01-01/consumption"
1113
"github.com/Azure/go-autorest/autorest/azure/auth"
1214

@@ -22,14 +24,16 @@ type UsageService struct {
2224
}
2325

2426
// NewService instantiates the Azure monitoring service
25-
func NewService(clientId string, clientSecret string, tenantId string, subscriptionId string) (*UsageService, error) {
26-
clientConfig := auth.NewClientCredentialsConfig(clientId, clientSecret, tenantId)
27+
func NewService(config azure.Config) (*UsageService, error) {
28+
clientConfig := auth.NewClientCredentialsConfig(config.ClientId, config.ClientSecret, config.TenantId)
29+
clientConfig.AADEndpoint = config.ActiveDirectoryEndpoint
30+
clientConfig.Resource = config.ResourceManagerEndpoint
2731
authorizer, err := clientConfig.Authorizer()
2832
if err != nil {
2933
return nil, err
3034
}
31-
forcastsClient := consumption.NewForecastsClient(subscriptionId)
32-
usageDetailsClient := consumption.NewUsageDetailsClient(subscriptionId)
35+
forcastsClient := consumption.NewForecastsClientWithBaseURI(config.ResourceManagerEndpoint, config.SubscriptionId)
36+
usageDetailsClient := consumption.NewUsageDetailsClientWithBaseURI(config.ResourceManagerEndpoint, config.SubscriptionId)
3337
forcastsClient.Authorizer = authorizer
3438
usageDetailsClient.Authorizer = authorizer
3539
service := &UsageService{

‎x-pack/metricbeat/module/azure/client.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ type mapResourceMetrics func(client *Client, resources []resources.GenericResour
3232

3333
// NewClient instantiates the an Azure monitoring client
3434
func NewClient(config Config) (*Client, error) {
35-
azureMonitorService, err := NewService(config.ClientId, config.ClientSecret, config.TenantId, config.SubscriptionId)
35+
azureMonitorService, err := NewService(config)
3636
if err != nil {
3737
return nil, err
3838
}

‎x-pack/metricbeat/module/azure/config.go‎

Lines changed: 44 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,38 @@ package azure
77
import (
88
"time"
99

10+
"github.com/elastic/beats/v7/libbeat/common"
11+
1012
"github.com/pkg/errors"
1113
)
1214

15+
const (
16+
// DefaultBaseURI is the default URI used for the service Insights
17+
DefaultBaseURI = "https://management.azure.com/"
18+
)
19+
20+
var (
21+
AzureEnvs = common.MapStr{
22+
"https://management.azure.com/": "https://login.microsoftonline.com/",
23+
"https://management.usgovcloudapi.net/": "https://login.microsoftonline.us/",
24+
"https://management.chinacloudapi.cn/": "https://login.chinacloudapi.cn/",
25+
"https://management.microsoftazure.de/": "https://login.microsoftonline.de/",
26+
}
27+
)
28+
1329
// Config options
1430
type Config struct {
15-
ClientId string `config:"client_id"`
16-
ClientSecret string `config:"client_secret"`
17-
TenantId string `config:"tenant_id"`
18-
SubscriptionId string `config:"subscription_id"`
19-
Period time.Duration `config:"period" validate:"nonzero,required"`
20-
Resources []ResourceConfig `config:"resources"`
21-
RefreshListInterval time.Duration `config:"refresh_list_interval"`
22-
DefaultResourceType string `config:"default_resource_type"`
23-
AddCloudMetadata bool `config:"add_cloud_metadata"`
31+
ClientId string `config:"client_id" validate:"required"`
32+
ClientSecret string `config:"client_secret" validate:"required"`
33+
TenantId string `config:"tenant_id" validate:"required"`
34+
SubscriptionId string `config:"subscription_id" validate:"required"`
35+
Period time.Duration `config:"period" validate:"nonzero,required"`
36+
Resources []ResourceConfig `config:"resources"`
37+
RefreshListInterval time.Duration `config:"refresh_list_interval"`
38+
DefaultResourceType string `config:"default_resource_type"`
39+
AddCloudMetadata bool `config:"add_cloud_metadata"`
40+
ResourceManagerEndpoint string `config:"resource_manager_endpoint"`
41+
ActiveDirectoryEndpoint string `config:"active_directory_endpoint"`
2442
}
2543

2644
// ResourceConfig contains resource and metric list specific configuration.
@@ -52,17 +70,24 @@ type DimensionConfig struct {
5270
}
5371

5472
func (conf *Config) Validate() error {
55-
if conf.SubscriptionId == "" {
56-
return errors.New("no subscription ID has been configured")
57-
}
58-
if conf.ClientSecret == "" {
59-
return errors.New("no client secret has been configured")
60-
}
61-
if conf.ClientId == "" {
62-
return errors.New("no client ID has been configured")
73+
if conf.ResourceManagerEndpoint == "" {
74+
conf.ResourceManagerEndpoint = DefaultBaseURI
6375
}
64-
if conf.TenantId == "" {
65-
return errors.New("no tenant ID has been configured")
76+
if conf.ActiveDirectoryEndpoint == "" {
77+
ok, err := AzureEnvs.HasKey(conf.ResourceManagerEndpoint)
78+
if err != nil {
79+
return errors.Wrap(err, "No active directory endpoint found for the resource manager endpoint selected.")
80+
}
81+
if ok {
82+
add, err := AzureEnvs.GetValue(conf.ResourceManagerEndpoint)
83+
if err != nil {
84+
return errors.Wrap(err, "No active directory endpoint found for the resource manager endpoint selected.")
85+
}
86+
conf.ActiveDirectoryEndpoint = add.(string)
87+
}
88+
if conf.ActiveDirectoryEndpoint == "" {
89+
return errors.New("no active directory endpoint has been configured")
90+
}
6691
}
6792
return nil
6893
}

0 commit comments

Comments
 (0)