Skip to content

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions#19519

Merged
kaiyan-sheng merged 4 commits intoelastic:7.xfrom
kaiyan-sheng:backport_19423_7.x
Jul 1, 2020
Merged

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions#19519
kaiyan-sheng merged 4 commits intoelastic:7.xfrom
kaiyan-sheng:backport_19423_7.x

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Jun 30, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #19423 to 7.x branch. Original message:

What does this PR do?

This PR is to add a parameter aws_partition for metricbeat aws cloudwatch module, so metricbeat can support aws china and aws us-gov account.

Why is it important?

Because there are three partitions aws, aws-cn, aws-us-gov of aws regions, now metricbeat can only support aws global region, here is a hardcode.

beats/x-pack/metricbeat/module/aws/aws.go

Lines 95 to 100 in 5ba7026

// Get IAM account name
awsConfig.Region = "us-east-1"
svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint(
config.AWSConfig.Endpoint, "iam", "", awsConfig))
req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. add aws_partition param in aws module config
metricbeat.modules:
  - module: aws
    aws_partition: aws-cn
    period: 300s
    access_key_id: 
    secret_access_key:
  1. run metricbeat , open debug log, then you can see it can retrive metrics from correct regions

Related issues

Use cases

  • no side affect
  • user can use metricbeat to get metrics from aws china and aws us-gov account except aws global account

Screenshots

aws global account, set aws_partition: aws in aws module config or remove the param

image

aws china account, set aws_partition: aws-cn in aws module config

image

@kwinstonix
[MetricBeat] add param aws_partition to support aws-cn, aws-us-gov …
9fd31ae 
…regions (#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov

(cherry picked from commit 1ea8c84)
@kaiyan-sheng kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jun 30, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 30, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 30, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Jun 30, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 30, 2020 •
edited
Loading

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19519 updated]

  • Reason: Aborted from #6

  • Start Time: 2020-07-01T13:47:20.258+0000

  • Duration: 111 min 26 sec

  • Commit: 1d53741

Test stats 🧪

Test Results
Failed 0
Passed 3680
Skipped 512
Total 4192

Log output

Expand to view the last 100 lines of log output 

[2020-07-01T15:04:22.791Z] WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
[2020-07-01T15:04:22.791Z] You should consider upgrading via the 'python -m pip install --upgrade pip' command.
[2020-07-01T15:04:22.791Z] S..
[2020-07-01T15:04:22.791Z] [success] 74.30% test_base.Test.test_export_function: 0.3363s
[2020-07-01T15:04:22.791Z] [success] 25.70% test_base.Test.test_export_function_invalid_conf: 0.1163s
[2020-07-01T15:04:22.791Z] ----------------------------------------------------------------------
[2020-07-01T15:04:22.791Z] Ran 3 tests in 0.454s
[2020-07-01T15:04:22.791Z] 
[2020-07-01T15:04:22.791Z] OK (SKIP=1)
[2020-07-01T15:04:22.791Z] >> python test: Unit Testing Complete
[2020-07-01T15:04:22.861Z] Recording test results
[2020-07-01T15:04:25.241Z] Stashed 2 file(s)
[2020-07-01T15:04:25.251Z] Archiving artifacts
[2020-07-01T15:38:02.174Z] .......Sending interrupt signal to process
[2020-07-01T15:38:04.906Z] Sending interrupt signal to process
[2020-07-01T15:38:17.170Z] Click here to forcibly terminate running steps
[2020-07-01T15:38:19.823Z] failed to run compiled magefile: signal: terminated
[2020-07-01T15:38:19.831Z] script returned exit code 255
[2020-07-01T15:38:19.912Z] Recording test results
[2020-07-01T15:38:21.596Z] Stashed 3 file(s)
[2020-07-01T15:38:21.605Z] Archiving artifacts
[2020-07-01T15:38:22.409Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-01T15:38:22.669Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-01T15:38:24.052Z] Failed in branch Filebeat x-pack
[2020-07-01T15:38:24.207Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:24.522Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-01T15:38:24.533Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Lint
[2020-07-01T15:38:24.612Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-01T15:38:24.686Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-01T15:38:24.760Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-01T15:38:24.845Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-01T15:38:24.928Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-01T15:38:25.002Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-01T15:38:25.080Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-01T15:38:25.167Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-01T15:38:25.244Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-01T15:38:25.319Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-01T15:38:25.394Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-01T15:38:25.471Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.548Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.633Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-01T15:38:25.710Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.800Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-01T15:38:25.873Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-01T15:38:25.957Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-01T15:38:26.324Z] + cat
[2020-07-01T15:38:26.324Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-01T15:38:26.324Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-01T15:38:32.926Z] runbld>>> runbld started
[2020-07-01T15:38:32.926Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-01T15:38:33.868Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19519' in order of occurrence in the config (last value wins).
[2020-07-01T15:38:35.257Z] runbld>>> Debug logging enabled.
[2020-07-01T15:38:35.257Z] runbld>>> Storing result
[2020-07-01T15:38:35.257Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-01T15:38:35.257Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:35.257Z] runbld>>> Adding system facts.
[2020-07-01T15:38:36.201Z] runbld>>> Adding vcs info for the latest commit:  bc31a955805ee892da6a6ebd2b5ee3a322506299
[2020-07-01T15:38:36.201Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-01T15:38:36.201Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-01T15:38:36.201Z] Processing JUnit reports with runbld...
[2020-07-01T15:38:36.201Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-01T15:38:36.461Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-01T15:38:36.461Z] runbld>>> DURATION: 18ms
[2020-07-01T15:38:36.461Z] runbld>>> STDOUT: 40 bytes
[2020-07-01T15:38:36.461Z] runbld>>> STDERR: 49 bytes
[2020-07-01T15:38:36.461Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-01T15:38:36.461Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:37.403Z] runbld>>> Storing build metadata: 
[2020-07-01T15:38:37.403Z] runbld>>> Adding test report.
[2020-07-01T15:38:37.403Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:38.343Z] runbld>>> Found 30 test output files
[2020-07-01T15:38:39.283Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4042 Skipped: 475
[2020-07-01T15:38:39.283Z] runbld>>> Storing result
[2020-07-01T15:38:39.283Z] runbld>>> FAILURES: 0
[2020-07-01T15:38:39.543Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-01T15:38:39.543Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:39.543Z] runbld>>> Email notification disabled by environment variable.
[2020-07-01T15:38:39.543Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-01T15:38:45.157Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519
[2020-07-01T15:38:45.302Z] [INFO] getVaultSecret: Getting secrets
[2020-07-01T15:38:45.388Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-01T15:38:46.500Z] + chmod 755 generate-build-data.sh
[2020-07-01T15:38:46.500Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5 ABORTED 6685973
[2020-07-01T15:38:46.500Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/steps/?limit=10000 -o steps-info.json
[2020-07-01T15:38:47.843Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/tests/?status=FAILED -o tests-errors.json

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Jun 30, 2020

CI failure should be fixed by #19521

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Jun 30, 2020

run tests please

ycombinator
ycombinator approved these changes Jul 1, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kaiyan-sheng kaiyan-sheng merged commit 60ea971 into elastic:7.x Jul 1, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_19423_7.x branch July 1, 2020 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator ycombinator approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport Team:Platforms Label for the Integrations - Platforms team [zube]: Done

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaiyan-sheng @elasticmachine @ycombinator @kwinstonix