Skip to content

Add awsfargate module for AWS ECS Fargate#22034

Merged
kaiyan-sheng merged 18 commits intoelastic:masterfrom
kaiyan-sheng:fargate_module
Nov 23, 2020
Merged

Add awsfargate module for AWS ECS Fargate#22034
kaiyan-sheng merged 18 commits intoelastic:masterfrom
kaiyan-sheng:fargate_module

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 20, 2020 •
edited
Loading

What does this PR do?

This PR is to add awsfargate module to collect monitoring metrics from AWS ECS Fargate task management endpoint for each container running under the same task as the Metricbeat is running.

Note: ~internal~ecs~pause container is skipped.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

I created an AWS CloudFormation file to create a cluster, define IAM role, create a task definition, and start the actual service.

Create an Elastic cloud env for testing and store the credentials into AWS secret manager.

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_AUTH --secret-string elastic:XXX

and

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_ID --secret-string YYYY

After storing the credentials into AWS secret manager, copy this file locally and copy the ARNs of these two secrets and put them into AWS CloudFormation yml file under TaskDefinition -> ContainerDefinition.

aws --region us-east-2 cloudformation create-stack --stack-name test-metricbeat-deployment --template-body file://./module/awsfargate/cloudformation.yml --parameters 'ParameterKey=SubnetID,ParameterValue=<subnet-id>' --capabilities CAPABILITY_NAMED_IAM

Make sure to replace<subnet-id> with your own subnet in this command. Please go to Services -> VPC -> Subnets to find subnet ID to use. You can also add several more containers under the TaskDefinition section.

Then you should see your task definition and cluster getting created, and containers start running in a minute or two.

Make sure there is no error log for the container and then go to Elastic Cloud to check for events from this module.

Clean up after test:

aws cloudformation delete-stack --stack-name test-metricbeat-deployment

Note: dashboard is added in #22941

Sample Event Output

data.json

Screenshots

Screen Shot 2020-10-21 at 7 21 23 PM

closes #22210

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 20, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 20, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 20, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #22034 updated]

  • Start Time: 2020-11-20T22:44:14.744+0000

  • Duration: 49 min 11 sec

Test stats 🧪

Test Results
Failed 0
Passed 2117
Skipped 476
Total 2593

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 2117
Skipped 476
Total 2593

@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels Oct 22, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 22, 2020
@sorantis
Copy link
Copy Markdown
Contributor

sorantis commented Oct 22, 2020

@kaiyan-sheng any reason this is a standalone module and note another metricset in aws module?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Oct 22, 2020

@sorantis awsfargate does not require any AWS credentials setup in the module configuration because it can only be run inside a fargate task in order to have access to ECS_CONTAINER_METADATA_URI_V4 ECS fargate task metadata endpoint. If we create a fargate metricset under current aws module, that means it requires the AWS credentials as a part of its configuration even though it's not gonna be used.

@sorantis
Copy link
Copy Markdown
Contributor

sorantis commented Oct 22, 2020

thanks for clarifying @kaiyan-sheng. Would this also translate to a standalone package?

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Oct 22, 2020

@sorantis I'm not sure on the package side. If it's one module map to one package, then this will be a standalone package. Do we have any package include more than one module?

@sorantis
Copy link
Copy Markdown
Contributor

sorantis commented Oct 22, 2020

i think it'll result in a separate package simply because it requires different configuration.

@ChrsMark
Copy link
Copy Markdown
Member

ChrsMark commented Oct 22, 2020

i think it'll result in a separate package simply because it requires different configuration.

👍

ChrsMark
ChrsMark reviewed Oct 22, 2020
View reviewed changes
@jsoriano jsoriano mentioned this pull request Oct 27, 2020
Open
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Nov 2, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 2633
Skipped 577
Total 3210

@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Nov 10, 2020 •
edited
Loading

TODO/Questions:

  • Add documentation
  • Add kibana dashboard ( Will be in a separate PR after this one is merged)
  • I added TestData function to create data.json file from the mock response body of task metadata endpoint. Is this OK? Do we have a function already to create data.json from mock data under metricbeat/mb/testing?
  • I copied several functions from docker module, maybe we can refactor them in a better way so I don't have to duplicate the code.
  • Should this module/metricset be in beta or experimental? -> beta

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review November 10, 2020 00:12
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Nov 10, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

ChrsMark
ChrsMark reviewed Nov 10, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good already! left some comments for now!

ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark reviewed Nov 18, 2020
View reviewed changes
ChrsMark
ChrsMark approved these changes Nov 23, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@kaiyan-sheng kaiyan-sheng mentioned this pull request Nov 23, 2020
Closed
@kaiyan-sheng kaiyan-sheng merged commit e8da544 into elastic:master Nov 23, 2020
@kaiyan-sheng kaiyan-sheng deleted the fargate_module branch November 23, 2020 17:58
@kaiyan-sheng kaiyan-sheng mentioned this pull request Nov 23, 2020
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng added v7.11.0 test-plan Add this PR to be manual test plan and removed needs_backport PR is waiting to be backported to other branches. labels Nov 23, 2020
kaiyan-sheng pushed a commit that referenced this pull request Nov 24, 2020
Cherry-pick #22034 to 7.x: Add awsfargate module for AWS ECS Fargate (#…
ed3ad27 
…22720)

* Add awsfargate module for AWS ECS Fargate (#22034)

* Add awsfargate module for AWS ECS Fargate

(cherry picked from commit e8da544)

* run mage fmt update
@kaiyan-sheng
Copy link
Copy Markdown
Contributor Author

kaiyan-sheng commented Nov 24, 2020

Dashboard is added in a separate PR: #22745

@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.11.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Metricbeat support for monitoring AWS Fargate

5 participants

@kaiyan-sheng @elasticmachine @sorantis @ChrsMark @andresrc