Cherry-pick of PR #22034 to 7.x branch. Original message:

What does this PR do?

This PR is to add awsfargate module to collect monitoring metrics from AWS ECS Fargate task management endpoint for each container running under the same task as the Metricbeat is running.

Note: ~internal~ecs~pause container is skipped.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

I created an AWS CloudFormation file to create a cluster, define IAM role, create a task definition, and start the actual service.

Create an Elastic cloud env for testing and store the credentials into AWS secret manager.

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_AUTH --secret-string elastic:XXX

and

aws --region us-east-2 ecretsmanager create-secret --name ELASTIC_CLOUD_ID --secret-string YYYY

After storing the credentials into AWS secret manager, copy this file locally and copy the ARNs of these two secrets and put them into AWS CloudFormation yml file under TaskDefinition -> ContainerDefinition.

aws --region us-east-2 cloudformation create-stack --stack-name test-metricbeat-deployment --template-body file://./module/awsfargate/cloudformation.yml --parameters 'ParameterKey=SubnetID,ParameterValue=<subnet-id>' --capabilities CAPABILITY_NAMED_IAM

Make sure to replace<subnet-id> with your own subnet in this command. Please go to Services -> VPC -> Subnets to find subnet ID to use. You can also add several more containers under the TaskDefinition section.

Then you should see your task definition and cluster getting created, and containers start running in a minute or two.

Make sure there is no error log for the container and then go to Elastic Cloud to check for events from this module.

Clean up after test:

aws cloudformation delete-stack --stack-name test-metricbeat-deployment

Sample Event Output

data.json

Screenshots

closes #22210