Cherry-pick of PR #16263 to 7.x branch. Original message:

What does this PR do?

This PR is to add support for custom endpoint configuration in the AWS modules for Filebeat and Metricbeat.

AWS Service endpoints: https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html

Why is it important?

There are users running in AWS private cloud regions, which require endpoint URL in configuration to access AWS API. The ap-northeast-3 Region in Japan is not returned by Region enumeration APIs, such as EC2.describeRegions API. To define endpoints for this Region, custom endpoint needs to be used:

https://{service}.{region}.amazonaws.com

So the Amazon EC2 endpoint for this Region would be ec2.ap-northeast-3.amazonaws.com.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Use endpoint config with regions to collect metrics. For example, the config below is to only collect EC2 cloudwatch metrics from us-west-1 region under amazonaws.com endpoint.

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  endpoint: amazonaws.com
  regions: 
    - us-west-1
  metricsets:
    - ec2

Similarly, testing endpoint config parameter with Filebeat s3 input can use config like below:

filebeat.inputs:
  - type: s3
    queue_url:   https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks
    credential_profile_name: elastic-beats
    endpoint: amazonaws.com

For testing autodiscovery aws_ec2 provider, config below can be used:

logging.level: debug
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      endpoint: amazonaws.com
      templates:
        - condition:
            equals:
              aws.ec2.tags.created-by: "ks"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: kaiyan
              password: kaiyan

• Make sure every metricset under aws module is working for Metricbeat
• Make sure every fileset under aws module is working for Filebeat
• Make sure autodiscover aws_ec2 provider is working
• Test for resourcegroupstaggingapi with endpoint: make sure tags for each service are collected with endpoint config provided.
• Test with/without regions config parameter: make sure if there is no regions specified, then metrics from all regions should be collected.

Related issues

#16245