Skip to content

[libbeat] add_cloud_metadata - startup blocked by AWS IMSDv2 token fetch #33058

@andrewkroh

Description

@andrewkroh

The AWS provider in the add_cloud_metadata processor was modified to support IMSDv2 in #28285. There are two issues affecting the implementation.

  1. Fetching of the IMSDv2 token blocks the startup of the processor. The processor was designed to try fetching metadata from all enabled providers (e.g. aws, gcp, openstack) in parallel and asynchronously from the main thread. This allows the Beat to startup without being blocked on Cloud provider detection logic.

    The implementation fetches the IMSDv2 token in a non-async manner such that it blocks startup and does not happen in parallel with all other provider implementations. In non-AWS environments the Beat blocks until a HTTP timeout is reached (default is 3s).

    The processor needs to be modified to fetch the IMSDv2 token asynchronously. Some changes might be needed to allow chaining the request to fetch the token and use the token in the AWS metadata service requests.

  2. (update: this should be fixed by [add_cloud_metadata] Remove logger for AWS/EC2 #36829) It logs warnings for something that is expected to fail when the Beat runs outside of AWS. This should not be logged as a warning. It should be handled in the same manner as a non-response is handled by all other provider implementations. If the error is returned in the result from the provider then the processor will log the disposition at the debug level.

{"log.level":"warn","@timestamp":"2022-09-12T17:09:41.120-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions