8.15 no longer has the fix for AWS S3 endpoint handling that was present in 8.14
8.15:
|
if config.AWSConfig.Endpoint != "" { |
|
// Add a custom endpointResolver to the awsConfig so that all the requests are routed to this endpoint |
|
awsConfig.EndpointResolverWithOptions = awssdk.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (awssdk.Endpoint, error) { |
|
return awssdk.Endpoint{ |
|
PartitionID: "aws", |
|
URL: config.AWSConfig.Endpoint, |
|
SigningRegion: awsConfig.Region, |
|
}, nil |
|
}) |
|
} |
8.14:
|
if config.AWSConfig.Endpoint != "" { |
|
|
|
// Parse a URL for the host regardless of it missing the scheme |
|
endpointUri, err := url.Parse(config.AWSConfig.Endpoint) |
|
if err != nil { |
|
return nil, fmt.Errorf("failed to parse endpoint: %w", err) |
|
} |
|
|
|
// For backwards compat: |
|
// If the endpoint does not start with S3, we will use the endpoint resolver to make all SDK requests use the specified endpoint |
|
// If the endpoint does start with S3, we will use the default resolver uses the endpoint field but can replace s3 with the desired service name like sqs |
|
if !strings.HasPrefix(endpointUri.Hostname(), "s3") { |
|
awsConfig.EndpointResolverWithOptions = awssdk.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (awssdk.Endpoint, error) { |
|
return awssdk.Endpoint{ |
|
PartitionID: "aws", |
|
Source: awssdk.EndpointSourceCustom, |
|
URL: config.AWSConfig.Endpoint, |
|
SigningRegion: awsConfig.Region, |
|
}, nil |
|
}) |
|
} |
|
} |
In AWS, the endpoint field is supposed to act kind of like a "base url" where service URLs are built using the value in the endpoint field. So when the SQS client makes a request, an endpoint field of s3.us-east1.amazonaws.com is transformed into sqs.us-east1.amazonaws.com, etc.
The 8.15 code forces all endpoints to use the value in the endpoint field instead of relying on the resolver to use the endpoint to "build" each service's endpoint (s3, sqs, etc). In the example above, this would cause the SQS client to directly query s3.us-east1.amazonaws.com
Even the 8.14 code has an issue that crops up with some customers. We should likely switch to only using a custom endpoint resolver when a user explicitly tells us to, for example by introducing a new setting called "static endpoint" or something similar, that when set to true, sets the endpoint resolver as it is set currently. This would be a breaking change.
An alternative would be introducing a setting called, "dynamic_endpoint" or something similar which, when set, sets the endpoint field without using a resolver.
8.15 no longer has the fix for AWS S3 endpoint handling that was present in 8.14
8.15:
beats/x-pack/filebeat/input/awss3/input.go
Lines 51 to 60 in 2f0dda8
8.14:
beats/x-pack/filebeat/input/awss3/input.go
Lines 83 to 104 in 7b6cfad
In AWS, the
endpointfield is supposed to act kind of like a "base url" where service URLs are built using the value in the endpoint field. So when the SQS client makes a request, an endpoint field ofs3.us-east1.amazonaws.comis transformed intosqs.us-east1.amazonaws.com, etc.The 8.15 code forces all endpoints to use the value in the
endpointfield instead of relying on the resolver to use the endpoint to "build" each service's endpoint (s3, sqs, etc). In the example above, this would cause the SQS client to directly querys3.us-east1.amazonaws.comEven the 8.14 code has an issue that crops up with some customers. We should likely switch to only using a custom endpoint resolver when a user explicitly tells us to, for example by introducing a new setting called "static endpoint" or something similar, that when set to true, sets the endpoint resolver as it is set currently. This would be a breaking change.
An alternative would be introducing a setting called, "dynamic_endpoint" or something similar which, when set, sets the
endpointfield without using a resolver.