Request

One of our users has requested the ability to ingest logs from S3-compatible storage services. However, the current implementation of the CLP package assumes that all S3 interactions are directed to AWS's default S3 endpoint (amazonaws.com).

This limitation restricts integration with other S3-compatible storage providers, which may use custom or non-AWS endpoints. It would be nice to add support for ingesting from custom S3 endpoints.

Possible implementation

Currently, when the user calls compress-from-s3.sh, the following steps occur:

1. The URL supplied by the user is parsed and validated. If the endpoint isn't amazonaws.com, an error is raised.

2. The URL is converted into an S3InputConfig object, which stores the region, bucket, and key of the input.

3. S3InputConfig is used to fetch all keys for a key prefix, if the user requested a key-prefix ingestion.

4. In the compression worker, S3InputConfig is used to generate a list of virtual-hosted style URLs, which is then passed to clp-s.

5. clp-s deserializes the virtual-hosted URLs using the S3Url class.

6. clp-s signs the url based on the fields in the S3Url class.

7. S3Url and the signature is serialized into another virtual-hosted URL.

Required changes

To support custom S3-compatible endpoints, we propose the following changes:

1. Add a new endpoint_url field to both the S3InputConfig and S3Url classes.

2. Most S3-compatible storage services is only accessible using path-style URLs. Therefore, clp-s would have to send the request using path-style URLs. As a solution, when the endpoint isn't amazonaws.com, both clp-s and the compression worker switch to path-style URLs.

These changes will enable the integration of custom S3-compatible endpoints while maintaining compatibility with AWS's default S3 endpoint.

Path style urls