Bug

While testing, I have noticed two issues for S3 ingestion:

1. It seems we can't specify a bucket with out a prefix: (i.e. if I want to ingest all files under ). Currently, we are not able to leave the prefix as empty.
2. The CLP can't ingest a single file, since the s3 config will require the input path to end with a "/".

I have missed those in #788.

The direct cause of such issue is that we tried to reuse the S3StorageConfig that was originally designed for output. The intension was to reduce code duplication and unnessary config conversion in the code. But if necessary, we might need to add a Ingestion specific S3StorageConfig.

CLP version

acefdcd

Environment

Ubuntu 22.04. But it's unrelated

Reproduction steps

Use the following input config.

logs_input:
  type: "s3"
  s3_config:
    region_code: "us-west-1"
    bucket: "private-clp-test-bucket"
    key_prefix: "logs/"
    aws_authentication:
      type: "profile"
      profile: "default"

Ingestion cmd: ./sbin/compress.sh clp-s/log.1.json --timestamp-key 't.$date'