Describe the bug
Once we started using chunk_id in our s3_object_key_format, fluentd started giving the following output in the config
2021-04-13 14:01:03.217825851 +0000 fluent.warn: {"message":"chunk key placeholder 'chunk_id' not replaced. template:${$.logging_path}/%Y/%m/%d/%{time_slice}_${chunk_id}.gz"}
To Reproduce
The config used for S3 with the output config mentioned below does the job of reproducing the error.
Expected behaviour
Unique chuck id being created for s3 log files
Your Environment
- Fluentd version:
fluentd 1.12.2
- Operating system:
NAME="Alpine Linux" ID=alpine VERSION_ID=3.12.5 PRETTY_NAME="Alpine Linux v3.12" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/"
- Kernel version:
4.14.225-169.362.amzn2.x86_64
Your Configuration
output: |-
<match **>
@type copy
<store>
@type s3
s3_bucket "logging-#{ENV['CLUSTER_ENV']}"
s3_region "#{ENV['AWS_REGION']}"
path ${$.logging_path}/%Y/%m/%d/
s3_object_key_format %{path}%{time_slice}_${chunk_id}.%{file_extension}
store_as gzip_command
<buffer time,$.CLOG.long_path>
@type file
path /fluentd/log/s3
timekey 600
timekey_wait 60
timekey_use_utc true
chunk_limit_size 256MB
total_limit_size 48GB
flush_at_shutdown true
overflow_action block
retry_forever true
retry_type periodic
</buffer>
<format>
@type json
</format>
</store>
Your Error Log
2021-04-12 16:08:47 +0000 [warn]: #0 failed to flush the buffer. retry_time=441 next_retry_seconds=2021-04-12 16:08:48 +0000 chunk="5bfc89a5262fb7daf14f2919e78de036" error_class=RuntimeError error="duplicated path is generated. use %{index} in s3_object_key_format: path = loadtester/sbx/system/2021/04/12/202104121550_.gz"
Additional context
s3_object_key_format %{path}%{time_slice}_%{hex_random}.%{file_extension} hex_random config instead of chunk_id works without any issue.
Describe the bug
Once we started using chunk_id in our s3_object_key_format, fluentd started giving the following output in the config
2021-04-13 14:01:03.217825851 +0000 fluent.warn: {"message":"chunk key placeholder 'chunk_id' not replaced. template:${$.logging_path}/%Y/%m/%d/%{time_slice}_${chunk_id}.gz"}To Reproduce
The config used for S3 with the output config mentioned below does the job of reproducing the error.
Expected behaviour
Unique chuck id being created for s3 log files
Your Environment
fluentd 1.12.2NAME="Alpine Linux" ID=alpine VERSION_ID=3.12.5 PRETTY_NAME="Alpine Linux v3.12" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/"4.14.225-169.362.amzn2.x86_64Your Configuration
Your Error Log
Additional context
s3_object_key_format %{path}%{time_slice}_%{hex_random}.%{file_extension}hex_random config instead of chunk_id works without any issue.