Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Describe the bug
I received a log message like below
2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted
event stream: 574180bytes
But I set to chunk_limit_size is 8M and chunk_full_threshold is 0.8.
8M*0.8 = 6.4M
chunk has free space more than 1.6M but log message say 574K.
I checked the chunk and realized that there are many records.
The message received was a bit unfriendly.
To Reproduce
- set large
chunk_limit_size
- set small
chunk_limit_records
- start fluentd
- push large number of minimum records
- show warn message
Expected behavior
2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk size limit exceeds for an emitted
event stream: 1400records
Your Environment
- Fluentd or td-agent version:
fluentd --version or td-agent --version
$ td-agent --version
td-agent 1.11.1
- Operating system:
cat /etc/os-release
$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
$ uname -r
4.14.177-139.254.amzn2.x86_64
If you hit the problem with older fluentd version, try latest version first.
Your Configuration
<source>
@type forward
@id input_forward
port 24224
bind "0.0.0.0"
</source>
<filter **.es>
@type prometheus
<metric>
name fluentd_input_status_num_records_total
type counter
desc The total number of incoming records
<labels>
tag ${tag}
hostname ${hostname}
</labels>
</metric>
</filter>
<match *.*.es>
@type elasticsearch
@id rawlog2elasticsearch
hosts "xxxx"
log_es_400_reason true
reconnect_on_error true
reload_on_failure true
reload_connections false
logstash_format true
logstash_prefix "raw-${tag[1]}-${tag[0]}"
<buffer tag>
@type "file"
path "/data/buffers/raw"
chunk_limit_size 8M
total_limit_size 1G
chunk_limit_records 1000
chunk_full_threshold 0.8
flush_mode interval
flush_interval 60s
flush_thread_count 1
retry_forever true
retry_type periodic
retry_wait 30
</buffer>
</match>
Your Error Log
2020-07-13 04:59:59 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 474709bytes
2020-07-13 05:00:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 553495bytes
2020-07-13 05:04:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 603165bytes
2020-07-13 05:04:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 551115bytes
2020-07-13 05:06:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 581636bytes
2020-07-13 05:07:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 562004bytes
2020-07-13 05:07:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 572290bytes
2020-07-13 05:11:10 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 627635bytes
2020-07-13 05:12:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 460620bytes
2020-07-13 05:13:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 485934bytes
2020-07-13 05:14:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 464083bytes
2020-07-13 05:16:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 466951bytes
Additional context
Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Describe the bug
I received a log message like below
But I set to chunk_limit_size is 8M and chunk_full_threshold is 0.8.
8M*0.8 = 6.4M
chunk has free space more than 1.6M but log message say 574K.
I checked the chunk and realized that there are many records.
The message received was a bit unfriendly.
To Reproduce
chunk_limit_sizechunk_limit_recordsExpected behavior
Your Environment
fluentd --versionortd-agent --versioncat /etc/os-releaseuname -rIf you hit the problem with older fluentd version, try latest version first.
Your Configuration
Your Error Log
Additional context