Describe the enhancement:
LineReader reads data into a buffer, and then appends the buffer to LineReader's inBuffer, the max memory used by inBuffer depends on the max size of line. but LimitReader will truncate the data at last. So i think the max memory used by inBuffer should be tye max_bytes + harvester_buffer_size and no relation with the max size of line.
Describe a specific use case for the enhancement or feature:
we use filebeat to collect log, but sometimes we got some log files with the max line size over than 200M. sure we try to control memeory usage by setting max_bytes, but we still get OOM. as a tool, we don't want filebeat using a lot of memory.
Describe the enhancement:
LineReader reads data into a buffer, and then appends the buffer to LineReader's inBuffer, the max memory used by inBuffer depends on the max size of line. but LimitReader will truncate the data at last. So i think the max memory used by inBuffer should be tye max_bytes + harvester_buffer_size and no relation with the max size of line.
Describe a specific use case for the enhancement or feature:
we use filebeat to collect log, but sometimes we got some log files with the max line size over than 200M. sure we try to control memeory usage by setting max_bytes, but we still get OOM. as a tool, we don't want filebeat using a lot of memory.