Skip to content

Memory usage about LineReader #20497

@wangtao334

Description

@wangtao334

Describe the enhancement:
LineReader reads data into a buffer, and then appends the buffer to LineReader's inBuffer, the max memory used by inBuffer depends on the max size of line. but LimitReader will truncate the data at last. So i think the max memory used by inBuffer should be tye max_bytes + harvester_buffer_size and no relation with the max size of line.

Describe a specific use case for the enhancement or feature:
we use filebeat to collect log, but sometimes we got some log files with the max line size over than 200M. sure we try to control memeory usage by setting max_bytes, but we still get OOM. as a tool, we don't want filebeat using a lot of memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:Services(Deprecated) Label for the former Integrations-Services teamenhancement

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions