Skip to content

in_tail take long time to found rotated files #2478

@metayd

Description

@metayd

related to issue #2477

Will there be any risk if I change read_more = true to read_more = false, such as losing record or other issues, just in my copy of in_tail source code?

We came across a problem here. We have ten log files, our log-generator writes record(0.5KB per record) to them by 1000 lines/s each and rotate it by renaming the file to another name(the inode changed but filename stay the same) if the size of the file reaches 100MB. We use in_tail to collect these files.

We found two problems.

  1. when the file is rotated, it takes a long time for in_tail to discover the new inode of the logfile, and after several rotate, it seems in_tail just lost some of the rotated files(rotate is quite fast)
  2. when starting the fluentd, I found it takes a long time for in_tail to start_watchers for these files when I look at the log of fluentd(I add some debug log after setup_watcher). And with the rotate issues, it seems fluentd finally lost some of the log files.

I'm not familiar with in_tail's performance, so maybe it's not in_tail's problem but the way I use it. The usage of CPU is not high and the flush speed is ok.

For the moment, I solve these two problems with change read_more = true to read_more = false(it take effects), but I'm not sure if there will be any risk with this change.

It would be very kind of you if you can give me some advice with these two problems.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions