Use case

We have a lot of servers with up to 300 application deployed on each server (mostly Java/Scala). Each application writes log files using log4j library. Each log file is rotated daily in the same directory, with up to 10 back up files possible. Events are fed to realtime processing system(critical to time delays up to 1 second) and to HBase. We want to replace current heavy log collector with lightweight solution.

Problem

We tried to use default tail plugin. Most basic configuration will look like this:

<source>
  @type tail
  read_from_head true
  path /var/log/*/app*.log  
  tag app.log.*
  format none
  refresh_interval 10
  pos_file /var/log/pos/app.log.pos
  rotate_wait 3600
  path_key path
  limit_recently_modified 86400
</source>

According to fluentd in_tail plugin documentation - every time file is rotated for such configuration, it will be treated by in_tail plugin as new one, and will be read from it's head, even though file inode has not changed, and in position file we have latest read position (but just for last line before rotation; during rotate_wait time position stored in memory only - https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_tail.rb#L554)

This doesn't suit our needs, because:

• we will generate a lot of unneeded traffic, which will be irrelevant to realtime part
• disk I/O increase
• longer ingestion into HBase

One of the possible solution proposed by @repeatedly in #1613 is to avoid '*' in file pattern discovery. This also doesn't suit us, because we'd like to reliably store events with no or minimum loss because they are very crucial for business analysis. Possible data loss scenario:

1. Fluentd goes down at 23:45 UTC
2. Log file is rotated at 00:00:00 UTC, app.log is renamed to app.2017-08-02.log (just for example)
3. Fluentd goes up at 00:15 UTC
4. Tail plugin reads all data from in new app.log file
5. Data from app.2017-08-02.log file from 23:45 - 00:00 UTC is lost. No way for us to find out at least latest offset in app.2017-08-02.log because it has been overwritten in position file.

Proposed solution

Add ability to track inodes inside fluentd in_tail plugin to enable combination of '*' in path configuration with rotation inside same directory and read_from_head set to true. In proposed solution we add follow_inodes flag (set to false by default to preserve current behavior). When follow_inodes is set to true and file rotation is detected - we do not reread file, instead we are checking if such inode is present inside pos file - if yes, we don't recreate TailWatcher. TailWatcher is closed after rotate_wait period and file marked as unwatched inside position file after limit_recently_modified time passed. Pos file is cleaned on startup as it was before, but also delete entries for non-existing files.

Please review, and let me know what do you think of such approach?