Implement an alternative log file collector (Name: logfile) for use with the agent based on the v2 input API.

The v2 input API allows file state to be stored in a generic key-value store, while requiring the Input type to implement all required logic for state and ACK handling. A generic implementation for stateful inputs is available in the filebeat/input/v2/input-cursor package (currently requires a static set of sources only).

Tasks

Phase 1 (feature-parity)

• create a special stateful input for the new logfile
• add harvesters to an input dynamically
• watch for file events
  • create an interface that is responsible for watching changes in the input files
  • refactor the current scanner so it implements the new interface

Missing features

• JSON reader
  • add it again
  • rename it to NDJSON
• multiline reader
• configurable cleaner options

Phase 2 (enhancements)

• make the input log rotation aware Add support for copytruncate method when rotating input logs with an external tool in filestream input #23457
• come up with a strategy for handling backpressure (high effort) Add support to combine update operations #25976
• address globbing issues on Windows (medium effort)
• add detailed metrics Support summary and detailed file metrics in filestream input #25045

Inputs based on log

• container logs reader (low effort) Add new parser to filestream input: container #26115

Marking it GA

• mark filestream input GA Make filestream input GA #26127
• write blogpost about it before release
• deprecate log input

- [ ] remove log input in 8.x

Related: #15324