Create a prototype input for Filebeat that implements the shipper API. The intent is to implement the data shipper as another instance of Filebeat so that we can completely reuse the existing Beat event pipeline and outputs.
The shipper input will be Elastic licensed and should be added to the https://github.com/elastic/beats/tree/main/x-pack/filebeat/input directory.
The shipper input should ideally start a single gRPC server and create a new beat.Client for each unique data stream. This will allow each data stream to have its own processor configuration and allows for shared processor pipelines to execute concurrently. This follows the pattern used in the AWS S3 input in #33658.
We can explore other approaches if the ideal approach is initially too complex. The path of least resistance will likely be to have one instance of the shipper input for each input unit received from the Elastic Agent, where there is an input unit per component connected to the shipper. This will result in multiple gRPC server instances, but given that the shipper gRPC communication uses Unix domain sockets / named pipes this may be an acceptable overhead.
Acceptance Criteria
Create a prototype input for Filebeat that implements the shipper API. The intent is to implement the data shipper as another instance of Filebeat so that we can completely reuse the existing Beat event pipeline and outputs.
The shipper input will be Elastic licensed and should be added to the https://github.com/elastic/beats/tree/main/x-pack/filebeat/input directory.
The shipper input should ideally start a single gRPC server and create a new
beat.Clientfor each unique data stream. This will allow each data stream to have its own processor configuration and allows for shared processor pipelines to execute concurrently. This follows the pattern used in the AWS S3 input in #33658.We can explore other approaches if the ideal approach is initially too complex. The path of least resistance will likely be to have one instance of the shipper input for each input unit received from the Elastic Agent, where there is an input unit per component connected to the shipper. This will result in multiple gRPC server instances, but given that the shipper gRPC communication uses Unix domain sockets / named pipes this may be an acceptable overhead.
Acceptance Criteria