This is a feature meta issue to allow filebeat and metricbeat to publish data to the shipper when run under Elastic agent. All other beats are out of scope.

An output for existing beats should be implemented that publishes to the shipper gRPC interface. When the shipper gRPC output is used, the beat output pipeline should be configured to be as simple as possible. Using a per beat disk queue with the shipper is forbidden. A memory queue may be used with the shipper output, but how it should be configured by users will require careful consideration. Ideally any necessary queue configuration can be made automatic.

Removing processors from beats is out of scope for this issue. Processors will be removed in a later issue.

This feature is considered complete when at least the following criteria are satisfied for both filebeat and metricbeat:

• A test exists proving data ingested by the beat is published to the shipper.
• A test exists proving there is no data loss when the shipper process restarts while the beat is publishing.
• A test exists proving there is no data loss when the shipper backpressures the beat (because the shipper queue is full for example).

The assignee of this issue is expected to create the development plan with all child issues for this feature. The following set of tasks should be included in the initial issues at a minimum:

• Creating a beats output that that publishes to the shipper gRPC interface.
• Defining a standard configuration for using a beat with the shipper that the control plane can easily apply: processors disabled, queues disabled, etc.
• Creating an integration test suite for the beat and shipper interactions.

UPD by @rdner

I split this in the following steps:

• Implement and document the basic experimental shipper output type #22
• Add integration tests for the shipper output #23
  • An event batch is published from an input to the shipper gRPC server
  • An event batch is not dropped when the gRPC server is not available but starts later
  • An event batch is not dropped when ResourceExhausted code is returned from the gRPC server, TTL does not decrease in this case
• Add support for input and data stream identification when sending events #24
• Implement the shipper's gRPC API #34