Experimental, Beta, and GA Criteria

• [Meta] elastic-agent-shipper journey to GA #197

Blockers for Beta Milestone 1 - Elasticsearch output and memory queue

NEW - Convert the shipper to a Filebeat input

• [Elastic Agent] Prototype a shipper gRPC input for Filebeat beats#35135
• Support the shipper as an instance of Filebeat configured to use the shipper protocol input elastic-agent#2521
• Allow queue configuration to be specified under the output type beats#35615 (enables setting shipper queue config via agent policy)
• Each shipper input unit should have a dedicated unix socket for inter-process communication elastic-agent#2581
• Add elastic_agent.shipper to the list of known monitoring datasets in Fleet elastic-agent#2654
• Update the shipper monitoring implementation to be a variant of Beat monitoring elastic-agent#2580
• [Elastic Agent] Implement the shipper PersistedIndex API in the Filebeat shipper input beats#35266
• [Elastic Agent] Implement input metrics for the shipper Filebeat input beats#35267
• Archive this repository and remove the elastic-agent-shipper binary from the release process #306
• Allow queue configuration to be specified under the output type beats#35615

Bugs

• github.com/elastic/beats/v7/libbeat/outputs/shipper tests are flaky beats#34984
• The shipper should report itself as FAILED when multiple outputs are configured. #171
• TestPersistedIndex/server_should_properly_shutdown is flaky #300
• Fail to start if the shipper is enabled but the shipper executable cannot be found elastic-agent#2446
• Disk queue re-sends last segment file when there is more than on segment on disk beats#32560
• Fix mage test:integration #244
• Fix so shipper doesn't stall while sending via elasticsearch output #255
• Fix elasticsearch output adding bulk index item #253
• include @timestamp in events sent via elasticsearch output #256
• Shipper output fails on large events/batches beats#34695
• Shipper output retries forever on all errors regardless of actual retry settings beats#34700
• Beats pipeline doesn't respect configured batch sizes on startup under agent beats#34703
• The Elasticsearch output should not report itself as degraded based only on the time between events #301

Features

• Implement design to handle queued events on policy update #286
• Investigate way to prove events ingested with & without shipper are functionally the same #257
• Support monitoring the shipper from the Elastic Agent #267
• Beats should use the proxy queue when the shipper output is active beats#34396

Elasticsearch Output V2

• Split large batches of documents if received 413 from Elasticsearch beats#29778
• Shipper doesn't support infinite retry / robust failure handling #262
• Allow the Elasticsearch output to load balance between multiple hosts #152
• Support output proxy configuration #172
• Expose go-elasticsearch BulkIndexer output performance metrics #153
• Decide what the default shipper Elasticsearch output configuration should be #291
• Tune / finalize the queue's batch and timing parameters #58

Monitoring

• Add a queue utilization metric to report how full the queue is as a percentage #265
• Add a counter for the number of bytes transmitted by the shipper #271
• Elastic agent metric dashboard for the shipper #54

Performance

• Onboard the shipper into the automated ingest performance testing framework #266
• The shipper event protocol should be a wrapper for JSON bytes #288

Testing

• Add document equivalency test case to end to end tests #290
• Add integration tests between the Beats shipper client and the shipper server beats#33205
• Add an end to end test suite for the shipper. #173
• Add integration tests for end to end acknowledgement #264

Technical Debt

• Move batch assembly / acknowledgment out of the Beats queue #186
• Shipper configuration tests are run against invalid configurations elastic-agent#2147
• Config parameters aren't passed through to the shipper output in Beats elastic-agent#2351
• Combine the shipper client into https://github.com/elastic/elastic-agent-client elastic-agent-shipper-client#20
• Meta Issue: Shipper configuration and I/O #226
  • Namespace config values in fleet output elastic-agent#1729
  • Input configs from units should hard-code shipper connection settings elastic-agent#1744
  • Standardize the shipper's gRPC TLS setup process #224
  • Look into changing how the shipper recieves gRPC connection config #225
• Investigate use of vtprotobuf to generate the shipper gRPC API #263

Documentation

• Document the shipper process and how it interacts with agent policy:
• Document the shipper feature flag and the current stability guarantees.
• Document why a user would want to enable the shipper.
• Document how the number of shippers (and therefore output connections) can be controlled via the agent policy.
• Document configuration of the shipper queues.
• Document any backwards compatibility or migration issues when moving from per beat outputs and queues to the centralized shipper.

Future Feature Milestones

Kafka Output

• [Fleet] Implement Kafka output form UI kibana#143324
• Make Kerberos support GA for Kafka output #168
• Performance test the shipper Kafka output #232
• Add "real" selector support for kafka output #203
• Add support for metrics observers, add output metrics:

  elastic-agent-shipper/output/kafka/client.go

  Line 41 in de8cd74

  //observer outputs.Observer TODO: what to do with observers?

• Add support for the retrying failed events:

  elastic-agent-shipper/output/kafka/client.go

  Lines 48 to 49 in de8cd74

  	failed []*messages.Event // TODO: Need to know how to deal with failed events
  	// batch publisher.Batch // and how to retry after a batch is complete.

• Support event codecs:

  elastic-agent-shipper/output/kafka/client.go

  Line 63 in de8cd74

  writer codec.Codec, // TODO: Proper codec support

• Support partition caching:

  elastic-agent-shipper/output/kafka/partition.go

  Lines 122 to 125 in de8cd74

  	// TODO: Do we need to cache the partition?
  	//if _, err := msg.data.Cache.Put("partition", partition); err != nil {
  	// return 0, fmt.Errorf("setting kafka partition in publisher event failed: %v", err)
  	//}

• Update sarama to the latest version
• Compatibility testing between existing Beats Kafka output configurations and the shipper Kafka output implementation.

Logstash Output

• Implement the Logstash output #129
• Add support for configuring the retry algorithm:

  elastic-agent-shipper/output/logstash/config.go

  Line 29 in de8cd74

  // Backoff Backoff `config:"backoff"` TODO

• Support configuring bulk max size:

  elastic-agent-shipper/output/logstash/config.go

  Line 21 in de8cd74

  // BulkMaxSize int `config:"bulk_max_size"` TODO

• Support load balancing between hosts:

  elastic-agent-shipper/output/logstash/config.go

  Line 20 in de8cd74

  // LoadBalance bool `config:"loadbalance"` TODO

• Support pipeline with the async client:

  elastic-agent-shipper/output/logstash/config.go

  Line 25 in de8cd74

  // Pipelining int `config:"pipelining" validate:"min=0"` TODO support async client

• Support slow start mode:

  elastic-agent-shipper/output/logstash/config.go

  Line 22 in de8cd74

  // SlowStart bool `config:"slow_start"` TODO

• Add event codec support: https://github.com/cmacknz/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/encode.go#L11-L12
• Compatibility testing between existing Logstash output configurations and the shipper Logstash output implementation.
• Performance test the Logstash output.

Global Processors

• Synchronize the shipper configuration model with what is implemented in the agent #161
• Prototype migrating a single processor from a beat to the shipper elastic-agent#1688
• Migrate the default global Beat processors to the shipper #292
• Migrate all processing from the inputs to the shipper elastic-agent#1689
• Improve processor error handling elastic-agent#1690
• Allow configuring global processors that apply to all events: https://github.com/elastic/ingest-dev/issues/2442.

Endpoint Security Support

• A misbehaving input should not be able to flood the shipper and prevent publishing the events of other inputs #213
• Enable mTLS on connections between inputs and the shipper #275
• Require use of an encrypted disk queue when Elastic Defend is installed

Disk Queue

• enrich diskqueue configuration with encryption_password when disk queue is encryption is enabled elastic-agent#1656
• Add disk queue options and operation to documentation #122
• The disk queue must assign an auto-incrementing ID to each element #27
• Immediately acknowledge events persisted to the disk queue when an input is subscribed to the acknowledgement stream.
• The disk queue must assign an auto-incrementing ID to each element #27
• Correctly handle enqueued events affected by agent policy changes #49
• Add explicit memory usage cap for the disk queue beats#30069
• Add disk queue metrics to shipper metrics #125
• Implement queue metrics for the disk queue #145

Future Enhancements - Not Yet Planned

• Explore an autoscaling Elasticsearch output based on the APM Server autoscaling work #175
• Investigate changing the pipeline order to improve performance #44
• Deadletter indexing should be taken care of automatically #245