You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 21, 2023. It is now read-only.
Right now, the shipper receives it's initial config in two pieces:
Output config (ES, queue settings) is received as a single output unit
gRPC settings (primarily the endpoint to use for the unit socket, and also TLS) are duplicated across multiple input units.
This is a bit of an awkward process for two reasons:
The shipper's event pipeline operates mostly as a single component, i.e, there's no reason for us to start the gRPC endpoint before we even have the config for the queue or the output, or shut down the gRPC while we keep the output working. This means that before we start the shipper, we have to wait for the config to arrive in pieces, and synchronize events across an otherwise asynchronous event channel.
We use the input units in a rather non-normal way. The gRPC endpoint settings we get from the input units is global and static for the lifetime of the shipper, which is at odds with how the elastic-agent API client API works, where the config connected to the unit has a lifetime explicitly stated by the expected state of the unit; the shipper must take a "set and forget" approach to the gRPC config which seems to be at odds with how the V2 API otherwise works.
There's two ways to make this a little cleaner:
Get all the event pipeline config from the output unit. This makes the shipper init code less complex, and makes it more clear what unit is responsible for the state of the shipper's event pipeline. Input configs would still be used for things like processors.
If for some reason we really thought that the shipper's main event pipeline really needed to be spread across multiple units, the gRPC settings should have their own dedicated unit. This allows us to at least use the V2 API in a more orthodox way, as opposed to grabbing a random input unit and treat its config as a static value.
This will probably generate some discussion on if the shipper should receive pipeline config across multiple units. As far as I can tell, there's only one use case for this: restarting the output while the server and queue remains up. Right now the shipper server isn't designed to operate like this, and I'm not sure how we should prioritize such a feature. Such a feature doesn't necessarily require multiple units, as the shipper could just diff the subsections of the config to see if only the output has changed. Having the shipper pipeline config across two units also creates a number of state questions: should we shut down the queue and output while the gRPC connection remains up? Does a failure of the queue/output also imply that gRPC is in a failed state, as no events can be ACK'ed? From the context of error reporting, will a user find this distinction meaningful?
Right now, the shipper receives it's initial config in two pieces:
This is a bit of an awkward process for two reasons:
There's two ways to make this a little cleaner:
This will probably generate some discussion on if the shipper should receive pipeline config across multiple units. As far as I can tell, there's only one use case for this: restarting the output while the server and queue remains up. Right now the shipper server isn't designed to operate like this, and I'm not sure how we should prioritize such a feature. Such a feature doesn't necessarily require multiple units, as the shipper could just diff the subsections of the config to see if only the output has changed. Having the shipper pipeline config across two units also creates a number of state questions: should we shut down the queue and output while the gRPC connection remains up? Does a failure of the queue/output also imply that gRPC is in a failed state, as no events can be ACK'ed? From the context of error reporting, will a user find this distinction meaningful?