-
Notifications
You must be signed in to change notification settings - Fork 25.8k
Optimize tsdb data stream timestamp parsing if ingest pipeline is used #84046
Copy link
Copy link
Closed
Labels
:StorageEngine/Data streamsData streams and their lifecyclesData streams and their lifecycles>enhancementTeam:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.
Description
For tsdb data streams, the @timestamp field is used to determine to which backing index a data stream resolves to.
Currently in IndexAbstraction.DataStream#getWriteIndex(...) the source of the document is parsed in order to extract the @timestamp field. If an ingest pipeline is active then this parsing is redundant, because ingest parse the source of a document to into a map of maps in order for a pipeline to process it.
After the final pipeline has executed, ingest can extract the @timestamp field and attach it as String/Long to the IndexRequest class as a field. Then IndexAbstraction.DataStream#getWriteIndex(...) can check for whether that field has been set and if so use that parse it into a Instant instance.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
:StorageEngine/Data streamsData streams and their lifecyclesData streams and their lifecycles>enhancementTeam:Data Management (obsolete)DO NOT USE. This team no longer exists.DO NOT USE. This team no longer exists.
Type
Fields
Give feedbackNo fields configured for issues without a type.