[WIP] Adapting ingest pipelines to new date parsing in ES#10543
[WIP] Adapting ingest pipelines to new date parsing in ES#10543ycombinator wants to merge 2 commits intoelastic:masterfrom ycombinator:fb-es-adapt-pipelines-time-parsing
Conversation
| @@ -1,5 +1,6 @@ | |||
| [ | |||
| { | |||
| "@timestamp": "2019-01-30T14:16:20.233Z", | |||
There was a problem hiding this comment.
Good news: the @timestamp field is back!
Bad news: if you compare the value of this field vs. the value in the raw log line that generated it, you'll notice that the time zone in this field, Z, is incorrect. It should be -0800 instead. Alternatively, the time here should be 22:16:20.233Z.
There was a problem hiding this comment.
Same goes for all other @timestamp fields in this PR that are being generated from timestamps with time zones in the raw logs.
There was a problem hiding this comment.
We definitively need support back for the time zones.
|
@webmat @spinscale @ruflin WIP / discussion PR to demonstrate the challenges with trying to adapt Filebeat ingest pipelines to work with the new timestamp parsing logic in ES. |
|
This PR is no longer necessary as the regressions in ES have been fixed in ES itself! Closing unmerged. |
Elasticsearch recently changed how it parses timestamps, which introduced a regression. Ideally, the regression can be fixed but this PR is the "backup plan".
This PR tries to adapt Filebeat's ingest pipelines to work with both the old timestamp parsing logic in Elasticsearch (in case users run Filebeat with an older version of ES) as well as the new parsing logic.
As a demonstration, at the moment this PR only adapts the pipelines in the
elasticsearchFilebeat module.