[Filebeat] Changes to text fields in elasticsearch module#10414
[Filebeat] Changes to text fields in elasticsearch module#10414ycombinator merged 7 commits intoelastic:masterfrom ycombinator:fb-ecs-text-fields
Conversation
|
Pinging @elastic/stack-monitoring |
|
jenkins, test this |
There was a problem hiding this comment.
One thing I spot now: Below the message fields seems to contain everything including timestamp. Is that expected?
For the took. The plan is to extract the unit later or just remove it?
There was a problem hiding this comment.
I didn't follow your question about the message field?
For the took, I think we should just the unit as we already have the actual numeric value in ms, which seems more useful? BTW, this is something we're doing in the case of the elasticsearch/server fileset as well:
There was a problem hiding this comment.
The point about message is that once the header has been parsed and moved to other fields, only the rest of the message should be left in message.
Right now, the message fields looks like an event.original, which is meant to contain the pristine initial copy of the event (if the user wants this).
For the took field, I would keep it until we're ready to interpret it, no? Right now event.duration is based on the took_millis (removed by the pipeline), for simplicity, which is often rounded to 0ms. Elasticsearch is too fast. But I think it's worth keeping the custom field with the more precise timing until we can leverage it better (or until the ES log format gives us nanoseconds).
webmat
left a comment
There was a problem hiding this comment.
I'm good with the text field changes.
However I'd keep the took custom field.
webmat
left a comment
There was a problem hiding this comment.
Code LGTM
You just need to update fields.go accordingly :-)
|
jenkins, test this |
|
jenkins, test this |
This PR is an offshoot of conversations and decisions made in #10372 w.r.t
textfields, but scoped to theelasticsearchmodule.