Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

This PR sets the full index name on startup. But for beats potentially running multiple days, the index must be set per event.

Plus, the PR creates a discrepancy between event timestamp and index name.

The problem is better solved with a configurable processor adjusting the events timestamp before passing the event to the output.

Something like

processors:
 - set_timezone:
     zone: "America/New_York
     fields: ["field1", "field2", ...]

??

yeah, something like date filter in logstash. If fields is empty and locale is available, adjust the events timestamp to the locale.

So this means that we always try to adjust the timestamp value independently from fields.
Had checked the date documentation.

After reading the docs from logstash i would prefer the following config

processors:
 - set_timezone:
     zone: "America/New_York /optional
     fields: ["field1", "field2", ...] /required
     when: 
          date:
               field1: "MMM dd yyyy HH:mm:ss" /required
     target: "field" /optional, default @timestamp

I think we have to add a new date condition to parse the date or is there similar to check for dates?

Or is it better to add a new parameter like layouts or something and then just try if we can parse with that layout

processors:
 - set_timezone:
     zone: "America/New_York /optional
     fields: "field1" /required
     layouts: ["MMM dd yyyy HH:mm:ss", ... , ....] /required
     target: "field" /optional, default @timestamp

Is there a reason we need this on the beats side? Assuming we use a field to set the time zone, could the conversion also done in Logstash?

I think the naming set_timezone is not ideally chosen. Maybe set_timestamp or something else. But i asked me same. Because to implement the same feature in two different products. I don't know 😕

In general if a feature is not required on the edge node and could be handled in LS or Ingest, that is the place I prefer to have it to keep beats as simple as possible.

Can we close this?

@maddin2016 Good question. I think we need a solution for it, the question is just where it happens in the stack. If the above described workaround works, I would suggest we close it?

@maddin2016 It's a good idea to add timezone to the event. We discussed between us, and we decided that we want the Beat to export timezone as optional. I am closing this PR as we will continue from here and add support for it in a separate PR.