Add processor to set timezone for an event by martinscholz83 · Pull Request #3902 · elastic/beats

martinscholz83 · 2017-04-04T12:00:17Z

This PR add a new processor to set timzone for an event. See #3867

elasticmachine · 2017-04-04T12:02:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2017-04-04T12:02:08Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

andrewkroh · 2017-04-04T12:48:04Z

libbeat/processors/add_locale/add_locale.go

+
+func newAddLocale(c common.Config) (processors.Processor, error) {
+	config := struct {
+		TimeZone string `config:"timezone"`


Should this just be completely automatic using time.Now().Location()? Example: https://play.golang.org/p/2Lpzo3KvWJ

That's what i not understand correctly. I thought we give the user the option to set the timezone. If not so you are right.

I think idea is to report the local timezone. It probably shouldn't be configurable at all. Just detect the local TZ and report it in the event.

Once you have the timezone, you can use Logstash to do further transformations on the data (like interpreting syslog dates that are missing the timezone) or converting @timestamp to the machine's local timezone (not recommended IMHO).

A more specific use case is analyzing logon events, the timezone can be used to determine if the event occurred during normal working hours. Then you can alert on people working after hours and send them on a vacation.

It looks like there's an even more direct way: time.Local.String().

time.Local.String() only gives you Local. I have changed the method to get the the timezone

Hmm, am I misunderstanding the docs at https://golang.org/pkg/time/#Location ?

Local represents the system's local time zone.

var Local *Location = &localLoc

On golang Playground i get UTC. On my machine i get

"beat": { "hostname": "4201halwsd00001", "name": "4201halwsd00001", "timezone": "Local", "version": "6.0.0-alpha1" },

???

Yep, you are right. I get the same thing "Local". So zone, _ := time.Now().Zone() should be sufficient because Now() always returns local time.

Ok. Changed up to time.Now().Zone(). I was not sure if Now() always returns local time.

andrewkroh · 2017-04-04T12:55:29Z

libbeat/processors/add_locale/add_locale.go

+}
+
+func (l addLocale) Run(event common.MapStr) (common.MapStr, error) {
+	zone, err := time.LoadLocation(l.timezone)


The Run method should be as simple and fast as possible. In newAddLocale() you I think can do this work once and store the result so that Run just becomes a map.Put() operation.

andrewkroh · 2017-04-04T13:33:35Z

libbeat/processors/add_locale/add_locale.go

+}
+
+func newAddLocale(c common.Config) (processors.Processor, error) {
+	config := struct {


This whole config definition and the Unpack call can now be removed.

andrewkroh

👍

martinscholz83 · 2017-04-04T14:02:32Z

Can we use this for the changelog?
Add new add_locale processor to export the local timezone with an event

ruflin · 2017-04-04T14:23:22Z

@maddin2016 Changelog entry LGTM. Don't forget to add the PR number at the end (see other lines)

andrewkroh · 2017-04-04T14:25:51Z

The config.full.yml files should mention the new processor. https://github.com/elastic/beats/blob/master/libbeat/_meta/config.full.yml#L45-L46

ohadravid · 2017-04-05T07:43:23Z

Hi,
I see that the locale is only read once on the beat init (if I understand the code correctly).
However, we usually want to track long-running servers which might go between DST and regular time, and the change won't be reflected if we use the plugin as it as.

martinscholz83 · 2017-04-05T07:56:52Z

Hi @ohadravid. For now we only export the timzone as a name CET, UTC, .... Maybe in the future we add an option to export the local time to a field, let say localtime. And then, you are right we have to move that logic into the Run method. @andrewkroh, it seems that there is a need for such an option to export the local time. What do you think to add this option now.

martinscholz83 · 2017-04-05T08:02:26Z

@ohadravid sorry 🙈 . You are right. Now we have CEST instead of CET.

ruflin · 2017-04-05T08:54:14Z

Servers traveling in time ... 🙈 We should report the time zone without DST.

BTW this reminds me we will need to add docs to show people what the output of this will be and how to configure it.

ohadravid · 2017-04-05T09:07:32Z

I think it will be more generic to just re-read the value at Run, since I do want to know the server is (or isn't) in DST (The fact that I know that CET should translate now to CEST does not mean the server really moved to the DST timezone).
It also affects computers which move between timezones (like laptops abroad).

martinscholz83 · 2017-04-05T10:03:17Z

Tested locally and timzone is exported as CEST. So i think that is fine, @ruflin?

ohadravid · 2017-04-05T11:32:26Z

@maddin2016 consider these two cases:

A server moves to DST.
We install the beat and start it, it sends CET with each event.
The server then moves to daylight savings times.
The beat will still send events with CET (until the beat/server are restarted).
A laptop which travels abroad.
We install the beat, it sends CET with each event.
The owner of the laptop fly to New York, change the timezone to EST.
The beat will still send events with CET (until the beat/laptop are restarted).

To me it is very important we re-read the timezone value on every event, so we can provide reliable data about the event's time as much as we can (For example, we might want to search for logins at odd hours (middle of the night) on the laptop).

Also, it's really nice to be able to see and be a part of the development effort. Thank you for making this so transparent :)

ruflin · 2017-04-05T11:48:02Z

@ohadravid Thanks for the inputs. Laptops moving from one time zone to an other is definitively something I was not thinking of. Just changed my opinion about the above then :-) @andrewkroh WDYT?

In case the processor is enabled, that means it is "refreshed" every time an even is created which can be up to 100k/s. In case we hit some issue here in the future we can think about caching it at least for 1s or something similar, but lets hit that wall first.

ruflin · 2017-04-05T11:48:40Z

libbeat/docs/processors-config.asciidoc

+- add_locale:
+-------------------------------------------------------------------------------
+
+NOTE: Please consider that `add_locale` differentiate between DST and regular time.


s/differentiate/differentiates/

andrewkroh · 2017-04-05T15:33:57Z

The beat.timestamp field needs to be added to the common fields.yml at https://github.com/elastic/beats/blob/master/libbeat/_meta/fields.common.yml#L8-L20.

andrewkroh · 2017-04-05T15:38:12Z

With changes due to daylight savings and laptops moving around the world, making the timezone automatically update is a requirement. So let's do that. But I would like to understand the impact. @maddin2016 could you run two benchmarks, one as the code is now, and one where it reads the timezone on each Run() and post the results. If you don't have time I can do it at a little later.

martinscholz83 · 2017-04-05T16:03:39Z

Here are the results. The first value is as the code is now. And the second is where it reads the timezone on each run

func BenchmarkConstruct(b *testing.B) {
	var testConfig = common.NewConfig()

	input := common.MapStr{}

	p, err := newAddLocale(*testConfig)
	if err != nil {
		b.Fatal(err)
	}

	for i := 0; i < b.N; i++ {
		_, err = p.Run(input)
	}
}

ruflin · 2017-04-06T08:12:28Z

@maddin2016 Thanks for the benchmarks. I would suggest to keep the benchmark code in the tests as it will be also useful to have these in the future.

BTW: Don't forget make update 😆

martinscholz83 · 2017-04-06T11:37:36Z

What about to move the logic into the run method?

ruflin · 2017-04-06T13:04:11Z

@maddin2016 Which logic are you referring to?

martinscholz83 · 2017-04-06T14:14:55Z

Sorry. I meant the read for timezone.

andrewkroh · 2017-04-06T22:53:06Z

libbeat/processors/add_locale/add_locale.go

+}
+
+func (l addLocale) Run(event common.MapStr) (common.MapStr, error) {
+	event.Put("beat.timezone", l.timezone)


Given the requirement to have this field automatically reflect the latest timezone, let's not cache the timezone like I requested previous (sorry for the change) and make Run always call time.Now().Zone(). Based on the benchmark results there wasn't a huge cost to doing this.

martinscholz83 added 2 commits April 4, 2017 12:14

Add add_locale processor

a45355d

register processor

b2ad649

Add tests

88cc4e7

andrewkroh reviewed Apr 4, 2017

View reviewed changes

martinscholz83 added 2 commits April 4, 2017 15:24

Change handling of getting local timezone

7d2782e

change test

7f84cb3

andrewkroh reviewed Apr 4, 2017

View reviewed changes

martinscholz83 added 2 commits April 4, 2017 15:44

Remove unnecassary config

dd74400

Simplified get zone

eb4dea6

andrewkroh approved these changes Apr 4, 2017

View reviewed changes

andrewkroh added :Processors libbeat review v6.0.0-alpha1 labels Apr 4, 2017

martinscholz83 force-pushed the add_locale branch from 8d3f1b3 to b214b2d Compare April 5, 2017 06:00

Update changelog

a06dd06

martinscholz83 force-pushed the add_locale branch from b214b2d to a06dd06 Compare April 5, 2017 06:04

Add description for config.asciidoc

e6353be

ruflin reviewed Apr 5, 2017

View reviewed changes

Fix grammar config.asciidoc

6914c9a

Add timezone to fields.common.yml

ed01967

ruflin mentioned this pull request Apr 6, 2017

Metricbeat index date timezone #3744

Closed

andrewkroh reviewed Apr 6, 2017

View reviewed changes

martinscholz83 added 2 commits April 7, 2017 08:35

Fetch timezone in run method

94dca42

Add benchmark test

f5f4b4c

martinscholz83 force-pushed the add_locale branch from 40efa97 to f5f4b4c Compare April 7, 2017 08:03

ruflin approved these changes Apr 7, 2017

View reviewed changes

ruflin mentioned this pull request Apr 7, 2017

Filebeat 'system' module assumes UTC in ingest pipeline #3898

Closed

andrewkroh approved these changes Apr 7, 2017

View reviewed changes

andrewkroh merged commit a66a719 into elastic:master Apr 7, 2017

This was referenced Apr 10, 2017

Add timezone metadata #2887

Closed

Add timezone to the exported event #3974

Closed

talevy mentioned this pull request Apr 10, 2017

Ingest Date Processor should support templated timezone/locale elastic/elasticsearch#24024

Closed

Conversation

martinscholz83 commented Apr 4, 2017

Uh oh!

elasticmachine commented Apr 4, 2017

Uh oh!

elasticmachine commented Apr 4, 2017

Uh oh!

andrewkroh Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martinscholz83 Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

martinscholz83 commented Apr 4, 2017

Uh oh!

ruflin commented Apr 4, 2017

Uh oh!

andrewkroh commented Apr 4, 2017

Uh oh!

ohadravid commented Apr 5, 2017

Uh oh!

martinscholz83 commented Apr 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinscholz83 commented Apr 5, 2017

Uh oh!

ruflin commented Apr 5, 2017

Uh oh!

ohadravid commented Apr 5, 2017

Uh oh!

martinscholz83 commented Apr 5, 2017

Uh oh!

ohadravid commented Apr 5, 2017

Uh oh!

ruflin commented Apr 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh commented Apr 5, 2017

Uh oh!

andrewkroh commented Apr 5, 2017

Uh oh!

martinscholz83 commented Apr 5, 2017

Uh oh!

ruflin commented Apr 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinscholz83 commented Apr 6, 2017

Uh oh!

ruflin commented Apr 6, 2017

Uh oh!

martinscholz83 commented Apr 6, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh Apr 4, 2017 •

edited

Loading

martinscholz83 Apr 4, 2017 •

edited

Loading

andrewkroh Apr 4, 2017 •

edited

Loading

martinscholz83 commented Apr 5, 2017 •

edited

Loading

ruflin commented Apr 6, 2017 •

edited

Loading