Filebeat Redis prospector type by ruflin · Pull Request #4180 · elastic/beats

ruflin · 2017-05-03T06:18:22Z

This PR adds a new prospector type which reads the slowlog from redis. This slowlog is not in a file but in memory in redis. Because of this filebeat connects to redis and reads out the slowlog. It is important to note that the slow log size is limited in redis that is why after fetching the events from the slowlog filebeat resets the slow log.

Example event looks as following:

{
  "@timestamp": "2017-05-16T06:27:17.000Z",
  "beat": {
    "hostname": "ruflin",
    "name": "ruflin",
    "read_timestamp": "2017-05-16T06:27:19.275Z",
    "version": "6.0.0-alpha2"
  },
  "message": "SET hello world",
  "redis": {
    "slowlog": {
      "args": [
        "world"
      ],
      "cmd": "SET",
      "duration": {
        "us": 11
      },
      "id": 38,
      "key": "hello"
    }
  }
}

All args are combined in the "message" field output for easy retrieval.

elasticmachine · 2017-05-03T06:48:35Z

filebeat/prospector/redis/config.go

[golint] _{reported by reviewdog 🐶}
exported type Config should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/doc.go

[golint] _{reported by reviewdog 🐶}
package comment should be of the form "Package redis ..."

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported type Harvester should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported method Harvester.Start should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/prospector.go

[golint] _{reported by reviewdog 🐶}
comment on exported type Prospector should be of the form "Prospector ..." (with optional leading article)

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/prospector.go

[golint] _{reported by reviewdog 🐶}
comment on exported function NewProspector should be of the form "NewProspector ..."

tsg · 2017-05-03T08:14:58Z

filebeat/_meta/common.full.p2.yml

I suggest somehow making it more clear that this is strictly about slow logs. People might assume that it's a general Redis input.

That is a tricky one. Thinking more about the prospectors I get the impression we will have prospector with multiple harvester types potentially (similar to what we have in metricbeat). For example in file there could be file.csv or file.xml. What if we not only have slowlog for redis? But TBH I'm not sure yet what we should do here. This could also well be overengineered and we will not need it.

For the moment I think I fix the above with call it Redis slowlog prospector in the docs but still redis as the prospector type.

tsg · 2017-05-03T08:17:13Z

filebeat/prospector/redis/doc.go

I'd use 2s or 5s as an example. 100 micros might affect the Redis performance, so we should be careful.

tsg · 2017-05-03T08:18:32Z

filebeat/prospector/redis/doc.go

small typo: "it can be configured"

tsg · 2017-05-03T08:26:00Z

filebeat/prospector/redis/harvester.go

Hmm, good question. In the end it needs to be the event timestamp, but I see two options:

live it like that here, but replace it in the FB redis module

already set it hear and also set beat.read_timestamp

I decided for option 2 for the moment. The reasons is that I think it most cases that is what people expect in @timestamp here. So also without postprocessing in ingest, the data is already correct.

tsg · 2017-05-03T08:26:26Z

filebeat/prospector/redis/harvester.go

I guess people can use drop_fields if they don't want it.

elasticmachine · 2017-05-15T12:02:00Z

filebeat/harvester/forwarder.go

[golint] _{reported by reviewdog 🐶}
comment on exported method Forwarder.Forward should be of the form "Forward ..."

elasticmachine · 2017-05-15T12:02:00Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported type Harvester should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported function NewHarvester should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported method Harvester.Start should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported method Harvester.Stop should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
exported method Harvester.ID should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/prospector.go

[golint] _{reported by reviewdog 🐶}
exported method Prospector.Stop should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/prospector.go

[golint] _{reported by reviewdog 🐶}
exported method Prospector.Wait should have comment or be unexported

ruflin · 2017-05-16T06:35:20Z

@tsg Ready for an other look. I think we are at the point now where we have to decide on what to do with it :-)

elasticmachine · 2017-05-18T15:41:27Z

filebeat/harvester/util.go

[golint] _{reported by reviewdog 🐶}
comment on exported var ValidType should be of the form "ValidType ..."

elasticmachine · 2017-05-18T15:41:27Z

filebeat/prospector/redis/harvester.go

[golint] _{reported by reviewdog 🐶}
comment on exported method Harvester.Run should be of the form "Run ..."

ruflin · 2017-05-19T14:29:17Z

@dedemorton So far I only added the minimial options of docs for this = almost nothing. The reason is that this kind of challenges our existing doc structure we have for prospectors. So far the 2 prospectors shared 95% of the config options. With this change, they can be very different. I'm thinking if we should start to organise docs similar to what we do with the modules and have a block for each prospector type. I will need your help here and suggest we will only tackle this after merging the PR.

dedemorton · 2017-05-20T00:51:54Z

@ruflin I remember you mentioned this problem awhile back ago in an email, but I haven't had a chance to follow up with you. What release are we targeting for this PR?

ruflin · 2017-05-22T13:01:09Z

@dedemorton Not sure if I mentioned that before (but could be). It's not urgent. If it happens, somewhere around 6.0

tsg · 2017-05-24T11:05:47Z

filebeat/filebeat.full.yml

small typo on "ofr".

@timestamp

This PR adds a new prospector type which reads the slowlog from redis. This slowlog is not in a file but in memory in redis. Because of this filebeat connects to redis and reads out the slowlog. It is important to note that the slow log size is limited in redis that is why after fetching the events from the slowlog filebeat resets the slow log. Example event looks as following: ``` { "@timestamp": "2017-05-16T06:27:17.000Z", "beat": { "hostname": "ruflin", "name": "ruflin", "read_timestamp": "2017-05-16T06:27:19.275Z", "version": "6.0.0-alpha2" }, "message": "SET hello world", "redis": { "slowlog": { "args": [ "world" ], "cmd": "SET", "duration": { "us": 11 }, "id": 38, "key": "hello" } } } ``` All args are combined in the "message" field output for easy retrieval.

ruflin · 2017-05-26T06:43:39Z

@tsg New version pushed
@dedemorton I created #4398 as a follow up Github issue for the docs.

ruflin added discuss Issue needs further discussion. Filebeat Filebeat in progress Pull request is currently in progress. labels May 3, 2017

elasticmachine reviewed May 3, 2017

View reviewed changes

tsg reviewed May 3, 2017

View reviewed changes

filebeat/prospector/redis/harvester.go Outdated

Copy link
Copy Markdown

Contributor

tsg May 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess people can use drop_fields if they don't want it.

ruflin mentioned this pull request May 10, 2017

Filebeat refactoring / restructuring #4277

Closed

14 tasks

ruflin force-pushed the fb-redis-approach branch 2 times, most recently from 456e2e0 to 9ba6162 Compare May 15, 2017 11:25

elasticmachine reviewed May 15, 2017

View reviewed changes

ruflin force-pushed the fb-redis-approach branch 5 times, most recently from 4a55634 to 36be381 Compare May 16, 2017 06:33

ruflin changed the title ~~[WIP] Filebeat Redis prospector type~~ Filebeat Redis prospector type May 16, 2017

ruflin added review and removed discuss Issue needs further discussion. labels May 16, 2017

ruflin mentioned this pull request May 18, 2017

Including protologbeat in list of community beats #4348

Merged

ruflin force-pushed the fb-redis-approach branch from 36be381 to a9bceb9 Compare May 18, 2017 14:58

elasticmachine reviewed May 18, 2017

View reviewed changes

ruflin mentioned this pull request May 19, 2017

Add varnishlogbeat and varnishstatbeat #4353

Merged

ruflin force-pushed the fb-redis-approach branch 2 times, most recently from e021c74 to 8a8caba Compare May 19, 2017 14:26

ruflin force-pushed the fb-redis-approach branch from 8a8caba to 1a29277 Compare May 22, 2017 13:03

tsg reviewed May 24, 2017

View reviewed changes

filebeat/filebeat.full.yml Outdated

Copy link
Copy Markdown

Contributor

tsg May 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small typo on "ofr".

ruflin force-pushed the fb-redis-approach branch from 1a29277 to e3e6c2f Compare May 26, 2017 06:41

ruflin mentioned this pull request May 26, 2017

Create Redis Prospector Configuration #4398

Closed

3 tasks

ruflin removed the in progress Pull request is currently in progress. label May 26, 2017

tsg merged commit 4ab7848 into elastic:master May 26, 2017

ruflin deleted the fb-redis-approach branch May 26, 2017 09:59

Conversation

ruflin commented May 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin commented May 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin commented May 19, 2017

Uh oh!

dedemorton commented May 20, 2017

Uh oh!

ruflin commented May 22, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin commented May 26, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ruflin commented May 3, 2017 •

edited

Loading