Update broker/pipeline setup by urso · Pull Request #4650 · elastic/beats

urso · 2017-07-11T04:33:21Z

remove settings: filebeat.spooler, filebeat.publish_async, filebeat.idle_timeout, queue_size, bulk_queue_size
introduce new setting: queue.<queue_type>
expose memory queue (type = 'mem') settings: events, flush.min_events, flush.timeout

Note: docs will be updated in a separate PR.

ruflin · 2017-07-11T06:50:21Z

metricbeat/metricbeat.reference.yml

It's a bit strange that we call it here queue and in the code broker. This will need some docs and changelog entry. But can happen in a follow up PR. Does this have to be in this PR?

These are the user-visible settings. I decided to rename the broker package to queue, but first I want to get the changes to the settings in and documented, so we won't introduce BC-breaking changes in case package updates PR gets merged late.

ruflin · 2017-07-11T19:32:33Z

auditbeat/auditbeat.reference.yml

s/send/sent/

"to be sent" is correct

ruflin · 2017-07-11T19:33:28Z

auditbeat/auditbeat.reference.yml

What happens when this is reached? Flush happens? Thinking if there should be flush.min, flush.max and flush.timeout or similar?

Semantics is a little different from old spooler here. The queue/broker does not flush its buffers. By default it basically stream the events right through to the outputs (if output takes somewhat longer it will be presented with all events available - up to bulk_max_size).

By configuring min_events, the queue will block the output until:

at least min events is reached

timeout is trigger

The min events is more of a hint, as the output might still receive a smaller buffer.

The events setting is the maximum number of events the queue can hold. It's independent of min_events and flush timer. Only min_events must be < events.

Naming is a little weird here... as it's more like 'block-processing' settings...

@urso what happens when events is reached? I assume it will block from adding more events? In case of metricbeat assuming sending events is slower then generating events but still all events are acked. What will happen?

once events is reached, the queue will block (no more space). Upon ACK of N events, another N events can be stored in the queue (N producers will be unblocked).

ruflin · 2017-07-11T19:35:08Z

libbeat/publisher/bc/publisher/pipeline.go

As quickly discussed, later we should probably rename broker package to queue to make the relation to the config more obvious.

ruflin · 2017-07-11T19:39:21Z

libbeat/publisher/broker/membroker/broker.go

Seems like the 3 params could be a config struct for borkers.

andrewkroh · 2017-07-11T20:09:47Z

winlogbeat/winlogbeat.reference.yml

I'll open a PR to delete this block of code from Winlogbeat so that queue isn't rejected.

hehe, thanks.

this PR is adjusting winlogbeat.

ruflin · 2017-07-12T05:49:16Z

CHANGELOG.asciidoc

Should we add a message to the 5.6 release that these values will change?

ruflin · 2017-07-12T05:49:48Z

CHANGELOG.asciidoc

Same here, we should probably have a message in 5.6 or even better, not let the beat start in 6.0 if these values are set?

ruflin · 2017-07-12T05:51:00Z

auditbeat/auditbeat.reference.yml

Is there a reason we changed the default to 5? Should probably be mentioned in the changelog?

pipeline can operate complete async -> take advantage of this fact

with default settings we have a high chance to publish very small batches (no flush timeout -> reduced latency) if producers don't have a high enough rate. In Logstash case we can easily compensate for this by pushing more batches on the link

Updated changelog.

ruflin · 2017-07-12T05:51:40Z

filebeat/config/config.go

Glad to see this one removed ;-)

ruflin · 2017-07-12T05:53:26Z

libbeat/publisher/broker/membroker/config.go

Kind of surprised to see a minimum here.

well, we need to buffer some data for pushing to the outputs. Without queue the pipeline won't be able to operate -> enforce minimum so users don't use 1 or 0.

- remove setting queue_size - remove unused setting bulk_queue_size - introduce settings namespace `queue` with default `mem.events: 4096` - add settings queue.mem.flush.min_events and queue.mem.flush.timeout - change default output.logstash.pipelining to 5 - remove spooler settings from filebeat config object - Update winlogbeat config validate check

urso · 2017-07-12T13:46:41Z

Translating old configurations:

rename queue_size -> queue.mem.events * 2
remove bulk_queue_size
rename or remove filebeat.spool_size -> queue.mem.flush.min_events
rename or remove filebeat.idle_timeout -> queue.mem.flush.timeout

dedemorton · 2018-01-22T18:45:49Z

I'm removing the needs_docs label here because I believe we've covered the doc requirements with the topic https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html and the removed settings that are documented in the breaking changes for the release.

urso added the in progress Pull request is currently in progress. label Jul 11, 2017

ruflin reviewed Jul 11, 2017

View reviewed changes

urso force-pushed the pipeline/config branch 2 times, most recently from fe27bb3 to cc174ea Compare July 11, 2017 12:09

ruflin approved these changes Jul 11, 2017

View reviewed changes

andrewkroh approved these changes Jul 11, 2017

View reviewed changes

urso force-pushed the pipeline/config branch 3 times, most recently from 57756c6 to b3f1b82 Compare July 12, 2017 02:19

urso changed the title ~~[WIP] update broker/pipeline setup~~ Update broker/pipeline setup Jul 12, 2017

urso force-pushed the pipeline/config branch from b3f1b82 to 61f5b7e Compare July 12, 2017 03:00

urso mentioned this pull request Jul 12, 2017

Publisher Pipeline #4598

Closed

22 tasks

urso added needs_docs review and removed in progress Pull request is currently in progress. labels Jul 12, 2017

ruflin approved these changes Jul 12, 2017

View reviewed changes

urso force-pushed the pipeline/config branch from 61f5b7e to cd45c0a Compare July 12, 2017 13:29

ruflin merged commit 7a6a5dd into elastic:master Jul 12, 2017

monicasarbu added the libbeat label Jul 17, 2017

urso deleted the pipeline/config branch July 19, 2017 13:18

tsg mentioned this pull request Jul 24, 2017

Document breaking changes in 6.0 #4737

Closed

28 tasks

dedemorton mentioned this pull request Jul 25, 2017

Beats doc updates in 6.0 #4540

Closed

42 tasks

dedemorton removed the needs_docs label Jan 22, 2018

Conversation

urso commented Jul 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urso Jul 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urso Jul 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urso commented Jul 12, 2017

Uh oh!

dedemorton commented Jan 22, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

urso commented Jul 11, 2017 •

edited

Loading

urso Jul 11, 2017 •

edited

Loading

urso Jul 12, 2017 •

edited

Loading