Skip to content

Adds buffer_ignore_exceeded_chunk option to Buffer#306

Closed
dterror-zz wants to merge 2 commits into
fluent:masterfrom
dterror-zz:buffer_ignore_exceeded_chunk-option-for-buffers
Closed

Adds buffer_ignore_exceeded_chunk option to Buffer#306
dterror-zz wants to merge 2 commits into
fluent:masterfrom
dterror-zz:buffer_ignore_exceeded_chunk-option-for-buffers

Conversation

@dterror-zz

Copy link
Copy Markdown

Controls whether exceeded chunk should raise an exception (and break the chain) or log and ignore
the exception. As discussed in #296

Controls whether exceeded_chunk should raise an
exception (and break the chain) or log and ignore
the exception.
@sonots

sonots commented Apr 23, 2014

Copy link
Copy Markdown
Member

This code will enqueue at lib/fluent/buffer.rb#L207-L209 even if queue size ecceeds buffer_queue_limit. It should not be occured.

(even if using @ignore_exceeded_chunk)
@dterror-zz

Copy link
Copy Markdown
Author

@sonots added a test case

@kzk

kzk commented Apr 25, 2014

Copy link
Copy Markdown
Member

-1 for this change. Here're two reasons:

First one is a philosophical thing. Fluentd's importance is 'simplicity'. The more we have options, the more beginners need to learn.

Second, you still have a limit on disk volume anyway. And if we have a disk full error while writing to the buffer, we possibly couldn't recover the data. What we want is an application level handling here.

Is just having a larger value for buffer_queue_limit would satisfy your use case?

@dterror-zz

Copy link
Copy Markdown
Author

@kzk yeah, for individual cases, increasing buffer_queue_limit is the way to go. This motivation for this feature is when a plugin is used in a chain and we want to make sure that failures in this particular plugin won't cause the chain to be broken.
Most of the discussion is in #296 and there's another pr specifically for out_copy #303.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Feature request or improve operations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants