@dterror

I'll still implement your idea of buffer_ignore_exceeded_chunk, but in a different pull-request, is that ok?

I am considering buffer error handling improvement. buffer_ignore_exceeded_chunk is a part of such changes. So please pending it for now.

@repeatedly implemented in #306

btw, you mentioned you were working on streaming errors to another plugin, that sounds like a very good idea, will it be based on Fluent::Log?

@dterror It is based on new Engine routing. We need more time to apply this change to current version.

After review the code, I have one idea.
How about ignore_individual_errors [2, 3] instead of ignore_individual_errors true.
The user specify ignore plugin indexes. Above example, second and third plugin errors are ignored.

Or specifying plugin id like ignore_individual_errors ["id2", "id3"]

@repeatedly I think that with the implementation of ignore_exceeded_chunk and other error handling improvements we can have a more granular control of what is allowed to fail on a per-plugin config level. With this ignore_individual_errors can be more 'general'. What do you think?

What does 'general' mean? ignore_indivisual_erros can be applied to another plugins or Fluentd core level?

I mean it won't need a granular control such as ignore_individual_errors ["id2", "id3"] because each plugin inside copy would be able to use a no-fail option (like buffer_ignore_exceeded_chunk for example).

@dterror I see. But only ignore_individual_errors true, the user can't raise an exception which should not be ignored, e.g. S3 plugin on your case.

They can ignore ignore_individual_errors in copy and use other no-fail solutions in the ones that can fail.

For example, for my use case in particular I could use:

<match foo.bar.*>
  type copy
  ignore_individual_errors true
  <store>
    type s3
    #...
  </store>
  <store>
    type elasticsearch
    #...
  </store>
</match>

If its okay for either to fail, but not both.

OR

<match foo.bar.*>
  type copy
  <store>
    type s3
    #...
  </store>
  <store>
    type elasticsearch
    buffer_ignore_exceeded_chunk true
    #...
  </store>
</match>

If S3 can't to fail.

It's my belief that buffer_ignore_exceeded_chunk was the only thing that would cause an application error.

It's my belief that buffer_ignore_exceeded_chunk was the only thing that would cause an application error.

Yeah but other error occurs. Hmm, Fluentd's config parameter can accept any type. I will add array support for other use-cases.

But only ignore_individual_errors true, the user can't raise an exception which should not be ignored, e.g. S3 plugin on your case.

Although I am not familiar with S3 plugin, ignore_individual_errors true still raises OutputChainError when one of <store> plugins raises error, so the user can raise an exception.
This looks fine enough for me. Am I missing something?

Oh, current implementation raises OutputChainError only when all of plugins raise errors. I think OutputChainError should be raised if one or some of <store> plugins raise error after all <store> plugins are processed.

@sonots

I think OutputChainError should be raised if one or some of plugins raise error after all plugins are processed.

The proposal of this PR want to ignore the exception which caused by un-important plugin. This is why I propose ignore_indivisual_errors [1, 2] approach to ignore specified plugins.

The reason to ignore the exception is (#296)

Raising an exception at this point will break the chain,

So, I thought as the ultimate goal of this PR is to create a option not to break the chain.

Suppressing to raise errors in each plugin, and raising OutputChainError after all plugins are processed achieves the goal.

Suppressing to raise errors in each plugin, and raising OutputChainError after all plugins are processed achieves the goal.

I think no. This behaviour breaks the transaction. Currently, Input plugin assumes emit is succeeded or raise an error. Your approach is succeeded and raise an error.
For example, in_http returns an error to client when emit failed.
In your approach, the user may get duplicated logs because almost clients have retry or fallback feature.

Your approach is succeeded and raise an error.

No, my approach is succeeded (when no plugin fail) or raises an error (when one or some of plugins fails).

No, my approach is succeeded (when no plugin fail) or raises an error (when one or some of plugins fails).

My 'succeeded' means out_copy processes remaining plugins if error occurred.
From OutputChain user side, Output plugins seems to process the event correctly but raise an error.

If we select your approach, out_copy should keep failed plugin lists for avoiding log duplication.
Your approach seems to good for bulk insert on database but I think hard to use on Fluentd like pluggable system because there are many error reasons.

P.S.
After support error stream, this restriction will be removed or relaxed.

I see your point @repeatedly. Covering more cases does make sense. I'll change the implementation.
However, I think ignore_indivisual_errors [1, 2] is a bit weird for referencing plugin indexes. How about adding it as a <store> arg. Something like:

<match foo.bar.*>
  type copy
  <store>
    type s3
    #...
  </store>
  <store ignore-failure>
    type elasticsearch
    #...
  </store>
</match>

Ouch, I deleted the comment... ;(

@dterror Fluentd plugin has built-in id parameter so using id with ignore_indivisual_errors is easy to implement.

But your approach also seems to easy.
Either way's fine with me :)

@sonots @kiyoto What do you think?

<store ignore-failure>

looks nice for me.

Another one idea.

This issue will be resolved by error stream.
So temporary, releasing out_copy_xxx.rb which includes this feature and add link to out_copy document.

I'm not sure which is the best...

How long until error stream is out? what will it look like and is there anything I can do help? If that's the 'proper' way to go, I think it's fair to wait.

That is, provided error stream is outlined and is not going to take too long. I still think ignore_individual_options is a very reasonable as a solution.

In my plan, v1 includes error stream and now working to implement v1 features.
The release date is not fixed but I want to release v1 series in this summer.

Since it looks ignore_individual_errors will be obsolete after error stream is implemented, I created another plugin just for the current use. See https://github.com/sonots/fluent-plugin-copy_ex