Describe the enhancement:
Once in a while people like to merge messages into a single line not based on a pattern but based on the number of lines that have to be merged. This may be caused by not having a clear usable pattern or by just wanting to reduce the number of lines in a message by combining several. There are situations that it may also be handy to combine the lines into a JSON-array that can be used by other applications.
I propose to introduce an extra multiline parameter kind that distinguishes this behavior. Of course all the other parameters are still valid so in theory you can combine the pattern and the max_lines parameters. Although in practice I do not expect that.
The values of the kind parameter would be <<empty>> (default and current implementation), merge, and merge-json, where merge-json will combine the messages in a JSON-array.
Describe a specific use case for the enhancement or feature:
It is when you know the number of lines of an event but there is no clear pattern.
Per example someone has dumped a database table one field per line. In that case you know the number of lines for a row (= number of columns) but creating a pattern for that may be hard. In this situation the configuration can be as follows:
multiline.kind: "merge"
multiline.pattern: ".*"
multiline.match: "before"
multiline.negate: false
multiline.max_lines: 13
where 13 is the number of columns in a row. This will create a single event for a single row. In case you would choose merge-json they would be combined in one JSON-array.
Another use-case is that someone just want to group a set of events that are similar. Per example the application is creating a lot of events and you want to put them in buckets of 300 each so that you can handle such group as a single event. In that case the configuration can be as follows:
multiline.kind: "merge"
multiline.pattern: ".*"
multiline.match: "before"
multiline.negate: false
multiline.max_lines: 300
A side-effect of the merge and merge-json options are that there are no lines discarded.
Describe the enhancement:
Once in a while people like to merge messages into a single line not based on a pattern but based on the number of lines that have to be merged. This may be caused by not having a clear usable pattern or by just wanting to reduce the number of lines in a message by combining several. There are situations that it may also be handy to combine the lines into a JSON-array that can be used by other applications.
I propose to introduce an extra multiline parameter
kindthat distinguishes this behavior. Of course all the other parameters are still valid so in theory you can combine thepatternand themax_linesparameters. Although in practice I do not expect that.The values of the
kindparameter would be<<empty>>(default and current implementation),merge, andmerge-json, wheremerge-jsonwill combine the messages in a JSON-array.Describe a specific use case for the enhancement or feature:
It is when you know the number of lines of an event but there is no clear pattern.
Per example someone has dumped a database table one field per line. In that case you know the number of lines for a row (= number of columns) but creating a pattern for that may be hard. In this situation the configuration can be as follows:
where 13 is the number of columns in a row. This will create a single event for a single row. In case you would choose
merge-jsonthey would be combined in one JSON-array.Another use-case is that someone just want to group a set of events that are similar. Per example the application is creating a lot of events and you want to put them in buckets of 300 each so that you can handle such group as a single event. In that case the configuration can be as follows:
A side-effect of the
mergeandmerge-jsonoptions are that there are no lines discarded.