Pinging @elastic/siem (Team:SIEM)

💚 Build Succeeded

I used a copy of this PR together with the new ATP module, it seems to be splitting on both arrays correctly.

The document had multiple events as an array under "values" key, and in each event there was also an "evidence" array of objects. One API response with 2 objects under value, and each has 2 objects under evidence returned 4 documents, one for each evidence corretly.

Will it be a good idea to create a split_event Filebeat processor instead of putting this just in the httpjson input, so that this functionality can be supported for other inputs also?

Will it be a good idea to create a split_event Filebeat processor instead of putting this just in the httpjson input, so that this functionality can be supported for other inputs also?

Do you have any examples on which other inputs might benefit from this? I think in theory it makes sense but maybe at a later date? The reason is that I feel if splitting is currently only working on json structure then the input should enforce the structure as well

Will it be a good idea to create a split_event Filebeat processor instead of putting this just in the httpjson input, so that this functionality can be supported for other inputs also?

Do you have any examples on which other inputs might benefit from this? I think in theory it makes sense but maybe at a later date? The reason is that I feel if splitting is currently only working on json structure then the input should enforce the structure as well

I agree with @P1llus, but in case we decide to do it as a separate processor, I am unsure about how to do it. The processor interface looks like:

type Processor interface {
	String() string // print full processor description
	Run(in *Event) (event *Event, err error)
}

And for this use case we would need something like:

type Processor interface {
	String() string // print full processor description
	Run(in *Event) ([]event *Event, err error)
}

I am not sure if there is any other way of doing this that is not as a Processor, or to embed the list of resulting events in a single one, but that feels a bit odd to me if it is the case. WDYT?

Will it be a good idea to create a split_event Filebeat processor instead of putting this just in the httpjson input, so that this functionality can be supported for other inputs also?

Do you have any examples on which other inputs might benefit from this? I think in theory it makes sense but maybe at a later date? The reason is that I feel if splitting is currently only working on json structure then the input should enforce the structure as well

I agree with @P1llus, but in case we decide to do it as a separate processor, I am unsure about how to do it. The processor interface looks like:

type Processor interface {
	String() string // print full processor description
	Run(in *Event) (event *Event, err error)
}

And for this use case we would need something like:

type Processor interface {
	String() string // print full processor description
	Run(in *Event) ([]event *Event, err error)
}

I am not sure if there is any other way of doing this that is not as a Processor, or to embed the list of resulting events in a single one, but that feels a bit odd to me if it is the case. WDYT?

After you mentioned it I do remember now that I have looked at split in processors before, and currently a processor cannot create multiple documents, as it will always run once for each incoming event.