Out of curiosity why do we need to populate these fields in the pipeline rather than have the endpoint populate the fields when it builds the rest of the event?

Out of curiosity why do we need to populate these fields in the pipeline rather than have the endpoint populate the fields when it builds the rest of the event?

Yeah @jrmolin or @ferullo can probably answer better here. I believe certain fields are more difficult than others to parse on the endpoint side. I think most of the information was retrieved in the ETW events for the original sensor and they have to track multiple events to get the information. But since we're already putting the information in the message field, I'm not sure why we couldn't just populate the dns.question.type and dns.quesiton.name fields?

@jonathan-buttner if/when Endpoint begins to populate this data will the pipeline still overwrite with Endpoint sets or will this code be effectively disabled?

@jonathan-buttner if/when Endpoint begins to populate this data will the pipeline still overwrite with Endpoint sets or will this code be effectively disabled?

The way this PR is right now, it'd overwrite what the endpoint puts in those fields. I can add a simple if to check if the fields are already present though and skip populating them.

I think that would be a good check to add. Marshall makes a good point about Endpoint doing this itself. The reality though is Endpoint won't get that done for 7.10 and functionally it doesn't matter if Endpoint or the pipeline does this until detection rules can be run in Endpoint. We should move this logic to Endpoint when that happens but if there's no reason to not merge this it seems like it'll give detection rules the data they need in the meantime.

Does that sound reasonable?

Once we add and ship this pipeline we can't really remove it from the package unless we can ensure that all endpoints that use it have been upgraded to populate the fields on the endpoint side, or we'd have to be comfortable breaking compatibility at some point between new package versions and old endpoints. So the risk is that long term we build up a bunch of these fixes applying subtly different modifications and end up with a complex pipeline if we're not careful about what we use it for.

I think the if check would be a good addition, and since we're only adding fields that don't exist in the document then this is a fine approach to make the dns fields available in 7.10. If we can stick to only adding fields that don't already exist with the pipeline and ensure that we don't modify existing fields then the pipeline shouldn't get too complex even if we add more in the future.

Once we add and ship this pipeline we can't really remove it from the package unless we can ensure that all endpoints that use it have been upgraded to populate the fields on the endpoint side, or we'd have to be comfortable breaking compatibility at some point between new package versions and old endpoints. So the risk is that long term we build up a bunch of these fixes applying subtly different modifications and end up with a complex pipeline if we're not careful about what we use it for.

I think the if check would be a good addition, and since we're only adding fields that don't exist in the document then this is a fine approach to make the dns fields available in 7.10. If we can stick to only adding fields that don't already exist with the pipeline and ensure that we don't modify existing fields then the pipeline shouldn't get too complex even if we add more in the future.

Yeah that's a good point. I agree, hopefully we can stick to adding fields that don't exist in the pipelines and avoid modifying existing fields. If we end up having to make breaking changes we should wait until an 8.x release for that.

I'll add the check to see if the field already exists.

I must have completely overlooked this but the endpoint is already populating dns.question.name 😅 . I'll switch this PR to just populate dns.question.type.

Doesn't look like linux or mac send dns events, but I assume once they do they can just populate the dns.question.name as well.