fix(azure.eventhub): handles properties as string and drops empty fields#14959
Conversation
As azure.eventhub.properties.raw
| return (((Map) object).size() == 0); | ||
| } else if (object instanceof List) { | ||
| ((List) object).removeIf(value -> dropEmptyFields(value)); | ||
| return (((List) object).length == 0); |
There was a problem hiding this comment.
It should be (((List) object).size() == 0) isn't it ?
There was a problem hiding this comment.
I see it works both ways:
However size() seems like the right approach since both List and Map extend Collection.
There was a problem hiding this comment.
Interesting, I thought painless is Java, but I think .length is an addon by painless.
🚀 Benchmarks reportTo see the full report comment with |
packages/azure/data_stream/eventhub/elasticsearch/ingest_pipeline/parsed-message.yml
Outdated
Show resolved
Hide resolved
…ine/parsed-message.yml Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
|
The sample event here has the properties with fields and the conditions property has empty string and also fields with values. Can we include this format as well to the pipeline tests? |
💚 Build Succeeded
History
cc @zmoog |
muthu-mps
left a comment
There was a problem hiding this comment.
Left a comment on adding additional events to the pipeline tests. Change looks good to me otherwise!
|
chemamartinez
left a comment
There was a problem hiding this comment.
LGTM! In case it’s useful to know you can also apply subobjects: false to solve the conflict. Done for example at #13925.
Yeah, but this would require to add mappings for these fields not to make test fail, which is something I am trying to avoid in a generic integration like |
Great point, I didn't think about using Since this is a generic integration, where users are supposed to customize pipeline and mapping, I'm trying to leave all the options on the table. |
|
Package azure - 1.28.3 containing this change is available at https://epr.elastic.co/package/azure/1.28.3/ |
…lds (elastic#14959) ### WHAT - Drops empty/null fields - Renames `azure.eventhub.properties` as `azure.eventhub.properties.raw` when `azure.eventhub.properties` is a string instead of an object. ### WHY **Drops empty/null fields** Sometimes Azure services emit log event like the following: ```json { "category": "NetworkSecurityGroupEvent", "operationName": "NetworkSecurityGroupEvents", "properties": { "conditions": { "": "", "destinationPortRange": "0-65535", "sourcePortRange": "0-65535" }, "direction": "Out", "macAddress": "00-11-22-33-4444", "primaryIPv4Address": "10.0.4.6", "primaryIPv6Address": "ace:ace:dead:beef::9", "priority": 65000, "ruleName": "DefaultRule_AllowVnetOutBound", "subnetPrefix": "10.0.4.0/25", "type": "allow", "vnetResourceGuid": "{a08d316f-3c0a-428d-84ec-2977078852a5}" } } ``` Elasticsearch cannot index `"": ""` field inside `properties.condition`, so we need to clean it up. **Renames `azure.eventhub.properties` as `azure.eventhub.properties.raw` when `azure.eventhub.properties` is a string instead of an object** Sometimes Azure sends `azure.eventhub.properties` as a string, this field should really be a an `object` instead, causing mapping errors like: ```text object mapping for [azure.eventhub.properties] tried to parse field [properties] as object, but found a concrete value ``` By renaming `azure.eventhub.properties` as `azure.eventhub.properties.raw`, we avoid the mapping error and give users the opportunity to handle the original value using a custom pipeline.
Proposed commit message
WHAT
azure.eventhub.propertiesasazure.eventhub.properties.rawwhenazure.eventhub.propertiesis a string instead of an object.WHY
Drops empty/null fields
Sometimes Azure services emit log event like the following:
{ "category": "NetworkSecurityGroupEvent", "operationName": "NetworkSecurityGroupEvents", "properties": { "conditions": { "": "", "destinationPortRange": "0-65535", "sourcePortRange": "0-65535" }, "direction": "Out", "macAddress": "00-11-22-33-4444", "primaryIPv4Address": "10.0.4.6", "primaryIPv6Address": "ace:ace:dead:beef::9", "priority": 65000, "ruleName": "DefaultRule_AllowVnetOutBound", "subnetPrefix": "10.0.4.0/25", "type": "allow", "vnetResourceGuid": "{a08d316f-3c0a-428d-84ec-2977078852a5}" } }Elasticsearch cannot index
"": ""field insideproperties.condition, so we need to clean it up.Renames
azure.eventhub.propertiesasazure.eventhub.properties.rawwhenazure.eventhub.propertiesis a string instead of an objectSometimes Azure sends
azure.eventhub.propertiesas a string, this field should really be a anobjectinstead, causing mapping errors like:By renaming
azure.eventhub.propertiesasazure.eventhub.properties.raw, we avoid the mapping error and give users the opportunity to handle the original value using a custom pipeline.Checklist
changelog.ymlfile.I have verified that any added dashboard complies with Kibana's Dashboard good practices