Enhance Azure platformlogs ingest pipeline for function app logs#14926
Enhance Azure platformlogs ingest pipeline for function app logs#14926kaiyan-sheng merged 7 commits intoelastic:mainfrom
Conversation
🚀 Benchmarks reportTo see the full report comment with |
| @@ -0,0 +1 @@ | |||
| { "properties": "{'appName':'test','roleInstance':'abc','message':'Test 2025-07-17T03:26:03.3543455Z.','category':'Function.timerTriggerFileInput.User','hostVersion':'4.1041.200.1','functionInvocationId':'809a97a9','functionName':'timerTriggerFileInput','hostInstanceId':'4bfe6a4e','level':'Information','levelId':2,'processId':27}", "time": "08/06/2025 14:26:04", "resourceId": "/SUBSCRIPTIONS/123456789/RESOURCEGROUPS/A/PROVIDERS/MICROSOFT.WEB/SITES/test", "level": "Informational", "category": "FunctionAppLogs", "operationName": "Microsoft.Web/sites/functions/log", "location": "brazilsouth", "EventStampType": "Stamp", "EventPrimaryStampName": "waws-prod-cq1-041", "EventStampName": "waws-prod-cq1-041", "Host": "ln0sdlwk000CYU", "EventIpAddress": "10.80.0.14"} | |||
There was a problem hiding this comment.
Just noticed the single quotes in this original log.
So those are not like https://github.com/elastic/integrations/blob/main/packages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-apigateway-log.json#L8 with '"' . So for example to have "appName"
Sorry for this comment, just saying if it worths to update in case sth is hidden
There was a problem hiding this comment.
Yeah, Functions logs are one of the Azure log categories that contain invalid JSON.
There was a problem hiding this comment.
Yeah this is one of the problem in this function log sample. I guess unfortunately we have to deal with it.
| - set: | ||
| field: temp_properties | ||
| value: '{{azure.platformlogs.properties}}' | ||
| if: "ctx.azure?.platformlogs?.properties instanceof String" | ||
| target_field: azure.platformlogs.properties.raw | ||
| ignore_failure: true | ||
| description: 'Temporarily store the properties string.' | ||
| - remove: | ||
| field: azure.platformlogs.properties | ||
| if: "ctx.temp_properties != null" | ||
| ignore_missing: true | ||
| description: 'Remove the properties string field.' | ||
| - script: | ||
| lang: painless | ||
| source: >- | ||
| if (ctx?.temp_properties != null) { | ||
| ctx.temp_properties = ctx.temp_properties.replace("'", "\""); | ||
| } | ||
| ignore_failure: true | ||
| description: 'Convert single quotes to double quotes in the properties string to make it valid JSON.' | ||
| - json: | ||
| field: temp_properties | ||
| target_field: azure.platformlogs.properties | ||
| if: "ctx.temp_properties != null" | ||
| ignore_failure: true | ||
| description: 'Parse the stringified JSON properties field into a nested object.' | ||
| - remove: | ||
| field: temp_properties | ||
| ignore_missing: true | ||
| description: 'Rename the field to `properties.raw` to avoid parse errors with the `properties` containing a string.' | ||
| description: 'Remove the temporary field after successful parsing.' |
There was a problem hiding this comment.
Here we're assuming that if properties is a string, it contains an embedded JSON document.
By replacing single-quotes with double-quotes, there's a non-zero risk of corrupting the string. I searched through test documents in Beats, integrations, and my archive of user cases: I only found strings with embedded JSON documents.
The benefits probably outweigh the minor risk.
There was a problem hiding this comment.
Yeah I agree! Maybe we will revisit if edge cases happen 🤔
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| description: > | ||
| Result description | ||
|
|
||
| - name: EventIpAddress |
There was a problem hiding this comment.
Do we need EventIpAddress and Host declaration in fields.yml? As these fields are already renamed here.
There was a problem hiding this comment.
Good catch! Let me check! Thanks!!
muthu-mps
left a comment
There was a problem hiding this comment.
Left comment on fields.yml, change looks good to me otherwise!
💚 Build Succeeded
History
|
|
|
Package azure - 1.28.2 containing this change is available at https://epr.elastic.co/package/azure/1.28.2/ |
Proposed commit message
When user trying to ingest function app logs using our azure integration,
@timestampfield is missing. After testing with the sample log, I found several issues with the new log format:propertiesfield: the log contains apropertiesfield that is a string with single quotes@timestampfield for logs using theM/d/yyyy H:mm:ssdate format. Thedateprocessor has been updated to include this format.EventIpAddressandEventStampName.Checklist
changelog.ymlfile.How to test it locally
and we will get: