azure logs: add ECS mapping for event.duration#11104
azure logs: add ECS mapping for event.duration#11104zmoog wants to merge 5 commits intoelastic:mainfrom
Conversation
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
signinlogs |
2450.98 | 2016.13 | -434.85 (-17.74%) | 💔 |
firewall_logs |
1694.92 | 1398.6 | -296.32 (-17.48%) | 💔 |
To see the full report comment with /test benchmark fullreport
There was a problem hiding this comment.
Suggest also
diff --git a/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml b/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
index e5e7c52bbd..fcdc5b2eb3 100644
--- a/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
@@ -72,10 +72,23 @@ processors:
field: azure.activitylogs.durationMs
target_field: event.duration
ignore_missing: true
+ - convert:
+ field: event.duration
+ tag: convert_event_duration
+ type: long
+ ignore_missing: true
+ on_failure:
+ - remove:
+ field: event.duration
+ - append:
+ field: error.message
+ value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- script:
lang: painless
- source: if (ctx.event.duration!= null) {ctx.event.duration = ctx.event.duration
- * params.param_nano;}
+ source: >
+ if (ctx.event.duration != null) {
+ ctx.event.duration = ctx.event.duration * params.param_nano;
+ }
params:
param_nano: 1000000
ignore_failure: trueand changelog/manifest updates.
|
Hey @efd6, thanks for suggesting the I was looking for a temporarily quick fix for the mapping, adding back |
The field value must be a `long` to align with ECS and correctly leverage the ecs@mappings component template.
|
|
/test |
💔 Build Failed
Failed CI StepsHistory
cc @zmoog |
| field: azure.activitylogs.durationMs | ||
| target_field: event.duration | ||
| ignore_missing: true | ||
| - convert: |
There was a problem hiding this comment.
Can't we adding back the mapping to ecs.yml instead of doing this ?
There was a problem hiding this comment.
I agree, we have seen that the conversion is not effective in communicating that the type should be a long.
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution! |
Proposed commit message
Convert the
event.durationfield value to thelongtype.Users reported mapping exceptions due to
event.durationstring values causing field mapping askeywordinstead oflong. See #10848 to learn more.Elasticsearch maps a field as a
keywordif it has a string value. This happens even on stack versions 8.13+ because ecs@mappings does not perform type coercion.By converting the
event.durationfield values to thelongtype, we ensure Elasticsearch uses the expected ECS field mapping aslong.Checklist
changelog.ymlfile.Related issues