Previous discussion: https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356
All Elastic Stack components are on version 7.13.2.
It looks like the AWS ALB support in the Filebeat AWS module produces http.request.referrer in its ingest pipeline, while Kibana Observability (such as the Logs section) expects url.* fields to be set (probably via uri_parts).
Currently, the Kibana Logs section displays access logs as follows, with a missing request path: (This display is apparently produced by generic_webserver.ts.)
[aws][access] 1.2.3.4 "GET HTTP/2.0" 200 152966
I believe the fix should be something like:
--- a/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
+++ b/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
@@ -77,7 +77,7 @@ processors:
(?:-|%{NUMBER:aws.elb.backend.http.response.status_code:long})
%{NUMBER:http.request.body.bytes:long}
%{NUMBER:http.response.body.bytes:long}
- \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|HTTP/%{NOTSPACE:http.version})\"
+ \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:_tmp.url_orig}) (?:-|HTTP/%{NOTSPACE:http.version})\"
\"%{DATA:user_agent.original}\"
%{ELBSSL}
ELBTCPLOG: >-
@@ -110,6 +110,11 @@ processors:
field: event.category
value: web
+ - uri_parts:
+ if: 'ctx.http != null'
+ field: _tmp.url_orig
+ ignore_failure: true
+
- set:
if: 'ctx.http == null'
field: 'aws.elb.protocol'Which results in correct display in Kibana Logs:
[aws][access] 1.2.3.4 "GET /blogs? HTTP/2.0" 200 6815
(I'm not sure why it always displays a ? despite there not being a query string, but I believe that's a separate Kibana issue.)
Previous discussion: https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356
All Elastic Stack components are on version 7.13.2.
It looks like the AWS ALB support in the Filebeat AWS module produces
http.request.referrerin its ingest pipeline, while Kibana Observability (such as the Logs section) expectsurl.*fields to be set (probably viauri_parts).Currently, the Kibana Logs section displays access logs as follows, with a missing request path: (This display is apparently produced by generic_webserver.ts.)
I believe the fix should be something like:
Which results in correct display in Kibana Logs:
(I'm not sure why it always displays a
?despite there not being a query string, but I believe that's a separate Kibana issue.)