Skip to content

Commit a71f099

Browse files
alexreal1314alexreal1314
committed
Merge branch 'main' into 16343-aws-set-host
2 parents 771490c + 2a491c1 commit a71f099

File tree

17 files changed

+158
-6
lines changed

17 files changed

+158
-6
lines changed

‎packages/aws/changelog.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,16 @@
44
- description: Set host.id alongside host.target.entity.id in CloudTrail pipeline for entity store host entity resolution.
55
type: enhancement
66
link: https://github.com/elastic/integrations/pull/17827
7+
- version: "6.3.2"
8+
changes:
9+
- description: Fix s3access ingest pipeline to handle host headers with ports and Source region field parsing.
10+
type: bugfix
11+
link: https://github.com/elastic/integrations/pull/17748
12+
- version: "6.3.1"
13+
changes:
14+
- description: Fix CloudTrail pipeline errors causing transient json and _conf fields to leak into indexed documents by cleaning them up in the on_failure handler, preventing dynamic mapping conflicts.
15+
type: bugfix
16+
link: https://github.com/elastic/integrations/pull/17806
717
- version: "6.3.0"
818
changes:
919
- description: Removed ECS field definitions from CDR transform destinations, now covered by ecs@mappings component template. Renamed overridden ECS fields files to ecs-overridden.yml for awsconfig and awsinspector transforms.

‎packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2107,3 +2107,8 @@ on_failure:
21072107
Processor '{{{ _ingest.on_failure_processor_type }}}'
21082108
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
21092109
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'
2110+
- remove:
2111+
field:
2112+
- json
2113+
- _conf
2114+
ignore_missing: true

‎packages/aws/data_stream/s3access/_dev/test/pipeline/test-s3-server-access.log‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@
99
67797214d75628047d9c76b18a78cded1a4b069b71f2a9d5a53649c38da8770b flow-log-test [14/Jul/2021:18:57:31 +0000] - svc:delivery.logs.amazonaws.com MVGXZXEVN3IG9S24 REST.PUT.OBJECT AWSLogs/000000000000/vpcflowlogs/us-gov-east-1/2021/07/13/000000000000_vpcflowlogs_us-gov-east-1_fl-_20210713T1855Z_f12aa632.log.gz "PUT /AWSLogs/000000000000/vpcflowlogs/us-gov-east-1/2021/07/13/000000000000_vpcflowlogs_us-gov-east-1_fl-0e7c13bf00cf15bfe_20210713T1855Z_f12aa632.log.gz HTTP/1.1" - - - 773 103 13 "-" "-" - 02SxwfXpO5UysN0GsKGa3uGDQ6E/W7+Hwo/luRH8p1VEexULoe66RCM+nja0dEq2JqLrtgjocvVRRkVt4= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader flow-log-test.s3.us-gov-west-1.amazonaws.com TLSv1.2 arn:aws:s3:us-west-1:123456789012:accesspoint/example-AP -
1010
b854390a51155554b82ce2759564a1135bce83133d004f4d2001f157e13985d7 flow-log-test [25/Mar/2025:19:28:02 +0000] - AmazonS3 366DB3C4B325AB11 S3.EXPIRE.OBJECT 0/chum/_vars/logtests/PlannerModule/5f6ea3b7da96ab304a77225d5b2b2a55e54b74e4ddfdf14b9b1d853d77515b88_9febba22f08b11ef8cf6020058a9efab/2024/12/30/164700/kitt_189/_spcu_sride__state_svx__feature__flags.sst "-" - - - 317 - - "-" "-" qsEq9bDa2VyxyZ4cz0c7oBnF67VYTTij DMlPb9al4CvVBck150CgpEIIYgtSI3HC/atetNVYwPtHZffW6jfpg+BrffhbT9/B - - - - - - -
1111
b854390a51155554b82ce2759564a1135bce83133d004f4d2001f157e13985d7 flow-log-test [25/Mar/2025:19:28:02 +0000] - AmazonS3 366DB3C4B325AB11 S3.EXPIRE.OBJECT 0/chum/_vars/logtests/PlannerModule/5f6ea3b7da96ab304a77225d5b2b2a55e54b74e4ddfdf14b9b1d853d77515b88_9febba22f08b11ef8cf6020058a9efab/2024/12/30/164700/kitt_189/_spcu_sride__state_svx__feature__flags.sst "-" - - - 317 - - "-" "-" qsEq9bDa2VyxyZ4cz0c7oBnF67VYTTij DMlPb9al4CvVBck150CgpEIIYgtSI3HC/atetNVYwPtHZffW6jfpg+BrffhbT9/B - - - - - - Yes
12-
b854390a51155554b82ce2759564a1135bce83133d004f4d2001f157e13985d7 flow-log-test [25/Mar/2025:19:28:02 +0000] - AmazonS3 366DB3C4B325AB11 S3.PUT.OBJECT test/special(chars*'!%)/75*75/log.txt "-" - - - 317 - - "-" "-" qsEq9bDa2VyxyZ4cz0c7oBnF67VYTTij DMlPb9al4CvVBck150CgpEIIYgtSI3HC/atetNVYwPtHZffW6jfpg+BrffhbT9/B - - - - - - Yes
12+
b854390a51155554b82ce2759564a1135bce83133d004f4d2001f157e13985d7 flow-log-test [25/Mar/2025:19:28:02 +0000] - AmazonS3 366DB3C4B325AB11 S3.PUT.OBJECT test/special(chars*'!%)/75*75/log.txt "-" - - - 317 - - "-" "-" qsEq9bDa2VyxyZ4cz0c7oBnF67VYTTij DMlPb9al4CvVBck150CgpEIIYgtSI3HC/atetNVYwPtHZffW6jfpg+BrffhbT9/B - - - - - - Yes
13+
28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef flow-log-test [09/Mar/2026:18:02:43 +0000] 89.160.20.156 - 366DB3C4B325AB11 REST.GET.BUCKET - "GET / HTTP/1.1" 403 AccessDenied 243 - 13 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/143.0.7499.4 Safari/537.36" - QdHzj6QLWVMoGDnJfussGKPweg8C0I2WgSv4z3pyYQdGPwWM4jl/0MXIVyrDVaWbPTC+kQDuxEQ= - - - test.s3.us-east-1.amazonaws.com:80 - - us-east-1

‎packages/aws/data_stream/s3access/_dev/test/pipeline/test-s3-server-access.log-expected.json‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1055,6 +1055,115 @@
10551055
"tags": [
10561056
"preserve_original_event"
10571057
]
1058+
},
1059+
{
1060+
"@timestamp": "2026-03-09T18:02:43.000Z",
1061+
"aws": {
1062+
"s3access": {
1063+
"bucket": "flow-log-test",
1064+
"bucket_owner": "28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef",
1065+
"bytes_sent": 243,
1066+
"error_code": "AccessDenied",
1067+
"host_header": "test.s3.us-east-1.amazonaws.com:80",
1068+
"host_id": "QdHzj6QLWVMoGDnJfussGKPweg8C0I2WgSv4z3pyYQdGPwWM4jl/0MXIVyrDVaWbPTC+kQDuxEQ=",
1069+
"http_status": 403,
1070+
"operation": "REST.GET.BUCKET",
1071+
"remote_ip": "89.160.20.156",
1072+
"request_id": "366DB3C4B325AB11",
1073+
"request_uri": "GET / HTTP/1.1",
1074+
"source_region": "us-east-1",
1075+
"total_time": 13,
1076+
"user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/143.0.7499.4 Safari/537.36"
1077+
}
1078+
},
1079+
"client": {
1080+
"address": "89.160.20.156",
1081+
"geo": {
1082+
"city_name": "Linköping",
1083+
"continent_name": "Europe",
1084+
"country_iso_code": "SE",
1085+
"country_name": "Sweden",
1086+
"location": {
1087+
"lat": 58.4167,
1088+
"lon": 15.6167
1089+
},
1090+
"region_iso_code": "SE-E",
1091+
"region_name": "Östergötland County"
1092+
},
1093+
"ip": "89.160.20.156"
1094+
},
1095+
"cloud": {
1096+
"provider": "aws",
1097+
"region": "us-east-1"
1098+
},
1099+
"ecs": {
1100+
"version": "8.11.0"
1101+
},
1102+
"event": {
1103+
"action": "REST.GET.BUCKET",
1104+
"category": [
1105+
"web"
1106+
],
1107+
"code": "AccessDenied",
1108+
"duration": 13000000,
1109+
"id": "366DB3C4B325AB11",
1110+
"kind": "event",
1111+
"original": "28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef flow-log-test [09/Mar/2026:18:02:43 +0000] 89.160.20.156 - 366DB3C4B325AB11 REST.GET.BUCKET - \"GET / HTTP/1.1\" 403 AccessDenied 243 - 13 - \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/143.0.7499.4 Safari/537.36\" - QdHzj6QLWVMoGDnJfussGKPweg8C0I2WgSv4z3pyYQdGPwWM4jl/0MXIVyrDVaWbPTC+kQDuxEQ= - - - test.s3.us-east-1.amazonaws.com:80 - - us-east-1",
1112+
"outcome": "failure",
1113+
"type": [
1114+
"access"
1115+
]
1116+
},
1117+
"geo": {
1118+
"city_name": "Linköping",
1119+
"continent_name": "Europe",
1120+
"country_iso_code": "SE",
1121+
"country_name": "Sweden",
1122+
"location": {
1123+
"lat": 58.4167,
1124+
"lon": 15.6167
1125+
},
1126+
"region_iso_code": "SE-E",
1127+
"region_name": "Östergötland County"
1128+
},
1129+
"http": {
1130+
"request": {
1131+
"method": "GET"
1132+
},
1133+
"response": {
1134+
"body": {
1135+
"bytes": 243
1136+
},
1137+
"status_code": 403
1138+
},
1139+
"version": "1.1"
1140+
},
1141+
"related": {
1142+
"ip": [
1143+
"89.160.20.156"
1144+
],
1145+
"user": [
1146+
"28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef"
1147+
]
1148+
},
1149+
"tags": [
1150+
"preserve_original_event"
1151+
],
1152+
"url": {
1153+
"original": "/",
1154+
"path": "/"
1155+
},
1156+
"user_agent": {
1157+
"device": {
1158+
"name": "Other"
1159+
},
1160+
"name": "HeadlessChrome",
1161+
"original": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/143.0.7499.4 Safari/537.36",
1162+
"os": {
1163+
"name": "Linux"
1164+
},
1165+
"version": "143.0.7499"
1166+
}
10581167
}
10591168
]
10601169
}

‎packages/aws/data_stream/s3access/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
description: "Pipeline for s3 server access logs"
2+
description: "Pipeline for S3 server access logs"
33

44
processors:
55
- set:
@@ -33,14 +33,16 @@ processors:
3333
(?:-|%{NUMBER:aws.s3access.object_size:long}) (?:-|%{NUMBER:aws.s3access.total_time:long}) (?:-|%{NUMBER:aws.s3access.turn_around_time:long})
3434
(?:-|\"-\"|\"%{DATA:aws.s3access.referrer}\") (?:-|\"(-|%{DATA:aws.s3access.user_agent})\") (?:-|%{S3KEY:aws.s3access.version_id})
3535
(?:-|%{S3ID:aws.s3access.host_id}) (?:-|%{S3VERSION:aws.s3access.signature_version}) (?:-|%{S3KEY:aws.s3access.cipher_suite})
36-
(?:-|%{WORD:aws.s3access.authentication_type}) (?:-|%{S3ID:aws.s3access.host_header}) (?:-|%{S3VERSION:aws.s3access.tls_version})(?: (?:-|%{S3REQUESTER:aws.s3access.access_point_arn}))?(?: (?:-|%{WORD:aws.s3access.aclrequired}))?
36+
(?:-|%{WORD:aws.s3access.authentication_type}) (?:-|%{S3ID:aws.s3access.host_header}) (?:-|%{S3VERSION:aws.s3access.tls_version})(?: (?:-|%{S3REQUESTER:aws.s3access.access_point_arn}))?(?: (?:-|%{S3ACLREQUIRED:aws.s3access.aclrequired}))?(?: (?:-|%{S3REGION:aws.s3access.source_region}))?(?: %{S3REGION:aws.s3access.source_region})?
3737
pattern_definitions:
3838
S3REQUESTER: "[a-zA-Z0-9\\/_\\.\\-%:@]+"
3939
S3REQUESTID: "[a-zA-Z0-9]+"
4040
S3OPERATION: "%{WORD}.%{WORD}.%{WORD}"
4141
S3KEY: "[a-zA-Z0-9\\/\\_\\!\\-\\.\\*\\'\\(\\)\\%\\+]+"
42-
S3ID: "[a-zA-Z0-9\\/_\\.\\-%+=]+"
42+
S3ID: "[a-zA-Z0-9\\/_\\.\\-%+=:]+"
4343
S3VERSION: "[a-zA-Z0-9.]+"
44+
S3ACLREQUIRED: "(-|Yes)"
45+
S3REGION: "[a-zA-Z][a-zA-Z0-9-]*"
4446
- grok:
4547
field: aws.s3access.host_header
4648
ignore_missing: true
@@ -187,6 +189,10 @@ processors:
187189
field: aws.s3access.aclrequired
188190
value: '{{aws.s3access.aclrequired}}'
189191
ignore_empty_value: true
192+
- set:
193+
field: aws.s3access.source_region
194+
value: '{{aws.s3access.source_region}}'
195+
ignore_empty_value: true
190196
- set:
191197
field: cloud.provider
192198
value: aws

‎packages/aws/data_stream/s3access/fields/fields.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,10 @@
9797
type: keyword
9898
description: |
9999
Whether the ACL was required for the request.
100+
- name: source_region
101+
type: keyword
102+
description: |
103+
The AWS Region from which the request originated.
100104
- name: access_point_arn
101105
type: keyword
102106
description: |

‎packages/aws/docs/s3.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ Refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ec
8282
| aws.s3access.request_uri | The Request-URI part of the HTTP request message. | keyword |
8383
| aws.s3access.requester | The canonical user ID of the requester, or a - for unauthenticated requests. | keyword |
8484
| aws.s3access.signature_version | The signature version, SigV2 or SigV4, that was used to authenticate the request or a - for unauthenticated requests. | keyword |
85+
| aws.s3access.source_region | The AWS Region from which the request originated. | keyword |
8586
| aws.s3access.tls_version | The Transport Layer Security (TLS) version negotiated by the client. | keyword |
8687
| aws.s3access.total_time | The total amount of time in milliseconds the request was in flight from the server's perspective. | long |
8788
| aws.s3access.turn_around_time | The total amount of time in milliseconds that Amazon S3 spent processing your request. | long |

‎packages/gitlab/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "2.5.0"
3+
changes:
4+
- description: Add explicit template_path to data streams using filestream input.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/17752
27
- version: "2.4.0"
38
changes:
49
- description: Remove the `forwarded` tag from filestream input by default.

‎packages/gitlab/data_stream/audit/manifest.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ title: "Audit"
22
type: logs
33
streams:
44
- input: filestream
5+
template_path: filestream.yml.hbs
56
title: GitLab Audit Logs
67
description: Gitlab Audit Logs
78
vars:

‎packages/gitlab/data_stream/auth/manifest.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ title: "Auth"
22
type: logs
33
streams:
44
- input: filestream
5+
template_path: filestream.yml.hbs
56
title: GitLab Auth Logs
67
description: GitLab Auth Logs
78
vars:

0 commit comments

Comments
 (0)