Skip to content

Commit 250871a

Browse files
moxarth-rathodmoxarth-rathod
committed
[AWS VPC Flow] Add support for v6 and v7 vpcflow logs
1 parent 7978fa3 commit 250871a

File tree

13 files changed

+2284
-32
lines changed

13 files changed

+2284
-32
lines changed

‎packages/aws/_dev/build/docs/vpcflow.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,10 @@ For more information on implementation, see the Amazon documentation on:
5252
This integration supports various plain text VPC flow log formats:
5353

5454
* The default pattern of 14 version 2 fields
55-
* A custom pattern including all 29 fields, version 2 though 5:
55+
* A custom pattern including all 39 fields, version 2 though 7:
5656

5757
```
58-
${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path}
58+
${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path} ${ecs-cluster-arn} ${ecs-cluster-name} ${ecs-container-instance-arn} ${ecs-container-instance-id} ${ecs-container-id} ${ecs-second-container-id} ${ecs-service-name} ${ecs-task-definition-arn} ${ecs-task-arn} ${ecs-task-id}
5959
```
6060

6161
### Advanced options

‎packages/aws/changelog.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# newer versions go on top
2-
- version: "3.14.2"
2+
- version: "3.15.0"
33
changes:
4-
- description: Remove unused agent files.
5-
type: bugfix
6-
link: https://github.com/elastic/integrations/pull/14995
4+
- description: Add support for v6 and v7 vpcflow logs.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/1
77
- version: "3.14.1"
88
changes:
99
- description: Fixed issue where empty DescribeConfigRules responses caused 'index out of bounds' errors in AWS Config integration.

‎packages/aws/data_stream/vpcflow/_dev/deploy/tf/files/test-extra-samples.log‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,6 @@
44
2 123456789010 eni-1235b8ca123456789 89.160.20.156 89.160.20.156 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
55
2 123456789010 eni-1235b8ca123456789 89.160.20.156 89.160.20.156 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK
66
2 123456789010 eni-1235b8ca123456789 89.160.20.156 172.31.16.139 0 0 1 4 336 1432917027 1432917142 ACCEPT OK
7-
2 123456789010 eni-1235b8ca123456789 172.31.16.139 89.160.20.156 0 0 1 4 336 1432917094 1432917142 REJECT OK
7+
2 123456789010 eni-1235b8ca123456789 172.31.16.139 89.160.20.156 0 0 1 4 336 1432917094 1432917142 REJECT OK
8+
3 000000000000 - 2a02:cf40:: 1.128.0.0 42910 80 17 51 33615 1755678754 1755678775 REJECT NODATA - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/backend-cluster backend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/backend-cluster/i-18de3682c033431da i-18de3682c033431da 601a6500-efa dea79277-480 payments-service arn:aws:ecs:us-east-1:000000000000:task-definition/orders:12 arn:aws:ecs:us-east-1:000000000000:task/backend-cluster/e504f037-35c e504f037-35c
9+
3 - 748335378900 tgw-0a12bc34de56f7890 tgw-attach-0456efgh - - vpc-0ghi9012 vpc-0abc1234 - - eni-0dd4ee5f66aa7bcc8 eni-0dd4ee5f66aa7bcc8 - - - 2a02:cf40:: 175.16.199.0 20385 3389 17 9 7659 1755592623 1755592681 SKIPDATA - - - - - - - ingress - -

‎packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-ecs-samples.log‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
3 000000000000 - 1.128.0.0 2a02:cf40:: 64072 22 17 46 14927 1755678619 1755678641 REJECT OK - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/frontend-cluster frontend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/frontend-cluster/i-82bfd814a3c34e21a i-82bfd814a3c34e21a 574fc0af-908 816bc224-160 payments-service arn:aws:ecs:us-east-1:000000000000:task-definition/backend:3 arn:aws:ecs:us-east-1:000000000000:task/frontend-cluster/6adbd0d6-211 6adbd0d6-211
2+
3 000000000000 - 175.16.199.0 67.43.156.0 63715 3306 6 6 1515 1755678634 1755678662 ACCEPT NODATA - - - - - - - - - - - - - egress - arn:aws:ecs:us-east-1:000000000000:cluster/orders-cluster orders-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/orders-cluster/i-1cb81722735e4bcfa i-1cb81722735e4bcfa 7a93be1b-d3f b20d4561-ebb orders-service arn:aws:ecs:us-east-1:000000000000:task-definition/backend:3 arn:aws:ecs:us-east-1:000000000000:task/orders-cluster/b42ecfbe-17f b42ecfbe-17f
3+
3 000000000000 - 216.160.83.56 89.160.20.128 29034 80 17 65 21349 1755678649 1755678705 ACCEPT SKIPDATA - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/frontend-cluster frontend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/frontend-cluster/i-f3f7cae399904a028 i-f3f7cae399904a028 212459c4-064 cff1c25a-463 frontend-service arn:aws:ecs:us-east-1:000000000000:task-definition/payments:7 arn:aws:ecs:us-east-1:000000000000:task/frontend-cluster/ae629347-4b3 ae629347-4b3
4+
3 000000000000 - 81.2.69.142 89.160.20.112 27556 80 17 97 48448 1755678664 1755678675 REJECT OK - - - - - - - - - - - - - egress - arn:aws:ecs:us-east-1:000000000000:cluster/payments-cluster payments-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/payments-cluster/i-98621a74abda45c08 i-98621a74abda45c08 e1b0b179-c52 1bcc4b5f-aa5 backend-service arn:aws:ecs:us-east-1:000000000000:task-definition/payments:7 arn:aws:ecs:us-east-1:000000000000:task/payments-cluster/3e1593ad-498 3e1593ad-498
5+
3 000000000000 - 81.2.69.144 81.2.69.192 63842 80 17 59 43499 1755678679 1755678730 ACCEPT OK - - - - - - - - - - - - - egress - arn:aws:ecs:us-east-1:000000000000:cluster/frontend-cluster frontend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/frontend-cluster/i-917b943179c94ed18 i-917b943179c94ed18 6b212e03-7f5 61717a95-f40 payments-service arn:aws:ecs:us-east-1:000000000000:task-definition/orders:12 arn:aws:ecs:us-east-1:000000000000:task/frontend-cluster/66afe9a3-36f 66afe9a3-36f
6+
3 000000000000 - 81.2.69.192 81.2.69.144 14797 8080 6 35 28455 1755678694 1755678736 ACCEPT OK - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/payments-cluster payments-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/payments-cluster/i-74911fdc5ae144248 i-74911fdc5ae144248 f006614f-149 baa46763-445 backend-service arn:aws:ecs:us-east-1:000000000000:task-definition/frontend:19 arn:aws:ecs:us-east-1:000000000000:task/payments-cluster/546efe2c-008 546efe2c-008
7+
3 000000000000 - 89.160.20.112 81.2.69.142 37855 443 6 35 22598 1755678709 1755678733 REJECT NODATA - - - - - - - - - - - - - egress - arn:aws:ecs:us-east-1:000000000000:cluster/orders-cluster orders-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/orders-cluster/i-bbc31b27d85c4df6b i-bbc31b27d85c4df6b 6d193c93-3e6 f0a94533-dfa orders-service arn:aws:ecs:us-east-1:000000000000:task-definition/frontend:19 arn:aws:ecs:us-east-1:000000000000:task/orders-cluster/a55434f0-3ff a55434f0-3ff
8+
3 000000000000 - 89.160.20.128 216.160.83.56 50702 22 17 12 43027 1755678724 1755678739 REJECT SKIPDATA - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/orders-cluster orders-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/orders-cluster/i-99bcaf339b1e48c9a i-99bcaf339b1e48c9a a68aa4bf-8af 3b4913a9-8ab frontend-service arn:aws:ecs:us-east-1:000000000000:task-definition/payments:7 arn:aws:ecs:us-east-1:000000000000:task/orders-cluster/a2a7bad8-0df a2a7bad8-0df
9+
3 000000000000 - 67.43.156.0 175.16.199.0 10302 443 6 11 40977 1755678739 1755678763 REJECT NODATA - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/frontend-cluster frontend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/frontend-cluster/i-d6a2499d931444a49 i-d6a2499d931444a49 4cd98d99-f22 443408a8-306 backend-service arn:aws:ecs:us-east-1:000000000000:task-definition/frontend:19 arn:aws:ecs:us-east-1:000000000000:task/frontend-cluster/28af8733-a61 28af8733-a61
10+
3 000000000000 - 2a02:cf40:: 1.128.0.0 42910 80 17 51 33615 1755678754 1755678775 REJECT NODATA - - - - - - - - - - - - - ingress - arn:aws:ecs:us-east-1:000000000000:cluster/backend-cluster backend-cluster arn:aws:ecs:us-east-1:000000000000:container-instance/backend-cluster/i-18de3682c033431da i-18de3682c033431da 601a6500-efa dea79277-480 payments-service arn:aws:ecs:us-east-1:000000000000:task-definition/orders:12 arn:aws:ecs:us-east-1:000000000000:task/backend-cluster/e504f037-35c e504f037-35c

0 commit comments

Comments
 (0)