Skip to content

Commit 6b66136

Browse files
author
kaiyan-sheng
committed
fix aws-s3 input for all data streams
1 parent c352776 commit 6b66136

8 files changed

Lines changed: 265 additions & 37 deletions

File tree

‎packages/aws/changelog.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# newer versions go on top
2-
- version: "2.17.1"
2+
- version: "2.18.0"
33
changes:
4-
- description: Add S3 polling option to s3access data stream
5-
type: bugfix
4+
- description: Add S3 polling option to data streams use aws-s3 input
5+
type: enhancement
66
link: https://github.com/elastic/integrations/pull/10460
77
- version: "2.17.0"
88
changes:

‎packages/aws/data_stream/cloudfront_logs/manifest.yml‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,65 @@ streams:
66
title: AWS CloudFront logs
77
description: Collect AWS CloudFront logs using s3 input
88
vars:
9+
- name: collect_s3_logs
10+
required: true
11+
show_user: true
12+
title: Collect logs via S3 Bucket
13+
description: To Collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue.
14+
type: bool
15+
multi: false
16+
default: false
17+
- name: bucket_arn
18+
type: text
19+
title: "[S3] Bucket ARN"
20+
multi: false
21+
required: false
22+
show_user: true
23+
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
24+
- name: bucket_list_prefix
25+
type: text
26+
title: "[S3] Bucket Prefix"
27+
multi: false
28+
required: false
29+
show_user: false
30+
description: Prefix to apply for the list request to the S3 bucket.
31+
- name: interval
32+
type: text
33+
title: "[S3] Interval"
34+
multi: false
35+
required: false
36+
show_user: false
37+
default: 1m
38+
description: "Time interval for polling listing of the S3 bucket. NOTE: Supported units for this parameter are h/m/s."
39+
- name: number_of_workers
40+
type: integer
41+
title: "[S3] Number of Workers"
42+
multi: false
43+
required: false
44+
show_user: false
45+
default: 5
46+
description: Number of workers that will process the S3 objects listed.
947
- name: visibility_timeout
1048
type: text
11-
title: Visibility Timeout
49+
title: "[SQS] Visibility Timeout"
1250
multi: false
1351
required: false
1452
show_user: false
1553
description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours.
1654
- name: api_timeout
1755
type: text
18-
title: API Timeout
56+
title: "[SQS] API Timeout"
1957
multi: false
2058
required: false
2159
show_user: false
2260
description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value.
2361
- name: queue_url
2462
type: text
25-
title: Queue URL
63+
title: "[SQS] Queue URL"
2664
multi: false
27-
required: true
65+
required: false
2866
show_user: true
29-
description: URL of the AWS SQS queue that messages will be received from.
67+
description: Mandatory if the "Collect logs via S3 Bucket" switch is off. URL of the AWS SQS queue that messages will be received from.
3068
- name: endpoint
3169
type: text
3270
title: Endpoint
@@ -42,7 +80,7 @@ streams:
4280
required: false
4381
show_user: false
4482
default: ""
45-
description: The name of the AWS region of the endpoint
83+
description: The name of the AWS region of the end point
4684
- name: fips_enabled
4785
type: bool
4886
title: Enable S3 FIPS

‎packages/aws/data_stream/ec2_logs/manifest.yml‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,27 +7,65 @@ streams:
77
description: Collect AWS EC2 logs using s3 input
88
enabled: false
99
vars:
10+
- name: collect_s3_logs
11+
required: true
12+
show_user: true
13+
title: Collect logs via S3 Bucket
14+
description: To Collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue.
15+
type: bool
16+
multi: false
17+
default: false
18+
- name: bucket_arn
19+
type: text
20+
title: "[S3] Bucket ARN"
21+
multi: false
22+
required: false
23+
show_user: true
24+
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
25+
- name: bucket_list_prefix
26+
type: text
27+
title: "[S3] Bucket Prefix"
28+
multi: false
29+
required: false
30+
show_user: false
31+
description: Prefix to apply for the list request to the S3 bucket.
32+
- name: interval
33+
type: text
34+
title: "[S3] Interval"
35+
multi: false
36+
required: false
37+
show_user: false
38+
default: 1m
39+
description: "Time interval for polling listing of the S3 bucket. NOTE: Supported units for this parameter are h/m/s."
40+
- name: number_of_workers
41+
type: integer
42+
title: "[S3] Number of Workers"
43+
multi: false
44+
required: false
45+
show_user: false
46+
default: 5
47+
description: Number of workers that will process the S3 objects listed.
1048
- name: visibility_timeout
1149
type: text
12-
title: Visibility Timeout
50+
title: "[SQS] Visibility Timeout"
1351
multi: false
1452
required: false
1553
show_user: false
1654
description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours.
1755
- name: api_timeout
1856
type: text
19-
title: API Timeout
57+
title: "[SQS] API Timeout"
2058
multi: false
2159
required: false
2260
show_user: false
2361
description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value.
2462
- name: queue_url
2563
type: text
26-
title: Queue URL
64+
title: "[SQS] Queue URL"
2765
multi: false
28-
required: true
66+
required: false
2967
show_user: true
30-
description: URL of the AWS SQS queue that messages will be received from.
68+
description: Mandatory if the "Collect logs via S3 Bucket" switch is off. URL of the AWS SQS queue that messages will be received from.
3169
- name: endpoint
3270
type: text
3371
title: Endpoint
@@ -43,7 +81,7 @@ streams:
4381
required: false
4482
show_user: false
4583
default: ""
46-
description: The name of the AWS region of the endpoint
84+
description: The name of the AWS region of the end point
4785
- name: fips_enabled
4886
type: bool
4987
title: Enable S3 FIPS

‎packages/aws/data_stream/elb_logs/manifest.yml‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,65 @@ streams:
66
title: AWS ELB Logs via S3
77
description: Collect AWS ELB logs using s3 input
88
vars:
9+
- name: collect_s3_logs
10+
required: true
11+
show_user: true
12+
title: Collect logs via S3 Bucket
13+
description: To Collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue.
14+
type: bool
15+
multi: false
16+
default: false
17+
- name: bucket_arn
18+
type: text
19+
title: "[S3] Bucket ARN"
20+
multi: false
21+
required: false
22+
show_user: true
23+
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
24+
- name: bucket_list_prefix
25+
type: text
26+
title: "[S3] Bucket Prefix"
27+
multi: false
28+
required: false
29+
show_user: false
30+
description: Prefix to apply for the list request to the S3 bucket.
31+
- name: interval
32+
type: text
33+
title: "[S3] Interval"
34+
multi: false
35+
required: false
36+
show_user: false
37+
default: 1m
38+
description: "Time interval for polling listing of the S3 bucket. NOTE: Supported units for this parameter are h/m/s."
39+
- name: number_of_workers
40+
type: integer
41+
title: "[S3] Number of Workers"
42+
multi: false
43+
required: false
44+
show_user: false
45+
default: 5
46+
description: Number of workers that will process the S3 objects listed.
947
- name: visibility_timeout
1048
type: text
11-
title: Visibility Timeout
49+
title: "[SQS] Visibility Timeout"
1250
multi: false
1351
required: false
1452
show_user: false
1553
description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours.
1654
- name: api_timeout
1755
type: text
18-
title: API Timeout
56+
title: "[SQS] API Timeout"
1957
multi: false
2058
required: false
2159
show_user: false
2260
description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value.
2361
- name: queue_url
2462
type: text
25-
title: Queue URL
63+
title: "[SQS] Queue URL"
2664
multi: false
27-
required: true
65+
required: false
2866
show_user: true
29-
description: URL of the AWS SQS queue that messages will be received from.
67+
description: Mandatory if the "Collect logs via S3 Bucket" switch is off. URL of the AWS SQS queue that messages will be received from.
3068
- name: endpoint
3169
type: text
3270
title: Endpoint
@@ -42,7 +80,7 @@ streams:
4280
required: false
4381
show_user: false
4482
default: ""
45-
description: The name of the AWS region of the endpoint
83+
description: The name of the AWS region of the end point
4684
- name: fips_enabled
4785
type: bool
4886
title: Enable S3 FIPS

‎packages/aws/data_stream/firewall_logs/manifest.yml‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,65 @@ streams:
66
title: AWS Network Firewall logs
77
description: Collect AWS Network Firewall logs using s3 input
88
vars:
9+
- name: collect_s3_logs
10+
required: true
11+
show_user: true
12+
title: Collect logs via S3 Bucket
13+
description: To Collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue.
14+
type: bool
15+
multi: false
16+
default: false
17+
- name: bucket_arn
18+
type: text
19+
title: "[S3] Bucket ARN"
20+
multi: false
21+
required: false
22+
show_user: true
23+
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
24+
- name: bucket_list_prefix
25+
type: text
26+
title: "[S3] Bucket Prefix"
27+
multi: false
28+
required: false
29+
show_user: false
30+
description: Prefix to apply for the list request to the S3 bucket.
31+
- name: interval
32+
type: text
33+
title: "[S3] Interval"
34+
multi: false
35+
required: false
36+
show_user: false
37+
default: 1m
38+
description: "Time interval for polling listing of the S3 bucket. NOTE: Supported units for this parameter are h/m/s."
39+
- name: number_of_workers
40+
type: integer
41+
title: "[S3] Number of Workers"
42+
multi: false
43+
required: false
44+
show_user: false
45+
default: 5
46+
description: Number of workers that will process the S3 objects listed.
947
- name: visibility_timeout
1048
type: text
11-
title: Visibility Timeout
49+
title: "[SQS] Visibility Timeout"
1250
multi: false
1351
required: false
1452
show_user: false
1553
description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours.
1654
- name: api_timeout
1755
type: text
18-
title: API Timeout
56+
title: "[SQS] API Timeout"
1957
multi: false
2058
required: false
2159
show_user: false
2260
description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value.
2361
- name: queue_url
2462
type: text
25-
title: Queue URL
63+
title: "[SQS] Queue URL"
2664
multi: false
27-
required: true
65+
required: false
2866
show_user: true
29-
description: URL of the AWS SQS queue that messages will be received from.
67+
description: Mandatory if the "Collect logs via S3 Bucket" switch is off. URL of the AWS SQS queue that messages will be received from.
3068
- name: endpoint
3169
type: text
3270
title: Endpoint

‎packages/aws/data_stream/route53_resolver_logs/manifest.yml‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -122,27 +122,65 @@ streams:
122122
title: AWS Route 53 Resolver Logs
123123
description: Collect AWS Resolver logs using s3 input
124124
vars:
125+
- name: collect_s3_logs
126+
required: true
127+
show_user: true
128+
title: Collect logs via S3 Bucket
129+
description: To Collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue.
130+
type: bool
131+
multi: false
132+
default: false
133+
- name: bucket_arn
134+
type: text
135+
title: "[S3] Bucket ARN"
136+
multi: false
137+
required: false
138+
show_user: true
139+
description: Mandatory if the "Collect logs via S3 Bucket" switch is on. It is a required parameter for collecting logs via the AWS S3 Bucket.
140+
- name: bucket_list_prefix
141+
type: text
142+
title: "[S3] Bucket Prefix"
143+
multi: false
144+
required: false
145+
show_user: false
146+
description: Prefix to apply for the list request to the S3 bucket.
147+
- name: interval
148+
type: text
149+
title: "[S3] Interval"
150+
multi: false
151+
required: false
152+
show_user: false
153+
default: 1m
154+
description: "Time interval for polling listing of the S3 bucket. NOTE: Supported units for this parameter are h/m/s."
155+
- name: number_of_workers
156+
type: integer
157+
title: "[S3] Number of Workers"
158+
multi: false
159+
required: false
160+
show_user: false
161+
default: 5
162+
description: Number of workers that will process the S3 objects listed.
125163
- name: visibility_timeout
126164
type: text
127-
title: Visibility Timeout
165+
title: "[SQS] Visibility Timeout"
128166
multi: false
129167
required: false
130168
show_user: false
131169
description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours.
132170
- name: api_timeout
133171
type: text
134-
title: API Timeout
172+
title: "[SQS] API Timeout"
135173
multi: false
136174
required: false
137175
show_user: false
138176
description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value.
139177
- name: queue_url
140178
type: text
141-
title: Queue URL
179+
title: "[SQS] Queue URL"
142180
multi: false
143-
required: true
181+
required: false
144182
show_user: true
145-
description: URL of the AWS SQS queue that messages will be received from.
183+
description: Mandatory if the "Collect logs via S3 Bucket" switch is off. URL of the AWS SQS queue that messages will be received from.
146184
- name: endpoint
147185
type: text
148186
title: Endpoint

0 commit comments

Comments
 (0)