Skip to content

AWS S3 input bulk fix for number_of_workers#13350

Merged
flexitrev merged 1 commit intomainfrom
aws3s_fix_oncemore
Apr 3, 2025
Merged

AWS S3 input bulk fix for number_of_workers#13350
flexitrev merged 1 commit intomainfrom
aws3s_fix_oncemore

Conversation

@flexitrev
Copy link
Copy Markdown
Contributor

@flexitrev flexitrev commented Mar 28, 2025 •
edited
Loading

  • Bug

Proposed commit message

Fixes number_of_workers setting so it is not dependent on S3 bucket collection being enabled

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • [] I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • [N/A] I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Modify number of worker settings on previous version of the integration without enabling S3 bucket selection.
Download policy
Observe that default number of workers is still present or not present at all, relying on aws-s3 input defaults

Modify number of worker settings on this version of the integration without enabling S3 bucket selection.
Download policy
Observe that default number of workers is the value of user input

Related issues

Screenshots

@flexitrev flexitrev changed the title Cloudtrail add actor and target (#12685) AWS S3 input bulk fix for number_of_workers Mar 28, 2025
@flexitrev flexitrev marked this pull request as ready for review March 28, 2025 23:23
@flexitrev flexitrev requested review from a team as code owners March 28, 2025 23:23
@flexitrev flexitrev mentioned this pull request Mar 28, 2025
Closed
3 tasks
@flexitrev flexitrev linked an issue Mar 28, 2025 that may be closed by this pull request
Remove max_number_of_messages for SQS+S3-based inputs #12101
Closed
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Mar 29, 2025 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6
efd6 reviewed Mar 30, 2025
View reviewed changes
@andrewkroh andrewkroh added Integration:sublime_security Sublime Security Integration:trellix_edr_cloud Trellix EDR Cloud Integration:symantec_endpoint_security Symantec Endpoint Security Integration:f5_bigip F5 BIG-IP Integration:canva Canva Integration:sentinel_one_cloud_funnel SentinelOne Cloud Funnel Integration:imperva_cloud_waf Imperva Cloud WAF Integration:aws_bedrock Amazon Bedrock Integration:servicenow ServiceNow Integration:tanium Tanium Integration:amazon_security_lake Amazon Security Lake Integration:aws AWS Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:jamf_protect Jamf Protect (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Mar 31, 2025
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Mar 31, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@flexitrev flexitrev requested a review from efd6 March 31, 2025 15:22
andrewkroh
andrewkroh approved these changes Apr 1, 2025
View reviewed changes
@flexitrev flexitrev enabled auto-merge (squash) April 1, 2025 22:57
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package carbon_black_cloud - 3.0.1 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/3.0.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package f5_bigip - 1.27.1 containing this change is available at https://epr.elastic.co/package/f5_bigip/1.27.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package imperva_cloud_waf - 1.11.3 containing this change is available at https://epr.elastic.co/package/imperva_cloud_waf/1.11.3/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package jamf_protect - 3.0.1 containing this change is available at https://epr.elastic.co/package/jamf_protect/3.0.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package sentinel_one_cloud_funnel - 1.11.1 containing this change is available at https://epr.elastic.co/package/sentinel_one_cloud_funnel/1.11.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package servicenow - 0.12.1 containing this change is available at https://epr.elastic.co/package/servicenow/0.12.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package sublime_security - 1.8.1 containing this change is available at https://epr.elastic.co/package/sublime_security/1.8.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package symantec_endpoint_security - 1.10.1 containing this change is available at https://epr.elastic.co/package/symantec_endpoint_security/1.10.1/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package tanium - 1.16.2 containing this change is available at https://epr.elastic.co/package/tanium/1.16.2/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 3, 2025

Package trellix_edr_cloud - 1.8.1 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.8.1/

@andrewkroh andrewkroh removed Integration:vsphere VMware vSphere Integration:crowdstrike CrowdStrike Integration:google_workspace Google Workspace dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs Integration:cisco_meraki Cisco Meraki Integration:cloud_security_posture Security Posture Management Integration:cloudflare_logpush Cloudflare Logpush Integration:awsfirehose Amazon Data Firehose Integration:prisma_cloud Palo Alto Prisma Cloud Integration:entityanalytics_entra_id Microsoft Entra ID Entity Analytics Integration:ti_crowdstrike CrowdStrike Falcon Intelligence Integration:entityanalytics_ad Active Directory Entity Analytics Integration:azure_openai Azure OpenAI Integration:microsoft_sentinel Microsoft Sentinel Integration:o365_metrics Microsoft Office 365 Metrics Integration:aws_mq Amazon MQ labels Apr 3, 2025
@leandrojmp leandrojmp mentioned this pull request Jul 2, 2025
Closed
@leandrojmp
Copy link
Copy Markdown
Contributor

leandrojmp commented Jul 2, 2025

Hello,

We just found that this issue still happens on the aws_logs package, the AWS Custom Logs integration.

@mykola-elastic mykola-elastic mentioned this pull request Jul 3, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@Kavindu-Dodan Kavindu-Dodan Kavindu-Dodan approved these changes

@efd6 efd6 efd6 approved these changes

@agithomas agithomas agithomas approved these changes

Assignees

No one assigned

Labels

Integration:amazon_security_lake Amazon Security Lake Integration:aws_bedrock Amazon Bedrock Integration:aws AWS Integration:canva Canva Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:f5_bigip F5 BIG-IP Integration:imperva_cloud_waf Imperva Cloud WAF Integration:jamf_protect Jamf Protect (Partner supported) Integration:sentinel_one_cloud_funnel SentinelOne Cloud Funnel Integration:servicenow ServiceNow Integration:sublime_security Sublime Security Integration:symantec_endpoint_security Symantec Endpoint Security Integration:tanium Tanium Integration:trellix_edr_cloud Trellix EDR Cloud Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Cloudflare Logpush] Setting number_of_workers is ignored if SQS is used. Remove max_number_of_messages for SQS+S3-based inputs

7 participants

@flexitrev @elasticmachine @efd6 @Kavindu-Dodan @leandrojmp @andrewkroh @agithomas