Skip to content

[8.x](backport #40699) Add asynchronous ACK handling to S3 and SQS inputs#41249

Merged
faec merged 2 commits into8.xfrom
mergify/bp/8.x/pr-40699
Oct 16, 2024
Merged

[8.x](backport #40699) Add asynchronous ACK handling to S3 and SQS inputs#41249
faec merged 2 commits into8.xfrom
mergify/bp/8.x/pr-40699

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Oct 16, 2024 •
edited by zube bot
Loading

Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This can best be tested by ingesting data from a live S3 or SQS queue. The scenario that most highlights the changed performance is ingesting many small individual objects.

Related issues

@faec @mergify
Add asynchronous ACK handling to S3 and SQS inputs (#40699)
0501fce 
Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip. With a default configuration, SQS queues with many small objects are now ingested up to 60x faster.

(cherry picked from commit d2867fd)

# Conflicts:
#	x-pack/filebeat/input/awss3/input_benchmark_test.go
#	x-pack/filebeat/input/awss3/sqs_s3_event_test.go
@mergify mergify bot requested review from a team as code owners October 16, 2024 02:53
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Oct 16, 2024
@mergify mergify bot requested a review from a team as a code owner October 16, 2024 02:53
@mergify mergify bot requested review from AndersonQ and leehinman and removed request for a team October 16, 2024 02:53
@mergify mergify bot assigned faec Oct 16, 2024
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Oct 16, 2024

Cherry-pick of d2867fd has failed:

On branch mergify/bp/8.x/pr-40699
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit d2867fdd9f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   go.sum
	modified:   x-pack/filebeat/_meta/config/filebeat.inputs.reference.xpack.yml.tmpl
	modified:   x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
	modified:   x-pack/filebeat/filebeat.reference.yml
	new file:   x-pack/filebeat/input/awss3/acks.go
	modified:   x-pack/filebeat/input/awss3/config.go
	modified:   x-pack/filebeat/input/awss3/config_test.go
	modified:   x-pack/filebeat/input/awss3/input_integration_test.go
	modified:   x-pack/filebeat/input/awss3/interfaces.go
	modified:   x-pack/filebeat/input/awss3/mock_interfaces_test.go
	modified:   x-pack/filebeat/input/awss3/s3.go
	modified:   x-pack/filebeat/input/awss3/s3_input.go
	modified:   x-pack/filebeat/input/awss3/s3_objects.go
	modified:   x-pack/filebeat/input/awss3/s3_objects_test.go
	modified:   x-pack/filebeat/input/awss3/s3_test.go
	modified:   x-pack/filebeat/input/awss3/sqs_input.go
	modified:   x-pack/filebeat/input/awss3/sqs_s3_event.go
	modified:   x-pack/filebeat/input/awss3/sqs_test.go
	modified:   x-pack/filebeat/module/aws/_meta/config.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/manifest.yml
	modified:   x-pack/filebeat/module/aws/s3access/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/s3access/manifest.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/config/input.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/manifest.yml
	modified:   x-pack/filebeat/modules.d/aws.yml.disabled

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/filebeat/input/awss3/input_benchmark_test.go
	both modified:   x-pack/filebeat/input/awss3/sqs_s3_event_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2024
@botelastic
Copy link
Copy Markdown

botelastic bot commented Oct 16, 2024

This pull request doesn't have a Team:<team> label.

@faec faec enabled auto-merge (squash) October 16, 2024 04:37
@faec faec merged commit c5a983f into 8.x Oct 16, 2024
@faec faec deleted the mergify/bp/8.x/pr-40699 branch October 16, 2024 05:09
@khushijain21 khushijain21 mentioned this pull request Jun 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@faec faec faec approved these changes

@AndersonQ AndersonQ Awaiting requested review from AndersonQ AndersonQ was automatically assigned from elastic/elastic-agent-data-plane

@leehinman leehinman Awaiting requested review from leehinman leehinman was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@faec faec

Labels

backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@faec