Skip to content

Add persistence acknowledgement support for shipper#32708

Merged
rdner merged 7 commits intoelastic:mainfrom
rdner:shipper-output-ack
Aug 23, 2022
Merged

Add persistence acknowledgement support for shipper#32708
rdner merged 7 commits intoelastic:mainfrom
rdner:shipper-output-ack

Conversation

@rdner
Copy link
Copy Markdown
Member

@rdner rdner commented Aug 16, 2022

What does this PR do?

Now the shipper output is able to receive acknowledgement of the
persisted events from the shipper server.

Also, added the UUID verification in case the client is reconnected to
a different server.

Why is it important?

To avoid data loss.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@rdner rdner self-assigned this Aug 16, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 16, 2022
@rdner rdner force-pushed the shipper-output-ack branch 3 times, most recently from d383836 to 9d1cde0 Compare August 16, 2022 14:15
rdner
rdner commented Aug 16, 2022
View reviewed changes
rdner
rdner commented Aug 16, 2022
View reviewed changes
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 16, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-08-22T15:12:09.511+0000

  • Duration: 126 min 34 sec

Test stats 🧪

Test Results
Failed 0
Passed 22656
Skipped 1937
Total 24593

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@rdner
Add persistence acknowledgement support for shipper
bd87ee6 
Now the shipper output is able to receive acknowledgement of the
persisted events from the shipper server.

Also, added the UUID verification in case the client is reconnected to
a different server.
@rdner rdner force-pushed the shipper-output-ack branch from 9d1cde0 to bd87ee6 Compare August 16, 2022 14:49
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Aug 16, 2022

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @rdner? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)
    To fixup this pull request, you need to add the backport labels for the needed
    branches, such as:
  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@rdner rdner added the backport-skip Skip notification from the automated backport with mergify label Aug 16, 2022
@rdner
Copy link
Copy Markdown
Member Author

rdner commented Aug 16, 2022

E2E test failure is not related to the change:

[2022-08-16T17:23:54.410Z] time="2022-08-16T17:23:52Z" level=fatal msg="could not add package to policy; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.logfile.streams.system.auth.vars.preserve_original_event: Preserve original event is required\ninputs.logfile.streams.system.auth.vars.tags: Tags is required"}"

@rdner rdner marked this pull request as ready for review August 16, 2022 18:18
@rdner rdner requested a review from a team as a code owner August 16, 2022 18:18
@rdner rdner requested review from a team, belimawr, cmacknz, faec and leehinman and removed request for a team August 16, 2022 18:18
@rdner rdner added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Aug 16, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Aug 16, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 16, 2022
@rdner rdner added the v8.5.0 label Aug 16, 2022
cmacknz
cmacknz reviewed Aug 17, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@cmacknz cmacknz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good. I think we'll need a way to benchmark throughput between a beat and the shipper to really know how well this performs.

@cmacknz cmacknz mentioned this pull request Aug 17, 2022
Closed
@faec faec mentioned this pull request Aug 18, 2022
Closed
faec
faec reviewed Aug 18, 2022
View reviewed changes
rdner added 2 commits August 22, 2022 13:59
@rdner
Remove redundant code
7426f1a
@rdner
Check UUID only once before the loop
f6795c3 
Also terminate the stream properly with the context.
@rdner rdner requested a review from faec August 23, 2022 11:06
@rdner rdner merged commit f992631 into elastic:main Aug 23, 2022
@rdner rdner deleted the shipper-output-ack branch August 23, 2022 16:50
@rdner rdner mentioned this pull request Aug 23, 2022
Merged
4 tasks
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@rdner @chrisberkhout
Add persistence acknowledgement support for shipper (#32708)
bb29dce 
Now the shipper output is able to receive acknowledgement of the
persisted events from the shipper server.

Also, added the UUID verification in case the client is reconnected to
a different server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cmacknz cmacknz cmacknz approved these changes

@belimawr belimawr Awaiting requested review from belimawr belimawr is a code owner automatically assigned from elastic/elastic-agent-data-plane

@leehinman leehinman Awaiting requested review from leehinman leehinman is a code owner automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec

Assignees

@rdner rdner

Labels

backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.5.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make the shipper output to acknowledge batches based on PersistedIndex not AcceptedCount

4 participants

@rdner @elasticmachine @cmacknz @faec