Skip to content

Detect grpc size limit error to avoid infinite retry#34762

Merged
faec merged 10 commits intoelastic:mainfrom
faec:grpc-size-check
Mar 9, 2023
Merged

Detect grpc size limit error to avoid infinite retry#34762
faec merged 10 commits intoelastic:mainfrom
faec:grpc-size-check

Conversation

@faec
Copy link
Copy Markdown
Contributor

@faec faec commented Mar 7, 2023 •
edited
Loading

In the shipper output, drop batches when Publish returns a ResourceExhausted message, since this indicates a deterministic error that will always fail, which will block the entire pipeline until the beat is restarted.

When #29778 is resolved we can change this case to split the batch instead of dropping it, this is just to prevent deadlocks in the meantime.

Fixes #34700.

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@faec faec added bug Team:Elastic-Agent Label for the Agent team labels Mar 7, 2023
@faec faec self-assigned this Mar 7, 2023
@faec faec requested a review from a team as a code owner March 7, 2023 19:51
@faec faec requested review from rdner and ycombinator and removed request for a team March 7, 2023 19:51
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 7, 2023
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 7, 2023

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 7, 2023
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 7, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @faec? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 7, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-03-07T19:58:14.096+0000

  • Duration: 67 min 33 sec

Test stats 🧪

Test Results
Failed 0
Passed 25987
Skipped 1962
Total 27949

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

ycombinator
ycombinator approved these changes Mar 9, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@faec faec merged commit 02e71a0 into elastic:main Mar 9, 2023
@faec faec deleted the grpc-size-check branch March 9, 2023 15:44
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@faec @chrisberkhout
Detect grpc size limit error to avoid infinite retry (#34762)
6c442a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator ycombinator approved these changes

@cmacknz cmacknz cmacknz approved these changes

@leehinman leehinman leehinman approved these changes

@rdner rdner Awaiting requested review from rdner rdner is a code owner automatically assigned from elastic/elastic-agent-data-plane

@fearful-symmetry fearful-symmetry Awaiting requested review from fearful-symmetry

Assignees

@faec faec

Labels

bug Team:Elastic-Agent Label for the Agent team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Shipper output retries forever on all errors regardless of actual retry settings

5 participants

@faec @elasticmachine @ycombinator @cmacknz @leehinman