Skip to content
This repository was archived by the owner on Sep 21, 2023. It is now read-only.

Increase max RPC size to 64MB#281

Merged
faec merged 3 commits intoelastic:mainfrom
faec:rpc-size
Mar 7, 2023
Merged

Increase max RPC size to 64MB#281
faec merged 3 commits intoelastic:mainfrom
faec:rpc-size

Conversation

@faec
Copy link
Copy Markdown
Contributor

@faec faec commented Mar 6, 2023

Increase the size limit of RPC calls to the shipper from its default of 4MB to 64MB.

This PR mitigates but does not fix elastic/beats#34695.

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.md or CHANGELOG-developer.md.

@faec faec added bug Something isn't working Team:Elastic-Agent Label for the Agent team labels Mar 6, 2023
@faec faec requested a review from a team as a code owner March 6, 2023 16:49
@faec faec requested review from belimawr and fearful-symmetry and removed request for a team March 6, 2023 16:49
@mergify mergify bot assigned faec Mar 6, 2023
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 6, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @faec? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 6, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-03-07T16:55:52.706+0000

  • Duration: 17 min 29 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@faec
Copy link
Copy Markdown
Contributor Author

faec commented Mar 6, 2023

/test

opts := []grpc.ServerOption{grpc.Creds(grpcTLS)}
opts := []grpc.ServerOption{
grpc.Creds(grpcTLS),
grpc.MaxRecvMsgSize(64 * (1 << 20)), // Allow RPCs of up to 64MB
Copy link
Copy Markdown

@ycombinator ycombinator Mar 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Looks like elastic-agent-shipper already indirectly pulls in the https://pkg.go.dev/github.com/docker/go-units#pkg-constants package. So you could use 64 * MiB here if you wanted the code to be more self-documenting.

Copy link
Copy Markdown

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a nitpick comment. LGTM.

github.com/dnephin/pflag v1.0.7 h1:oxONGlWxhmUct0YzKTgrpQv9AUA1wtPBn7zuSjJqptk=
github.com/dnephin/pflag v1.0.7/go.mod h1:uxE91IoWURlOiTUIA8Mq5ZZkAv3dPUfZNaT80Zm7OQE=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
Copy link
Copy Markdown

@ycombinator ycombinator Mar 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Odd. I thought we this module (elastic-agent-shipper) was already using github.com/docker/go-units as an indirect dependency via the github.com/elastic/beats/v7 direct dependency:

 ~/development/github/elastic-agent-shipper >> main $ go mod graph | grep go-units
github.com/elastic/beats/v7@v7.0.0-alpha2.0.20220810153818-dd118efed5a5 github.com/docker/go-units@v0.4.0

Wonder why there was no mention of github.com/docker/go-units in go.sum already, then? 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, it's 0.4.0 vs 0.5.0, when I made the dependency explicit it grabbed the latest version

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But shouldn't there have been two lines for 0.4.0 in go.sum already? I thought go.sum captured all dependencies - direct and indirect - but I guess I'm misunderstanding.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, it looks like there are entries for 0.4.0 and even 0.3.3 in go.sum. I guess unless you run go mod tidy these will stay there. All good, carry on 🙂.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did run go mod tidy; they're still needed as indirect dependencies 😭

@faec faec merged commit 5e26116 into elastic:main Mar 7, 2023
@faec faec deleted the rpc-size branch March 7, 2023 18:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bug Something isn't working Team:Elastic-Agent Label for the Agent team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shipper output fails on large events/batches

4 participants