perf(p2p/conn): Remove p2p allocations for wrapping outbound packets#3423
Merged
perf(p2p/conn): Remove p2p allocations for wrapping outbound packets#3423
Conversation
3 tasks
Contributor
Author
|
This did succesfully lower the CPU time as expected, and also appears to have lowered GC by ~10%. (Very interestingly, after this change the largest CPU time segment on Osmosis is no longer GC, now ranking at third, and is now instead the scheduler) |
cason
reviewed
Jul 5, 2024
cason
left a comment
There was a problem hiding this comment.
Great work.
Left, again, some implementation-related questions.
|
Maybe a stupid question, but why do we use protobufs for low-level communication (packets, ping, pong)? Of course we have an overhead here, but what is the advantage of that? Those messages are super simple, size-prefixed messages, ordinary network exchanging []byte s. |
Collaborator
There isn't any advantage imho. |
mergify bot
pushed a commit
that referenced
this pull request
Jul 10, 2024
…3423) Closes #2951 Remove p2p allocations for wrapping outbound packets. Every packet send would require 3 new heap objects due to how we used pointers / did this wrapping. This costs us: 13% of all system heap allocations, and 20% of the CPU time of writing packets (under "normal" osmosis load. I don't have big blocks benchmarks after the secret connection improvement got in). This should be a 25% speedup to `channel.writePacketMsgTo`, making writing getting closer to the bound of just proto-marshalling a packet. This solution is simple (just make a buffer in the struct) since these packet sends are all single threaded. --- #### PR checklist - [X] Tests written/updated - N/A, existing tests cover safety here - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [X] Updated relevant documentation (`docs/` or `spec/`) and code comments - [X] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> (cherry picked from commit 35894b2)
4 tasks
melekes
added a commit
that referenced
this pull request
Jul 10, 2024
…(backport #3423) (#3465) Closes #2951 Remove p2p allocations for wrapping outbound packets. Every packet send would require 3 new heap objects due to how we used pointers / did this wrapping. This costs us: 13% of all system heap allocations, and 20% of the CPU time of writing packets (under "normal" osmosis load. I don't have big blocks benchmarks after the secret connection improvement got in). This should be a 25% speedup to `channel.writePacketMsgTo`, making writing getting closer to the bound of just proto-marshalling a packet. This solution is simple (just make a buffer in the struct) since these packet sends are all single threaded. --- #### PR checklist - [X] Tests written/updated - N/A, existing tests cover safety here - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [X] Updated relevant documentation (`docs/` or `spec/`) and code comments - [X] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3423 done by [Mergify](https://mergify.com). Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
mattac21
pushed a commit
that referenced
this pull request
Sep 9, 2025
…3423) Closes #2951 Remove p2p allocations for wrapping outbound packets. Every packet send would require 3 new heap objects due to how we used pointers / did this wrapping. This costs us: 13% of all system heap allocations, and 20% of the CPU time of writing packets (under "normal" osmosis load. I don't have big blocks benchmarks after the secret connection improvement got in). This should be a 25% speedup to `channel.writePacketMsgTo`, making writing getting closer to the bound of just proto-marshalling a packet. This solution is simple (just make a buffer in the struct) since these packet sends are all single threaded. --- - [X] Tests written/updated - N/A, existing tests cover safety here - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [X] Updated relevant documentation (`docs/` or `spec/`) and code comments - [X] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
mattac21
pushed a commit
that referenced
this pull request
Sep 10, 2025
…3423) Closes #2951 Remove p2p allocations for wrapping outbound packets. Every packet send would require 3 new heap objects due to how we used pointers / did this wrapping. This costs us: 13% of all system heap allocations, and 20% of the CPU time of writing packets (under "normal" osmosis load. I don't have big blocks benchmarks after the secret connection improvement got in). This should be a 25% speedup to `channel.writePacketMsgTo`, making writing getting closer to the bound of just proto-marshalling a packet. This solution is simple (just make a buffer in the struct) since these packet sends are all single threaded. --- - [X] Tests written/updated - N/A, existing tests cover safety here - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [X] Updated relevant documentation (`docs/` or `spec/`) and code comments - [X] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2951
Remove p2p allocations for wrapping outbound packets. Every packet send would require 3 new heap objects due to how we used pointers / did this wrapping. This costs us: 13% of all system heap allocations, and 20% of the CPU time of writing packets (under "normal" osmosis load. I don't have big blocks benchmarks after the secret connection improvement got in).
This should be a 25% speedup to
channel.writePacketMsgTo, making writing getting closer to the bound of just proto-marshalling a packet.This solution is simple (just make a buffer in the struct) since these packet sends are all single threaded.
PR checklist
.changelog(we use unclog to manage our changelog)docs/orspec/) and code comments