perf(p2p/conn): Buffer secret connection writes by ValarDragon · Pull Request #3346 · cometbft/cometbft

ValarDragon · 2024-06-26T15:24:27Z

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end.

I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine.

We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors.

PR checklist

Tests written/updated
Changelog entry added in .changelog (we use unclog to manage our changelog)
Updated relevant documentation (docs/ or spec/) and code comments
Title follows the Conventional Commits spec

…115) * Buffer secret connection writes * Add changelog * Add changelog v2

zmanian · 2024-06-26T20:58:35Z

I think at very least we should also make a breaking change to increase the frame size from 1k.

If we aren't going to adaptive frames....

melekes

Thanks @ValarDragon ❤️

p2p/conn/secret_connection.go

cason

I would request to make this PR simple by just adding the connWriter to buffer writes, which is good in general.

p2p/conn/evil_secret_connection_test.go

p2p/conn/secret_connection.go

ValarDragon · 2024-06-27T20:26:36Z

I think at very least we should also make a breaking change to increase the frame size from 1k.
If we aren't going to adaptive frames....

Agreed! I'm down to (in separate PRs) increase frame size, at least to have that in next coordinate release if putting in another secret transport layer doesn't work out. (Or at minimum raising frame size) The amount of chacha20poly1305 overhead was really surprising to me though, I hope that improves proportionately with larger frames

p2p/conn/secret_connection.go

cason

Good.

Can we really measure the improvements that this provides?

.changelog/unreleased/improvements/3346-buffer-secret-connection-writes.md

p2p/conn/secret_connection.go

…on-writes.md Co-authored-by: Daniel <daniel.cason@informal.systems>

ValarDragon · 2024-07-01T10:55:49Z

Yes will get it! Sorry for delay, two silly issues got in the way. (Snapshot service was down when I first did it, and I didn't download result in time during my second profile so it get retented. Re-running mainnet benchmarks now to get the ratio here)

ValarDragon · 2024-07-01T12:18:05Z

This successfully eliminates the overhead coming from the write packet, but not flush (which makes sense as flush is dealing with the case where we can't fill one frame anyway. This is because the flush size is parameterized to be equal to the frame size right now. I don't know if this is coincidence or design today, they are both freely variable parameters)

We should expect the ratio of seal to file.Write in the sendPacketMsg call (the non-flush throttle case) to be indicative of what we are speeding up. So on this latest benchmark, its a ratio of 2.5 seal : 1.5 file write. Normalized, 1s of sealing needed .6s of net con write.

Originally this case was: 1s of sealing, needed 1.73s of net conn write. So this is an almost 3x speedup to this part of the bottleneck! (And now the unneeded buffer .Put and .Get matter to optimize out)

p2p/conn/secret_connection.go

cason

Minor suggestions. Lets merge this.

…ometbft#115) * Buffer secret connection writes * Add changelog * Add changelog v2

Closes #3198 Similar to #3346 , buffers the secret connection reads. This is a notable savings to CPU time. (25% of recvRoutine time, on Osmosis' version) --- #### PR checklist - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

melekes · 2024-07-10T11:31:43Z

@mergify backport v1.x

mergify · 2024-07-10T11:32:05Z

backport v1.x

✅ Backports have been created

Details

#3485 perf(p2p/conn): Buffer secret connection writes (backport #3346) has been created for branch v1.x

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end. I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine. ![image](https://github.com/cometbft/cometbft/assets/6440154/b7a43188-a69b-41b1-9506-2f66a2d63a74) ![image](https://github.com/cometbft/cometbft/assets/6440154/95f15ff0-94b8-419c-8759-1155d63d32f8) We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors. --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> Co-authored-by: Daniel <daniel.cason@informal.systems> (cherry picked from commit 8422f57)

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end. I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine. ![image](https://github.com/cometbft/cometbft/assets/6440154/b7a43188-a69b-41b1-9506-2f66a2d63a74) ![image](https://github.com/cometbft/cometbft/assets/6440154/95f15ff0-94b8-419c-8759-1155d63d32f8) We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors. --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3346 done by [Mergify](https://mergify.com). Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>

Closes #3198 Similar to #3346 , buffers the secret connection reads. This is a notable savings to CPU time. (25% of recvRoutine time, on Osmosis' version) --- #### PR checklist - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> (cherry picked from commit e1eabe0) # Conflicts: # p2p/conn/evil_secret_connection_test.go

…3419) (#3489) Closes #3198 Similar to #3346 , buffers the secret connection reads. This is a notable savings to CPU time. (25% of recvRoutine time, on Osmosis' version) --- #### PR checklist - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3419 done by [Mergify](https://mergify.com). --------- Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

…) (cometbft#3485) component of cometbft#3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end. I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine. ![image](https://github.com/cometbft/cometbft/assets/6440154/b7a43188-a69b-41b1-9506-2f66a2d63a74) ![image](https://github.com/cometbft/cometbft/assets/6440154/95f15ff0-94b8-419c-8759-1155d63d32f8) We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors. --- - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request cometbft#3346 done by [Mergify](https://mergify.com). Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end. I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine. ![image](https://github.com/cometbft/cometbft/assets/6440154/b7a43188-a69b-41b1-9506-2f66a2d63a74) ![image](https://github.com/cometbft/cometbft/assets/6440154/95f15ff0-94b8-419c-8759-1155d63d32f8) We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors. --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> Co-authored-by: Daniel <daniel.cason@informal.systems>

Closes #3198 Similar to #3346 , buffers the secret connection reads. This is a notable savings to CPU time. (25% of recvRoutine time, on Osmosis' version) --- - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end. I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine. ![image](https://github.com/cometbft/cometbft/assets/6440154/b7a43188-a69b-41b1-9506-2f66a2d63a74) ![image](https://github.com/cometbft/cometbft/assets/6440154/95f15ff0-94b8-419c-8759-1155d63d32f8) We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors. --- #### PR checklist - [ ] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> Co-authored-by: Daniel <daniel.cason@informal.systems>

Closes #3198 Similar to #3346 , buffers the secret connection reads. This is a notable savings to CPU time. (25% of recvRoutine time, on Osmosis' version) --- - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

Buffer secret connection writes

a963b49

ValarDragon requested a review from a team as a code owner June 26, 2024 15:24

ValarDragon requested a review from a team June 26, 2024 15:24

Add changelog

40973fa

zmanian approved these changes Jun 26, 2024

View reviewed changes

ValarDragon added a commit to osmosis-labs/cometbft that referenced this pull request Jun 26, 2024

perf(p2p/secretconn): Buffer secret connection writes cometbft#3346 (#…

9f773de

…115) * Buffer secret connection writes * Add changelog * Add changelog v2

Lower buffer constant

9cdbc95

melekes approved these changes Jun 27, 2024

View reviewed changes

p2p/conn/secret_connection.go Outdated Show resolved Hide resolved

melekes added the p2p label Jun 27, 2024

cason reviewed Jun 27, 2024

View reviewed changes

p2p/conn/evil_secret_connection_test.go Outdated Show resolved Hide resolved

p2p/conn/evil_secret_connection_test.go Outdated Show resolved Hide resolved

p2p/conn/secret_connection.go Outdated Show resolved Hide resolved

p2p/conn/secret_connection.go Show resolved Hide resolved

Apply cason's comments

82cf0e2

ValarDragon commented Jun 27, 2024

View reviewed changes

p2p/conn/secret_connection.go Show resolved Hide resolved

cason approved these changes Jun 28, 2024

View reviewed changes

.changelog/unreleased/improvements/3346-buffer-secret-connection-writes.md Outdated Show resolved Hide resolved

p2p/conn/secret_connection.go Show resolved Hide resolved

cason changed the title ~~perf(p2p/secretconn): Buffer secret connection writes~~ perf(p2p/conn): Buffer secret connection writes Jun 28, 2024

Update .changelog/unreleased/improvements/3346-buffer-secret-connecti…

3bd2f4d

…on-writes.md Co-authored-by: Daniel <daniel.cason@informal.systems>

cason reviewed Jul 1, 2024

View reviewed changes

p2p/conn/secret_connection.go Outdated Show resolved Hide resolved

cason reviewed Jul 1, 2024

View reviewed changes

p2p/conn/secret_connection.go Outdated Show resolved Hide resolved

cason reviewed Jul 1, 2024

View reviewed changes

p2p/conn/secret_connection.go Outdated Show resolved Hide resolved

cason reviewed Jul 1, 2024

View reviewed changes

ValarDragon and others added 3 commits July 2, 2024 17:54

Apply @cason's comments

bfdf174

Reduce commit diff by 1 line

b504419

Merge branch 'main' into dev/buffer_secretconn_writes

1b724d1

cason enabled auto-merge July 3, 2024 07:10

ValarDragon mentioned this pull request Jul 3, 2024

Prevent busy-waiting in consensus gossip routines, when Send fails #3414

Closed

2 tasks

ValarDragon and others added 2 commits July 3, 2024 22:52

Merge branch 'main' into dev/buffer_secretconn_writes

889d2e3

fix merge conflict lint issues

f096de1

One last lint failure

8c4d0d1

cason added this pull request to the merge queue Jul 3, 2024

Merged via the queue into main with commit 8422f57 Jul 3, 2024

cason deleted the dev/buffer_secretconn_writes branch July 3, 2024 22:36

ValarDragon mentioned this pull request Jul 3, 2024

perf(p2p/conn): Use a read buffer on the secret connection #3419

Merged

4 tasks

itsdevbear pushed a commit to berachain/cometbft that referenced this pull request Jul 4, 2024

perf(p2p/secretconn): Buffer secret connection writes cometbft#3346 (c…

5f565c9

…ometbft#115) * Buffer secret connection writes * Add changelog * Add changelog v2

mergify bot mentioned this pull request Jul 10, 2024

perf(p2p/conn): Buffer secret connection writes (backport #3346) #3485

Merged

4 tasks

mergify bot mentioned this pull request Jul 10, 2024

perf(p2p/conn): Use a read buffer on the secret connection (backport #3419) #3489

Merged

4 tasks

Conversation

ValarDragon commented Jun 26, 2024 • edited by cason Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

zmanian commented Jun 26, 2024

Uh oh!

melekes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cason left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ValarDragon commented Jun 27, 2024

Uh oh!

Uh oh!

cason left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ValarDragon commented Jul 1, 2024

Uh oh!

ValarDragon commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cason left a comment

Choose a reason for hiding this comment

Uh oh!

melekes commented Jul 10, 2024

Uh oh!

mergify bot commented Jul 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ValarDragon commented Jun 26, 2024 •

edited by cason

Loading

ValarDragon commented Jul 1, 2024 •

edited

Loading

mergify bot commented Jul 10, 2024 •

edited

Loading