perf(metarepos): add mrpb.StorageNodeUncommitReport pool#446
Merged
Conversation
This was referenced May 16, 2023
Member
Author
This was referenced May 16, 2023
Codecov ReportPatch coverage:
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. Additional details and impacted files@@ Coverage Diff @@
## mr_remove_otel_label #446 +/- ##
========================================================
+ Coverage 63.24% 63.25% +0.01%
========================================================
Files 131 131
Lines 17914 17914
========================================================
+ Hits 11329 11332 +3
Misses 6017 6017
+ Partials 568 565 -3
☔ View full report in Codecov by Sentry. |
This was referenced May 16, 2023
This patch adds an `mrpb.StorageNodeUncommitReport` pool, which is
`proto/mrpb.storageNodeUncommitReportPool`. It reduces heap objects allocated repeatedly whenever
processing reports received from storage nodes.
Below heap profiling is the previous result.
```
flat flat% sum% cum cum%
70865580 8.38% 44.73% 71293773 8.43% github.com/kakao/varlog/internal/metarepos.(*reportCollectExecutor).processReport
```
Now here it is after applying this patch.
```
flat flat% sum% cum cum%
20734504 2.10% 75.92% 42122814 4.26% github.com/kakao/varlog/internal/metarepos.(*reportCollectExecutor).processReport
```
626fe12 to
0d5c17a
Compare
fe4c70e to
4c624da
Compare
Member
Author
Member
Author
ijsong
added a commit
that referenced
this pull request
Jul 20, 2023
This change makes mrpb.StorageNodeUncommitReport is reused when it is replaced with a new one. Previous PR, #446, introduced a pool for mrpb.StorageNodeUncommitReport. However, the old StorageNodeUncommitReport stored to reportContext was not reused, and this PR improves it. To release the StorageNodeUncommitReport registered to reportContext, some methods of reportContext are changed: - (*reportContext).getReport returns a value of StorageNodeUncommitReport rather than a pointer. It makes releasing the StorageNodeUncommitReport happen at any time. - add (*reportContext).swapReport to set a new StorageNodeUncommitReport and get the old one simultaneously. - getReport and swapReport return the boolean result to indicate whether the result report is zero value. BenchmarkSwapReport evaluates the performance improvements of this change. - getAndSavePtr: Previous implementation. - getAndSave: Call getReport and saveReport. - swap: Call swapReport. ``` BenchmarkSwapReport/getAndSavePtr-16 5947423 207.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6220454 198.7 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6133489 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6131910 199.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 5592459 201.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6180243 200.1 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6172357 199.2 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6211966 198.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6095289 198.3 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6043491 197.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6143407 200.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5938540 203.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6162986 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6172598 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6105918 197.0 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6077460 196.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5953898 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6131799 199.9 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6139296 200.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6115653 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/swap-16 35501544 33.83 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35982945 34.05 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36518960 34.21 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34620783 34.52 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35967453 33.88 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36135454 34.17 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34492548 35.16 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35581888 34.14 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 32028504 33.78 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36024037 33.94 ns/op 0 B/op 0 allocs/op ```
ijsong
added a commit
that referenced
this pull request
Jul 28, 2023
This change makes mrpb.StorageNodeUncommitReport is reused when it is replaced with a new one. Previous PR, #446, introduced a pool for mrpb.StorageNodeUncommitReport. However, the old StorageNodeUncommitReport stored to reportContext was not reused, and this PR improves it. To release the StorageNodeUncommitReport registered to reportContext, some methods of reportContext are changed: - (*reportContext).getReport returns a value of StorageNodeUncommitReport rather than a pointer. It makes releasing the StorageNodeUncommitReport happen at any time. - add (*reportContext).swapReport to set a new StorageNodeUncommitReport and get the old one simultaneously. - getReport and swapReport return the boolean result to indicate whether the result report is zero value. BenchmarkSwapReport evaluates the performance improvements of this change. - getAndSavePtr: Previous implementation. - getAndSave: Call getReport and saveReport. - swap: Call swapReport. ``` BenchmarkSwapReport/getAndSavePtr-16 5947423 207.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6220454 198.7 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6133489 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6131910 199.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 5592459 201.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6180243 200.1 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6172357 199.2 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6211966 198.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6095289 198.3 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6043491 197.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6143407 200.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5938540 203.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6162986 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6172598 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6105918 197.0 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6077460 196.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5953898 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6131799 199.9 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6139296 200.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6115653 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/swap-16 35501544 33.83 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35982945 34.05 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36518960 34.21 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34620783 34.52 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35967453 33.88 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36135454 34.17 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34492548 35.16 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35581888 34.14 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 32028504 33.78 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36024037 33.94 ns/op 0 B/op 0 allocs/op ```
ijsong
added a commit
that referenced
this pull request
Jul 28, 2023
This change makes mrpb.StorageNodeUncommitReport is reused when it is replaced with a new one. Previous PR, #446, introduced a pool for mrpb.StorageNodeUncommitReport. However, the old StorageNodeUncommitReport stored to reportContext was not reused, and this PR improves it. To release the StorageNodeUncommitReport registered to reportContext, some methods of reportContext are changed: - (*reportContext).getReport returns a value of StorageNodeUncommitReport rather than a pointer. It makes releasing the StorageNodeUncommitReport happen at any time. - add (*reportContext).swapReport to set a new StorageNodeUncommitReport and get the old one simultaneously. - getReport and swapReport return the boolean result to indicate whether the result report is zero value. BenchmarkSwapReport evaluates the performance improvements of this change. - getAndSavePtr: Previous implementation. - getAndSave: Call getReport and saveReport. - swap: Call swapReport. ``` BenchmarkSwapReport/getAndSavePtr-16 5947423 207.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6220454 198.7 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6133489 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6131910 199.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 5592459 201.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6180243 200.1 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6172357 199.2 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6211966 198.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6095289 198.3 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6043491 197.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6143407 200.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5938540 203.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6162986 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6172598 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6105918 197.0 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6077460 196.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5953898 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6131799 199.9 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6139296 200.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6115653 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/swap-16 35501544 33.83 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35982945 34.05 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36518960 34.21 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34620783 34.52 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35967453 33.88 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36135454 34.17 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34492548 35.16 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35581888 34.14 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 32028504 33.78 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36024037 33.94 ns/op 0 B/op 0 allocs/op ```
ijsong
added a commit
that referenced
this pull request
Jul 28, 2023
…537) ### What this PR does This change makes mrpb.StorageNodeUncommitReport is reused when it is replaced with a new one. Previous PR, #446, introduced a pool for mrpb.StorageNodeUncommitReport. However, the old StorageNodeUncommitReport stored to reportContext was not reused, and this PR improves it. To release the StorageNodeUncommitReport registered to reportContext, some methods of reportContext are changed: - (*reportContext).getReport returns a value of StorageNodeUncommitReport rather than a pointer. It makes releasing the StorageNodeUncommitReport happen at any time. - add (*reportContext).swapReport to set a new StorageNodeUncommitReport and get the old one simultaneously. - getReport and swapReport return the boolean result to indicate whether the result report is zero value. BenchmarkSwapReport evaluates the performance improvements of this change. - getAndSavePtr: Previous implementation. - getAndSave: Call getReport and saveReport. - swap: Call swapReport. ``` BenchmarkSwapReport/getAndSavePtr-16 5947423 207.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6220454 198.7 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6133489 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6131910 199.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 5592459 201.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6180243 200.1 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6172357 199.2 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6211966 198.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6095289 198.3 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSavePtr-16 6043491 197.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6143407 200.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5938540 203.6 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6162986 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6172598 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6105918 197.0 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6077460 196.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 5953898 197.8 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6131799 199.9 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6139296 200.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/getAndSave-16 6115653 198.5 ns/op 32 B/op 1 allocs/op BenchmarkSwapReport/swap-16 35501544 33.83 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35982945 34.05 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36518960 34.21 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34620783 34.52 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35967453 33.88 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36135454 34.17 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 34492548 35.16 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 35581888 34.14 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 32028504 33.78 ns/op 0 B/op 0 allocs/op BenchmarkSwapReport/swap-16 36024037 33.94 ns/op 0 B/op 0 allocs/op ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This patch adds an
mrpb.StorageNodeUncommitReportpool, which isproto/mrpb.storageNodeUncommitReportPool. It reduces heap objects allocated repeatedly wheneverprocessing reports received from storage nodes.
Below heap profiling is the previous result.
Now here it is after applying this patch.