container: reduce the number of fetched iterator items#3558
Merged
cthulhu-rider merged 1 commit intomasterfrom Sep 4, 2025
Merged
container: reduce the number of fetched iterator items#3558cthulhu-rider merged 1 commit intomasterfrom
cthulhu-rider merged 1 commit intomasterfrom
Conversation
An attempt to list 2K+ containers ended up this way:
error handler/util.go:34 call method {"status": 500, "request_id": "0cedec2b-935d-4575-a0c2-94ed4650c978", "method": "ListBuckets", "bucket": "", "object": "", "description": "something went wrong", "error": "list user containers via connection pool: status: code = 1024 message = json error: too big: size"}
We're trying to get 2K items from the iterator in a single call, each item is
base64 of CID, so it's 42 bytes, but then it's wrapped into VM type/value
struct like this:
{"value":"","type":"ByteString"}
which adds another 32 bytes. For 2K elements this yields 144K JSON (not
including other data which is also present) which NeoGo refuses to return
given that the maximum size is 128K (nspcc-dev/neo-go#3185).
Fetching in smaller chunks should fix the problem (for other iterator-based
methods as well).
Signed-off-by: Roman Khimov <roman@nspcc.ru>
carpawell
approved these changes
Sep 4, 2025
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3558 +/- ##
=======================================
Coverage 25.65% 25.66%
=======================================
Files 661 661
Lines 49451 49451
=======================================
+ Hits 12688 12691 +3
+ Misses 35755 35752 -3
Partials 1008 1008 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
cthulhu-rider
approved these changes
Sep 4, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An attempt to list 2K+ containers ended up this way:
We're trying to get 2K items from the iterator in a single call, each item is base64 of CID, so it's 42 bytes, but then it's wrapped into VM type/value struct like this:
which adds another 32 bytes. For 2K elements this yields 144K JSON (not including other data which is also present) which NeoGo refuses to return given that the maximum size is 128K (nspcc-dev/neo-go#3185). Fetching in smaller chunks should fix the problem (for other iterator-based methods as well).