Fix query node scalability issues#4366
Merged
mnaamani merged 2 commits intoJoystream:carthagefrom Oct 18, 2022
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
mnaamani
approved these changes
Oct 18, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4360
Partially relies on Joystream/hydra#510, which introduces additional optimalization and prevents unnecessary event entity size bloat.
Processor parameters
By setting
BATCH_SIZE=100andQUEUE_FACTOR=1we limit the number of events fetched from the indexer to100in a single query. This is because in worst case scenario an event can be ~5 MB in size (max block size), which gives 500 MB of data. Serializing data to a string larger than 512 MB can already cause issues on 32-bit platforms.With
QUEUE_MAX_CAP_FACTOR=4we allow 4x more events to be stored in memory, so ~2 GB in worst case scenario.Impact of this PR on indexing/processing speed
Empty blocks
~19 blocks / second=>~19 blocks / second(no change)~1000 blocks / second => ~1000 blocks / second(no change)batchSize=100 members migration (4400 members over 176 blocks)
26s=>14s(-12s / -46%)27s=>30s(+3s / +11%)batchSize=1000 members migration (4400 members over 18 blocks)
135s=>6s(-129s / -95%)ERROR=>68s┆Issue is synchronized with this Asana task by Unito