chore(rpc): optimize genesis file chunking#4349
Merged
Conversation
added 27 commits
October 28, 2024 14:46
Unexported field because it's only used by the RPC APIs.
Started refactoring it to smaller functions for easier testing as well.
…ing. Added unit tests as well.
Neither a pointer to a `GenesisDoc`, nor a slice of chunks.
Added parameter defining the chunk size so that the function doesn't create a new buffer at every iteration of the loop.
These functions optionally accept a list of callbacks to execute at shutdown. We use this feature to delete the genesis file's chunks from disk when the server shuts down by passing a function that handles the deletion as a parameter. However, we could pass any other function we wish to execute at this stage.
The function now first checks for and deletes any existing chunks directory. While a Node's graceful shutdown procedure deletes the genesis file chunks and their enclosing directory, if the Node (or its RPC HTTP server) crashes, it's possible that the genesis file chunks remain on disk and are present upon restart. This is not necessarily a problem, but if the genesis file changes for any reason, its chunks will also differ. Therefore, to be safe, we re-create the chunks from scratch at startup.
Somewhat confusingly, test were using `fGenesis` to indicate the genesis file, but uses `gFile`. This changes all test variables to use the `gFile` prefix.
melekes
approved these changes
Oct 29, 2024
From 755 to 700, i.e., only the owner can read, write, and execute.
Collaborator
Author
|
See here for a visualization of the changes in memory usage after applying these changes. |
cason
reviewed
Oct 29, 2024
cason
left a comment
There was a problem hiding this comment.
Left some generic comments, looks like a good feature, but this PR includes some stuff not directly associated to it.
.changelog/unreleased/improvements/4235-4234-environment-stores-one-genesis.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Daniel <daniel.cason@informal.systems>
Collaborator
Author
🤔 What parts do you think aren't associated with the feature? |
cason
approved these changes
Oct 29, 2024
They now call `ServeWithShutdown` and `ServeTLSWithShutdown` with a nil `shutdownTasks` list respectively.
They were calling `ServeWithShutdown` and `ServeTLSWithShutdown` with `nil` as the `shutdownTasks` parameter. This is incorrect because it creates a list containing one `nil` element. Consequently, when `ServeWithShutdown` and `ServeTLSWithShutdown` will execute the `for` loop over `shutdownTasks`, they will encounter a `nil` pointer dereference, causing them to panic.
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1289.
Changes
EnvironmenttypeThe
Environmenttype will:map[int]stringwhere the keys are the chunks' IDs, and the values are the chunk's path on disk.types.GenesisDocanymoreInitGenesisChunkswill create chunks and store them on disk, 1 file per chunk.NodetypeThe
Nodetype will:GenesisDocanymore.Environmentobject when creating it in theConfigureRPC()method.RPC
/genesis_chunkedThis endpoint will implement the chunking strategy described here.
PR checklist
.changelog(we use unclog to manage our changelog)docs/orspec/) and code comments