Skip to content

chore(rpc): optimize genesis file chunking#4349

Merged
alesforz merged 46 commits intomainfrom
alesforz/opt-gen-chunking
Oct 29, 2024
Merged

chore(rpc): optimize genesis file chunking#4349
alesforz merged 46 commits intomainfrom
alesforz/opt-gen-chunking

Conversation

@alesforz
Copy link
Collaborator

@alesforz alesforz commented Oct 28, 2024

Closes #1289.

Changes

Environment type

The Environment type will:

  • store the genesis file's path on disk
  • store a map[int]string where the keys are the chunks' IDs, and the values are the chunk's path on disk.
  • not store a pointer to a types.GenesisDoc anymore
  • not store a slice of genesis file chunks anymore. Instead the method InitGenesisChunks will create chunks and store them on disk, 1 file per chunk.

Node type

The Node type will:

  • not store a pointer to a GenesisDoc anymore.
  • pass the genesis file's path on disk to a new Environment object when creating it in the ConfigureRPC() method.

RPC /genesis_chunked

This endpoint will implement the chunking strategy described here.


PR checklist

  • Tests written/updated
  • Changelog entry added in .changelog (we use unclog to manage our changelog)
  • Updated relevant documentation (docs/ or spec/) and code comments

Alessandro added 27 commits October 28, 2024 14:46
Unexported field because it's only used by the RPC APIs.
Started refactoring it to smaller functions for easier testing as well.
Neither a pointer to a `GenesisDoc`, nor a slice of chunks.
- renamed `deleteGenesisChunks` to `Cleanup`
- better docs for test func
- fixed bug in `TestInitGenesisChunks`
Added parameter defining the chunk size so that the function doesn't create a
new buffer at every iteration of the loop.
These functions optionally accept a list of callbacks to execute at shutdown.
We use this feature to delete the genesis file's chunks from disk when the
server shuts down by passing a function that handles the deletion as a
parameter.
However, we could pass any other function we wish to execute at this stage.
The function now first checks for and deletes any existing chunks directory.
While a Node's graceful shutdown procedure deletes the genesis file chunks and
their enclosing directory, if the Node (or its RPC HTTP server) crashes, it's
possible that the genesis file chunks remain on disk and are present upon restart.

This is not necessarily a problem, but if the genesis file changes for any reason,
its chunks will also differ. Therefore, to be safe, we re-create the chunks from
scratch at startup.
@alesforz alesforz self-assigned this Oct 28, 2024
@alesforz alesforz added backlog A prioritized task in the team's backlog rpc labels Oct 28, 2024
@alesforz alesforz requested a review from a team as a code owner October 28, 2024 16:26
@alesforz alesforz requested a review from a team October 28, 2024 16:26
Alessandro and others added 3 commits October 28, 2024 17:45
Somewhat confusingly, test were using `fGenesis` to indicate the genesis file,
but uses `gFile`.
This changes all test variables to use the `gFile` prefix.
Copy link
Collaborator

@melekes melekes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alesforz ❤️

main q: #4349 (comment)

minor q: #4349 (comment)

@alesforz
Copy link
Collaborator Author

See here for a visualization of the changes in memory usage after applying these changes.

Copy link

@cason cason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some generic comments, looks like a good feature, but this PR includes some stuff not directly associated to it.

mergify bot and others added 2 commits October 29, 2024 13:13
@alesforz
Copy link
Collaborator Author

Left some generic comments, looks like a good feature, but this PR includes some stuff not directly associated to it.

🤔 What parts do you think aren't associated with the feature?

Copy link

@cason cason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change was more complex than I initially thought, approved.

mergify bot and others added 5 commits October 29, 2024 13:59
They now call `ServeWithShutdown` and `ServeTLSWithShutdown` with a nil
`shutdownTasks` list respectively.
They were calling `ServeWithShutdown` and `ServeTLSWithShutdown` with `nil` as
the `shutdownTasks` parameter. This is incorrect because it creates a list
containing one `nil` element. Consequently, when `ServeWithShutdown` and
`ServeTLSWithShutdown` will  execute the `for` loop over `shutdownTasks`, they will
encounter a `nil` pointer dereference, causing them to panic.
@alesforz alesforz added this pull request to the merge queue Oct 29, 2024
Merged via the queue into main with commit 48db7ac Oct 29, 2024
@alesforz alesforz deleted the alesforz/opt-gen-chunking branch October 29, 2024 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backlog A prioritized task in the team's backlog P:storage-optimization Priority: Give operators greater control over storage and storage optimization P:tech-debt Priority: Technical debt that needs to be paid off to enable us to move faster, reliably rpc

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

Optimize genesis file chunking

4 participants