Skip to content

Optimize genesis file chunking #1289

@andynog

Description

@andynog

As part of #1037, we need to optimize how the genesis file is chunked.

we need to implement logic to:

  • Determine the number of chunks (based on the chunk size and file size). This logic already exists, might need to adapt for optimal chunk size.
  • Store the chunks in the file system (TBD in the tmp or data folder inside home)
  • Return chunks loaded from the file system (not from storage) for the genesis_chunked endpoint.
  • [OPTIONAL] Validate chunks (re-create the file from chunks) and hash it to ensure it matches hash in storage or hash of genesis in the file system (config folder). Maybe this could be validated when the node starts to ensure nothing has been tampered with.

DOD

  • Do not store chunked genesis files in memory.
  • Store the chunks on storage or file system and fetch them on demand.
  • Rethink the default size of the genesis chunk and use an appropriate value.

This completes the work started in Phase II of #1037, thus finalizing the implementation of the chunking strategy discussed here.

Metadata

Metadata

Assignees

Labels

P:storage-optimizationPriority: Give operators greater control over storage and storage optimizationP:tech-debtPriority: Technical debt that needs to be paid off to enable us to move faster, reliablybacklogA prioritized task in the team's backlogrpc

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions