Description
We presently include the uncompressed genesis state for supported networks in our binary. We have several of these files:
[3.3M] common/eth2_network_config/built_in_network_configs/chiado/genesis.ssz
[3.1M] common/eth2_network_config/built_in_network_configs/gnosis/genesis.ssz
[5.2M] common/eth2_network_config/built_in_network_configs/mainnet/genesis.ssz
[ 28M] common/eth2_network_config/built_in_network_configs/prater/genesis.ssz
[ 15M] common/eth2_network_config/built_in_network_configs/ropsten/genesis.ssz
[2.8M] common/eth2_network_config/built_in_network_configs/sepolia/genesis.ssz
The total of these files is 57.4M, which goes straight to our hips binary size (presently ~110M). I suspect we could significantly reduce the size of the binaries by storing compressed genesis.ssz bytes in the binary and then decompressing on-demand (i.e. at startup).
I propose that we use snappy compression, since it's used by the P2P layer and therefore available in the binary.
Before committing to this change, I would be keen to know the time it takes to decompress the state at startup. Perhaps getting numbers for mainnet and Prater would be good. We want to be careful not to slow-down BN/VC startup.
Details
The method for including the genesis.ssz can be a bit tricky to understand because it's written in macros. I think this should be fairly straight-forward once you get your head across it. I've included some links below to give a lay of the land.
The bytes are added to the binary here:
|
include_bytes!(concat!( |
|
$base_dir, |
|
"/", |
|
$this_crate::predefined_networks_dir!(), |
|
"/", |
|
$config_dir, |
|
"/", |
|
$filename |
|
)) |
The genesis.ssz file used by the include_bytes! macro is generated here (this is where we'd want to do the snappy compression):
|
// Extract genesis state from genesis.ssz.zip |
|
let archive_path = network.genesis_state_archive(); |
|
let archive_file = File::open(&archive_path) |
|
.map_err(|e| format!("Failed to open archive file {:?}: {:?}", archive_path, e))?; |
|
|
|
let mut archive = |
|
ZipArchive::new(archive_file).map_err(|e| format!("Error with zip file: {}", e))?; |
|
|
|
let mut file = archive.by_name(GENESIS_FILE_NAME).map_err(|e| { |
|
format!( |
|
"Error retrieving file {} inside zip: {}", |
|
GENESIS_FILE_NAME, e |
|
) |
|
})?; |
|
let mut outfile = File::create(&genesis_ssz_path) |
|
.map_err(|e| format!("Error while creating file {:?}: {}", genesis_ssz_path, e))?; |
|
io::copy(&mut file, &mut outfile) |
|
.map_err(|e| format!("Error writing file {:?}: {}", genesis_ssz_path, e))?; |
The application accesses the included bytes here (this is where we'd want to do the snappy decompression) (there might also be other places it is accessed):
|
/// Attempts to deserialize `self.beacon_state`, returning an error if it's missing or invalid. |
|
pub fn beacon_state<E: EthSpec>(&self) -> Result<BeaconState<E>, String> { |
|
let spec = self.chain_spec::<E>()?; |
|
let genesis_state_bytes = self |
|
.genesis_state_bytes |
|
.as_ref() |
|
.ok_or("Genesis state is unknown")?; |
|
|
|
BeaconState::from_ssz_bytes(genesis_state_bytes, &spec) |
|
.map_err(|e| format!("Genesis state SSZ bytes are invalid: {:?}", e)) |
|
} |
Description
We presently include the uncompressed genesis state for supported networks in our binary. We have several of these files:
The total of these files is 57.4M, which goes straight to our
hipsbinary size (presently ~110M). I suspect we could significantly reduce the size of the binaries by storing compressedgenesis.sszbytes in the binary and then decompressing on-demand (i.e. at startup).I propose that we use snappy compression, since it's used by the P2P layer and therefore available in the binary.
Before committing to this change, I would be keen to know the time it takes to decompress the state at startup. Perhaps getting numbers for mainnet and Prater would be good. We want to be careful not to slow-down BN/VC startup.
Details
The method for including the
genesis.sszcan be a bit tricky to understand because it's written in macros. I think this should be fairly straight-forward once you get your head across it. I've included some links below to give a lay of the land.The bytes are added to the binary here:
lighthouse/common/eth2_config/src/lib.rs
Lines 129 to 137 in dfcb336
The
genesis.sszfile used by theinclude_bytes!macro is generated here (this is where we'd want to do the snappy compression):lighthouse/common/eth2_network_config/build.rs
Lines 30 to 47 in dfcb336
The application accesses the included bytes here (this is where we'd want to do the snappy decompression) (there might also be other places it is accessed):
lighthouse/common/eth2_network_config/src/lib.rs
Lines 98 to 108 in dfcb336