Add pre-migration command to migration script by alecps · Pull Request #192 · celo-org/optimism

alecps · 2024-07-19T00:09:38Z

Addresses #171

Adds pre-migration subcommand where the ancients migration is run in parallel with the rsync command that copies all other data besides ancients over the the newDB. No transformations are applied to non-ancient blocks or state during the pre-migration. Motivation is to shorten the time needed to run the full migration and minimize downtime.
Adds timers throughout the code to measure latency of different sections
Testing locally on alfajores data, this yielded a ~58% speedup on the rsync command in the final migration. This reduced the length of the final migration by ~18s on alfajores during local testing. It's likely the speedup will be more apparent on mainnet where the db is larger.
Adds--delete flag to the rsync command so that it can be re-run and only apply changes to oldDB
Also moves scan for extra ancient blocks to the pre-migration step. This was adding ~20s to the script before that will now happen in parallel to the other pre-migration steps
removes flags for clearing the db

Tested

op-geth and op-node start up successfully on the migrated alfajores db

… onlyAncients flag

op-chain-ops/cmd/celo-migrate/README.md

piersy

Before going ahead with this change I'd like to know how well it works.

It should be fairly simple to test on a local machine by taking an alfajores datadir, pre-migrating it, then running celo in full sync for a while, an hour or two, and then performing a full migration from scratch from that point and comparing the times taken there with a migration into the pre-migrated datadir. Specifically taking account of the time taken to perform the non ancients migration.

codecov-commenter · 2024-07-24T22:28:59Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.32%. Comparing base (e1fa92a) to head (ec9a6c3).
Report is 30 commits behind head on celo7.

Additional details and impacted files

@@           Coverage Diff           @@
##            celo7     #192   +/-   ##
=======================================
  Coverage   61.32%   61.32%           
=======================================
  Files          20       20           
  Lines        1753     1753           
  Branches       71       71           
=======================================
  Hits         1075     1075           
  Misses        646      646           
  Partials       32       32

Flag	Coverage Δ
cannon-go-tests	`81.03% <ø> (ø)`
chain-mon-tests	`27.14% <ø> (ø)`
sdk-tests	`16.44% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

alecps · 2024-07-25T04:05:12Z

@piersy I've added a fair amount of cleanup to this PR, including some code to log duration of different parts of the script to identify where slowdowns are.

I was able to measure the speedup of pre-running the rsync command to be ~58% according to rsync's logs. This led to a speedup of ~18s for migrating alfajores on my machine versus running a full migration where only the ancient blocks are pre-migrated.

Output after running rsync with pre-migration

sent 10717398995 bytes received 23874 bytes 220977791.11 bytes/sec
total size is 16985666076 speedup is 1.58
INFO [07-24|19:24:58.243] TIMER process=copyDbExceptAncients duration=48.593819875s

Output after running rsync without pre-migration

sent 16987957677 bytes received 75244 bytes 259359281.24 bytes/sec
total size is 16985666076 speedup is 1.00
INFO [07-24|23:30:17.836] TIMER process=copyDbExceptAncients duration=1m6.435774166s

The way the code is factored now, the non-ancients step isn't effected by the rsync command. So the duration for this step was ~24s for both runs. The speedup we're concerned with is in the copyDbExceptAncients step.

The overall speedup from running the entire pre-migration step was ~8m7s on alfajores data with local setup

I still need to test that op-geth and op-node can run on the migrated data after these changes

alecps · 2024-07-25T04:22:31Z

@carterqw2 I ended up removing the --update flag because after closer inspection this seems to skip any files that might have been modified more recently on the newDB. Since we don't want to preserve any modifications to the newDB that happened after the last pre-migration, I think we might be better off overwriting those files if there are any. Let me know if I'm missing something.

In practice, I didn't see any difference in this behavior when testing with / without this flag, which indicates there were no files more recently updated in the newDB in my case, which is as expected.

piersy · 2024-07-25T15:59:30Z

@piersy I've added a fair amount of cleanup to this PR, including some code to log duration of different parts of the script to identify where slowdowns are.

I was able to measure the speedup of pre-running the rsync command to be ~58% according to rsync's logs. This led to a speedup of ~18s for migrating alfajores on my machine versus running a full migration where only the ancient blocks are pre-migrated.

Output after running rsync with pre-migration

sent 10717398995 bytes received 23874 bytes 220977791.11 bytes/sec
total size is 16985666076 speedup is 1.58
INFO [07-24|19:24:58.243] TIMER process=copyDbExceptAncients duration=48.593819875s

Output after running rsync without pre-migration

sent 16987957677 bytes received 75244 bytes 259359281.24 bytes/sec
total size is 16985666076 speedup is 1.00
INFO [07-24|23:30:17.836] TIMER process=copyDbExceptAncients duration=1m6.435774166s

The way the code is factored now, the non-ancients step isn't effected by the rsync command. So the duration for this step was ~24s for both runs. The speedup we're concerned with is in the copyDbExceptAncients step.

The overall speedup from running the entire pre-migration step was ~8m7s on alfajores data with local setup

I still need to test that op-geth and op-node can run on the migrated data after these changes

Hey @alecps for the times listed there, for how long did you run the node between the migrations?

op-chain-ops/cmd/celo-migrate/db.go

op-chain-ops/cmd/celo-migrate/non-ancients.go

op-chain-ops/cmd/celo-migrate/main.go

piersy · 2024-07-25T17:28:06Z

Looking good, a few comments:

I noticed that the command-line help seems to print some log statements. Those should probably be removed.

INFO [07-25|18:27:38.253] Beginning Cel2 Migration
NAME:
   celo-migrate pre - Perform a  pre-migration of ancient blocks and copy over all other data without transforming it. This should be run a day before the full migration command is run to minimize downtime.

USAGE:
   celo-migrate pre [command options] [arguments...]

OPTIONS:

          --old-db value
                Path to the old Celo chaindata dir, can be found at '<datadir>/celo/chaindata'

          --new-db value
                Path to write migrated Celo chaindata, note the new node implementation expects
                to find this chaindata at the following path '<datadir>/geth/chaindata

          --batch-size value                  (default: 50000)
                Batch size to use for block migration, larger batch sizes can speed up migration
                but require more memory. If increasing the batch size consider also increasing
                the memory-limit

          --buffer-size value                 (default: 0)
                Buffer size to use for ancient block migration channels. Defaults to 0. Included
                to facilitate testing for performance improvements.

          --memory-limit value                (default: 7500)
                Memory limit in MiB, should be set lower than the available amount of memory in
INFO [07-25|18:27:38.253] Finished migration successfully!
                your system to prevent out of memory errors

          --clear-non-ancients                (default: false)
                Use this to reset all data except ancients. This flag should be used if a full
                migration has already been performed on the new db.

          --measure-time                      (default: false)
                Use this to log how long each section of the script takes to run

          --help, -h                          (default: false)
                show help
(END)

alecps · 2024-07-25T23:24:54Z

@piersy I didn't run the node, I just pulled down the latest snapshot and had the old snapshot from probably ~3 weeks ago

op-chain-ops/cmd/celo-migrate/db.go

op-chain-ops/cmd/celo-migrate/main.go

piersy

One small point about naming, apart from that all good!

* add pre-migration command, rsync and ancients run in parallel, remove onlyAncients flag * remove block and state migration sub-commands * make non ancient migration its own step, add flag to measure time * add more granular timers * open db without freezer in state migration, remove clearAll * fix error * remove update flag from rsync command, add rsync comments * delete commented out versions of checkForPrevFullMigration * remove aliases * remove clearNonAncients flag * remove measureTime flag, always log time measurements * remove logging from help text * remove db reset * move scan for extra ancients into pre-migration * update README * rename extraAncientNumHashes to strayAncientBlocks

This works by loading the database of a celo node. It then removes all existing blocks and generates a new genesis block including the existing state tree. Migrate to urfave/cli/v2 Update op-chain-ops/cmd/op-migrate/main.go Co-authored-by: Karl Bartel <karl@karl.berlin> Combine Cel2 migration scripts (#148) * Initial script to play with celo DB history migration * Can Read All the headers Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com> * Adds new command to migrate ancients db * Adds comment * Adds extension methods for transformation * Implements Transform CeloBody * Adds impl that runs steps in a concurrent pipeline * Adds transformHead, verify hashing works cleanup * add migration for non-frozen blocks * copy over entire db and modify in place, works with op-geth at piersy/minimal-data-migration * remove unecessary copying, cleanup code * close and reopen DBs * migrate newdb in place * saving progress Co-authored-by: Mariano Cortesi <mcortesi@gmail.com> * Refactor code to improve database migration process * better logging * refactor: inline parMigrateAncientRange * Remove frozen blocks from nonAncient DB * check hash matches on nonAncients migration * clean up branch Removes unused code, move code for better separation of concerns. * decode into new types * fix transformHeader * make old freezer not readonly so that .meta files are created * add configurable memory limit * add comment about memory * Added celo-dbmigrate Makefile target * Added dockerfile for celo-dbmigrate and celo-migrate tools * Workflow for running cel2-migration-tool * Update cel2-migration-tool image registry * update op-geth to point to https://github.com/celo-org/op-geth/commits/piersy/for-use-with-migrated-celo-datadir-use-gas-limit-differentiation-rebased-celo6/ * add celo6 logging * rename scripts to celo-migrate-state and celo-migrate-blocks * first pass at combining scripts * saving progress on testing * fix lint error, use %w to fmt errors * add updated state migration input files to testdata * add ability to run block and state migration seperately or together * add option for migrating only frozen blocks * remove old scripts * minor logging improvements in block migrations * invert clearNonAncients flag logic --> keepNonAncients, make dry-run flag only apply to state migration * adds README, improves logging * fix lint err * Fix Makefile and Dockerfile * move createNewDbIfNotExists * rename keep-non-ancients * update TODO to add more context and state changes * Remove channel buffers from ancients migration Co-authored-by: Valentin Rodygin <carterqw@gmail.com> * bump default batch size to 100000 * add back extended usage string * add info on state migration to README * remove --state-dry-run flag * update default batch size to 50k * Adding building for op images * Setting our values for image registry and repository * update README * fix logging when newAncients > oldAncients * fix return value when skipping ancients * skip transforming block bodies that have already been transformed * misc. fixes to get re-runs with --keep-non-ancients working * adds TODO * addresses cosmetic feedback * add flag for specifying a buffer * Show progress on rsync * Update to latest op-geth * state-migration: Refactor subtask * state-migration: Use EIP1559 settings from deploy config Fixes #135 * state-migration: Enable Fjord hardfork during migration Fixes #160 * state-migration: Deterministicly set migration block timestamp Fixes #157 Sets the timestamp to be 5s after the last block. * state-migration: Set WithdrawalsHash in Cel2 migration block * fixup! Fix Makefile and Dockerfile * add note to README about using snapshots for pre-migration * Set blob gas header fields for transition block These are now required to be set since cancun was activated. * Use InitialBaseFee for pre-gingerbread transitionb * Fix warnings about capitalized error strings * Output chain config as marshalled JSON * state-migration: Handle accounts with existing balance Fixes #158 * remove allocs file, add instructions for how to generate allocs file to README, update TODOs --------- Co-authored-by: Mariano Cortesi <mariano@clabs.co> Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com> Co-authored-by: Mariano Cortesi <mcortesi@gmail.com> Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com> Co-authored-by: Paul Lange <palango@gmx.de> Co-authored-by: Valentin Rodygin <carterqw@gmail.com> Co-authored-by: Piers Powlesland <pierspowlesland@gmail.com> Set balance of `CeloDistributionSchedule` contract (#162) * state-migration: Initialize CeloDistributionSchedule Fixes #155 * state-migration: Don't fail when distribution schedule update errors * Review comments state-migration: Set ParentBeaconRoot (#176) This allows header validation to pass during snap sync state-migration: Set address of distribution schedule (#177) state-migration: Read total supply directly from state (#182) * state-migration: Read totalSupply directly from storage * Added trigger for updated dependencies * Removen token bindings --------- Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com> Fix l2 block older than l1 origin error (#184) (#187) * Revert to using time.Now() for migration block Instead of simply adding 5 to the parent block time. We really do need a deterministic time for the migration block so that all parties that run the migration arrive at the same migration block but the problem is that op-geth requires that the L2 migration block (aka l2 origin) occurs after the l1 origin (I guess the point where you deploy the bridge contracts to the l1). When we migrate a partially synced datadir the block before the transition block will be very old, up to 4 years old! So of course it occurs before the l1 origin. So a fix just to get things working is to use time.Now(), but probably we should make this a configurable parameter. * add flag to specify timestamp * Update op-chain-ops/cmd/celo-migrate/main.go --------- Co-authored-by: piersy <pierspowlesland@gmail.com> Migration script fixes (#179) * Fixed migration for datadirs without ancients The script was assuming that ancients would have been migrated and was considering the numAncients-1 to be the next block to migrate but when numAncients is zero that's a problem. Also remved logic for picking up where db migration left of for the level db since it was complicating the logic and that process takes a few seconds, which is nothing compared with the minutes taken to migrate the ancients. * Ensure that we set gas limit if migrating at pre-gingerbread point Fix migration script gap in migrated blocks (#189) * Fix migration script gap in migrated blocks The range of ancient blocks to remove from the non ancients database was off by one and resulted in a gap between ancients and non ancients. Also corrected some log statements that were off by one. Add pre-migration command to migration script (#192) * add pre-migration command, rsync and ancients run in parallel, remove onlyAncients flag * remove block and state migration sub-commands * make non ancient migration its own step, add flag to measure time * add more granular timers * open db without freezer in state migration, remove clearAll * fix error * remove update flag from rsync command, add rsync comments * delete commented out versions of checkForPrevFullMigration * remove aliases * remove clearNonAncients flag * remove measureTime flag, always log time measurements * remove logging from help text * remove db reset * move scan for extra ancients into pre-migration * update README * rename extraAncientNumHashes to strayAncientBlocks state-migration: Fail if account would be overwritten (#202) * state-migration: Fail if account would be overwritten * Review changes * Review changes 2 * Fail in unclear state * more changes * Use whitelist to decide if nonce and state are overwritten Cosmetic changes to the migration script - Use more lists for added readability - Capitalize Alfajores and Celo - Reorder scripting instructions to fit the actual order or operations - Use GitHub callouts migration: Add tests (#217) * migration: Add tests for state migration * migration: Fix issues shown by tests * migration: pass allowlist into state migration Allows for easier testing * migration: Add test with allowlist * Correct overwrite counter * Use in memory DB migration: Add working allowlist for Alfajores (#220) * migration: Simplify tests * migration: Add working allowlist for Alfajores Adapt migration code to changes in StateDB StateDB.CreateAccount used to copy existing balance, now it does not any more. migration: Set fields correctly for migration block (#212) migration: Enable Granite (#226) Write genesis file in state migration (#219) * squash of #167 * add writeGenesis * open old freezer in readonly mode, fix locking error * remove devAlloc * Revert "open old freezer in readonly mode, fix locking error" This reverts commit e3fddea. * fix locking error * fix lint error, check errors, add comment * remove comment * filter extra genesis fields * fix issue with genesis extra data * update testdata --------- Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com> migration: Overwrite create2deployer code (#233) migration: Allow 'createx' preinstall (#238) The code already exists on Alfajores and matches the one that would be deployed, therefore we just allow this address. add migration-block-number flag (#245) * add migration-block-number flag * address feedback * move migration-block-number flag out of state migration options Fixes for re-running migration script on same destination db (#246) * add reset flag * add --checksum to rsync options

* add pre-migration command, rsync and ancients run in parallel, remove onlyAncients flag * remove block and state migration sub-commands * make non ancient migration its own step, add flag to measure time * add more granular timers * open db without freezer in state migration, remove clearAll * fix error * remove update flag from rsync command, add rsync comments * delete commented out versions of checkForPrevFullMigration * remove aliases * remove clearNonAncients flag * remove measureTime flag, always log time measurements * remove logging from help text * remove db reset * move scan for extra ancients into pre-migration * update README * rename extraAncientNumHashes to strayAncientBlocks

add pre-migration command, rsync and ancients run in parallel, remove…

301ef7d

… onlyAncients flag

alecps requested review from carterqw2, palango and piersy July 19, 2024 00:09

carterqw2 reviewed Jul 19, 2024

View reviewed changes

op-chain-ops/cmd/celo-migrate/README.md Outdated Show resolved Hide resolved

piersy reviewed Jul 19, 2024

View reviewed changes

alecps added 3 commits July 23, 2024 12:28

remove block and state migration sub-commands

96a6657

make non ancient migration its own step, add flag to measure time

f8521d3

add more granular timers

b2d7918

alecps added 3 commits July 24, 2024 18:37

open db without freezer in state migration, remove clearAll

eb580e4

fix error

edb958e

remove update flag from rsync command, add rsync comments

9e8b5e5

alecps requested review from carterqw2 and piersy July 25, 2024 04:25