Skip to content

Fix l2 block older than l1 origin error#184

Merged
alecps merged 3 commits intocelo6from
piersy/l2-block-older-than-l1-origin
Jun 27, 2024
Merged

Fix l2 block older than l1 origin error#184
alecps merged 3 commits intocelo6from
piersy/l2-block-older-than-l1-origin

Conversation

@piersy
Copy link
Copy Markdown

@piersy piersy commented Jun 27, 2024

We were encountering the following error when running op-node with a migrated datadir:

ERROR[06-25|16:53:24.779] Derivation pipeline not ready, failed to reset engine err="reset: cannot reset block derivation to start at L2 block 0xdf1dfd6566270260a33f4a33406358e93b8402d4bb6829bb35d685259cc1c3c2:23912612 with time 1714956104 older than its L1 origin 0xbbed3612407993e67f8ca7a423b181837ae164a531941e78f5ee48e766d39cad:1729797 with time 1718312256, time invariant is broken"

This pr fixes that by using time.Now for the time of the l2 transition block.
We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.
Copy link
Copy Markdown

@alecps alecps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Should we put this behind a flag so that header.Time + 5 is used for the actual migration, but time.Now() can be optionally used instead for development?

@piersy
Copy link
Copy Markdown
Author

piersy commented Jun 27, 2024

Hey @alecps, I was thinking that we might want to use the flag for the actual migration, it would give us flexibility, for example if we defined the celo-blockchain end block but then had some problems deploying the contracts to L1 and ended up with the L1 contract deployment happening after the end block. The thing is you don't want to deploy the l1 contracts too far in advance since op-node will try to "catch up" to ensure that the time between l1 origin and now is filled with the appropriate number of l2 blocks.

@alecps alecps merged commit 0985902 into celo6 Jun 27, 2024
@alecps alecps deleted the piersy/l2-block-older-than-l1-origin branch June 27, 2024 19:24
piersy added a commit that referenced this pull request Jun 28, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: alecps <alec@cLabs.co>
alecps pushed a commit that referenced this pull request Jun 28, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: alecps <alec@cLabs.co>
alecps pushed a commit that referenced this pull request Jun 28, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
jcortejoso pushed a commit that referenced this pull request Jul 1, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Sep 10, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Sep 10, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Sep 10, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Sep 13, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Sep 17, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
palango pushed a commit that referenced this pull request Sep 24, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
palango pushed a commit that referenced this pull request Sep 24, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Oct 12, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Oct 14, 2024
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
alecps pushed a commit that referenced this pull request Oct 15, 2024
This works by loading the database of a celo
node. It then removes all existing blocks and
generates a new genesis block including the
existing state tree.

Migrate to urfave/cli/v2

Update op-chain-ops/cmd/op-migrate/main.go

Co-authored-by: Karl Bartel <karl@karl.berlin>

Combine Cel2 migration scripts (#148)

* Initial script to play with celo DB history migration

* Can Read All the headers

Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com>

* Adds new command to migrate ancients db

* Adds comment

* Adds extension methods for transformation

* Implements Transform CeloBody

* Adds impl that runs steps in a concurrent pipeline

* Adds transformHead, verify hashing works

cleanup

* add migration for non-frozen blocks

* copy over entire db and modify in place, works with op-geth at piersy/minimal-data-migration

* remove unecessary copying, cleanup code

* close and reopen DBs

* migrate newdb in place

* saving progress

Co-authored-by: Mariano Cortesi <mcortesi@gmail.com>

* Refactor code to improve database migration process

* better logging

* refactor: inline parMigrateAncientRange

* Remove frozen blocks from nonAncient DB

* check hash matches on nonAncients migration

* clean up branch

Removes unused code, move code for better separation of concerns.

* decode into new types

* fix transformHeader

* make old freezer not readonly so that .meta files are created

* add configurable memory limit

* add comment about memory

* Added celo-dbmigrate Makefile target

* Added dockerfile for celo-dbmigrate and celo-migrate tools

* Workflow for running cel2-migration-tool

* Update cel2-migration-tool image registry

* update op-geth to point to https://github.com/celo-org/op-geth/commits/piersy/for-use-with-migrated-celo-datadir-use-gas-limit-differentiation-rebased-celo6/

* add celo6 logging

* rename scripts to celo-migrate-state and celo-migrate-blocks

* first pass at combining scripts

* saving progress on testing

* fix lint error, use %w to fmt errors

* add updated state migration input files to testdata

* add ability to run block and state migration seperately or together

* add option for migrating only frozen blocks

* remove old scripts

* minor logging improvements in block migrations

* invert clearNonAncients flag logic --> keepNonAncients, make dry-run flag only apply to state migration

* adds README, improves logging

* fix lint err

* Fix Makefile and Dockerfile

* move createNewDbIfNotExists

* rename keep-non-ancients

* update TODO to add more context and state changes

* Remove channel buffers from ancients migration

Co-authored-by: Valentin Rodygin <carterqw@gmail.com>

* bump default batch size to 100000

* add back extended usage string

* add info on state migration to README

* remove --state-dry-run flag

* update default batch size to 50k

* Adding building for op images

* Setting our values for image registry and repository

* update README

* fix logging when newAncients > oldAncients

* fix return value when skipping ancients

* skip transforming block bodies that have already been transformed

* misc. fixes to get re-runs with --keep-non-ancients working

* adds TODO

* addresses cosmetic feedback

* add flag for specifying a buffer

* Show progress on rsync

* Update to latest op-geth

* state-migration: Refactor subtask

* state-migration: Use EIP1559 settings from deploy config

Fixes #135

* state-migration: Enable Fjord hardfork during migration

Fixes #160

* state-migration: Deterministicly set migration block timestamp

Fixes #157

Sets the timestamp to be 5s after the last block.

* state-migration: Set WithdrawalsHash in Cel2 migration block

* fixup! Fix Makefile and Dockerfile

* add note to README about using snapshots for pre-migration

* Set blob gas header fields for transition block

These are now required to be set since cancun was activated.

* Use InitialBaseFee for pre-gingerbread transitionb

* Fix warnings about capitalized error strings

* Output chain config as marshalled JSON

* state-migration: Handle accounts with existing balance

Fixes #158

* remove allocs file, add instructions for how to generate allocs file to README, update TODOs

---------

Co-authored-by: Mariano Cortesi <mariano@clabs.co>
Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com>
Co-authored-by: Mariano Cortesi <mcortesi@gmail.com>
Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>
Co-authored-by: Paul Lange <palango@gmx.de>
Co-authored-by: Valentin Rodygin <carterqw@gmail.com>
Co-authored-by: Piers Powlesland <pierspowlesland@gmail.com>

Set balance of `CeloDistributionSchedule` contract (#162)

* state-migration: Initialize CeloDistributionSchedule

Fixes #155

* state-migration: Don't fail when distribution schedule update errors

* Review comments

state-migration: Set ParentBeaconRoot (#176)

This allows header validation to pass during snap sync

state-migration: Set address of distribution schedule (#177)

state-migration: Read total supply directly from state (#182)

* state-migration: Read totalSupply directly from storage

* Added trigger for updated dependencies

* Removen token bindings

---------

Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>

Fix l2 block older than l1 origin error (#184) (#187)

* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>

Migration script fixes (#179)

* Fixed migration for datadirs without ancients

The script was assuming that ancients would have been migrated and was
considering the numAncients-1 to be the next block to migrate but when
numAncients is zero that's a problem.

Also remved logic for  picking up where db migration left of for the
level db since it was complicating the logic and that process takes a
few seconds, which is nothing compared with the minutes taken to migrate
the ancients.

* Ensure that we set gas limit if migrating at pre-gingerbread point

Fix migration script gap in migrated blocks (#189)

* Fix migration script gap in migrated blocks

The range of ancient blocks to remove from the non ancients database was
off by one and resulted in a gap between ancients and non ancients.

Also corrected some log statements that were off by one.

Add pre-migration command to migration script (#192)

* add pre-migration command, rsync and ancients run in parallel, remove onlyAncients flag

* remove block and state migration sub-commands

* make non ancient migration its own step, add flag to measure time

* add more granular timers

* open db without freezer in state migration, remove clearAll

* fix error

* remove update flag from rsync command, add rsync comments

* delete commented out versions of checkForPrevFullMigration

* remove aliases

* remove clearNonAncients flag

* remove measureTime flag, always log time measurements

* remove logging from help text

* remove db reset

* move scan for extra ancients into pre-migration

* update README

* rename extraAncientNumHashes to strayAncientBlocks

state-migration: Fail if account would be overwritten (#202)

* state-migration: Fail if account would be overwritten

* Review changes

* Review changes 2

* Fail in unclear state

* more changes

* Use whitelist to decide if nonce and state are overwritten

Cosmetic changes to the migration script

- Use more lists for added readability
- Capitalize Alfajores and Celo
- Reorder scripting instructions to fit the actual order or operations
- Use GitHub callouts

migration: Add tests (#217)

* migration: Add tests for state migration

* migration: Fix issues shown by tests

* migration: pass allowlist into state migration

Allows for easier testing

* migration: Add test with allowlist

* Correct overwrite counter

* Use in memory DB

migration: Add working allowlist for Alfajores (#220)

* migration: Simplify tests

* migration: Add working allowlist for Alfajores

Adapt migration code to changes in StateDB

StateDB.CreateAccount used to copy existing balance, now it does not any
more.

migration: Set fields correctly for migration block (#212)

migration: Enable Granite (#226)

Write genesis file in state migration (#219)

* squash of #167

* add writeGenesis

* open old freezer in readonly mode, fix locking error

* remove devAlloc

* Revert "open old freezer in readonly mode, fix locking error"

This reverts commit e3fddea.

* fix locking error

* fix lint error, check errors, add comment

* remove comment

* filter extra genesis fields

* fix issue with genesis extra data

* update testdata

---------

Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>

migration: Overwrite create2deployer code (#233)

migration: Allow 'createx' preinstall (#238)

The code already exists on Alfajores and matches the one that would be
deployed, therefore we just allow this address.

add migration-block-number flag (#245)

* add migration-block-number flag

* address feedback

* move migration-block-number flag out of state migration options

Fixes for re-running migration script on same destination db  (#246)

* add reset flag

* add --checksum to rsync options
karlb pushed a commit that referenced this pull request Oct 16, 2024
This works by loading the database of a celo
node. It then removes all existing blocks and
generates a new genesis block including the
existing state tree.

Migrate to urfave/cli/v2

Update op-chain-ops/cmd/op-migrate/main.go

Co-authored-by: Karl Bartel <karl@karl.berlin>

Combine Cel2 migration scripts (#148)

* Initial script to play with celo DB history migration

* Can Read All the headers

Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com>

* Adds new command to migrate ancients db

* Adds comment

* Adds extension methods for transformation

* Implements Transform CeloBody

* Adds impl that runs steps in a concurrent pipeline

* Adds transformHead, verify hashing works

cleanup

* add migration for non-frozen blocks

* copy over entire db and modify in place, works with op-geth at piersy/minimal-data-migration

* remove unecessary copying, cleanup code

* close and reopen DBs

* migrate newdb in place

* saving progress

Co-authored-by: Mariano Cortesi <mcortesi@gmail.com>

* Refactor code to improve database migration process

* better logging

* refactor: inline parMigrateAncientRange

* Remove frozen blocks from nonAncient DB

* check hash matches on nonAncients migration

* clean up branch

Removes unused code, move code for better separation of concerns.

* decode into new types

* fix transformHeader

* make old freezer not readonly so that .meta files are created

* add configurable memory limit

* add comment about memory

* Added celo-dbmigrate Makefile target

* Added dockerfile for celo-dbmigrate and celo-migrate tools

* Workflow for running cel2-migration-tool

* Update cel2-migration-tool image registry

* update op-geth to point to https://github.com/celo-org/op-geth/commits/piersy/for-use-with-migrated-celo-datadir-use-gas-limit-differentiation-rebased-celo6/

* add celo6 logging

* rename scripts to celo-migrate-state and celo-migrate-blocks

* first pass at combining scripts

* saving progress on testing

* fix lint error, use %w to fmt errors

* add updated state migration input files to testdata

* add ability to run block and state migration seperately or together

* add option for migrating only frozen blocks

* remove old scripts

* minor logging improvements in block migrations

* invert clearNonAncients flag logic --> keepNonAncients, make dry-run flag only apply to state migration

* adds README, improves logging

* fix lint err

* Fix Makefile and Dockerfile

* move createNewDbIfNotExists

* rename keep-non-ancients

* update TODO to add more context and state changes

* Remove channel buffers from ancients migration

Co-authored-by: Valentin Rodygin <carterqw@gmail.com>

* bump default batch size to 100000

* add back extended usage string

* add info on state migration to README

* remove --state-dry-run flag

* update default batch size to 50k

* Adding building for op images

* Setting our values for image registry and repository

* update README

* fix logging when newAncients > oldAncients

* fix return value when skipping ancients

* skip transforming block bodies that have already been transformed

* misc. fixes to get re-runs with --keep-non-ancients working

* adds TODO

* addresses cosmetic feedback

* add flag for specifying a buffer

* Show progress on rsync

* Update to latest op-geth

* state-migration: Refactor subtask

* state-migration: Use EIP1559 settings from deploy config

Fixes #135

* state-migration: Enable Fjord hardfork during migration

Fixes #160

* state-migration: Deterministicly set migration block timestamp

Fixes #157

Sets the timestamp to be 5s after the last block.

* state-migration: Set WithdrawalsHash in Cel2 migration block

* fixup! Fix Makefile and Dockerfile

* add note to README about using snapshots for pre-migration

* Set blob gas header fields for transition block

These are now required to be set since cancun was activated.

* Use InitialBaseFee for pre-gingerbread transitionb

* Fix warnings about capitalized error strings

* Output chain config as marshalled JSON

* state-migration: Handle accounts with existing balance

Fixes #158

* remove allocs file, add instructions for how to generate allocs file to README, update TODOs

---------

Co-authored-by: Mariano Cortesi <mariano@clabs.co>
Co-authored-by: Alec Schaefer <alecps@users.noreply.github.com>
Co-authored-by: Mariano Cortesi <mcortesi@gmail.com>
Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>
Co-authored-by: Paul Lange <palango@gmx.de>
Co-authored-by: Valentin Rodygin <carterqw@gmail.com>
Co-authored-by: Piers Powlesland <pierspowlesland@gmail.com>

Set balance of `CeloDistributionSchedule` contract (#162)

* state-migration: Initialize CeloDistributionSchedule

Fixes #155

* state-migration: Don't fail when distribution schedule update errors

* Review comments

state-migration: Set ParentBeaconRoot (#176)

This allows header validation to pass during snap sync

state-migration: Set address of distribution schedule (#177)

state-migration: Read total supply directly from state (#182)

* state-migration: Read totalSupply directly from storage

* Added trigger for updated dependencies

* Removen token bindings

---------

Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>

Fix l2 block older than l1 origin error (#184) (#187)

* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>

Migration script fixes (#179)

* Fixed migration for datadirs without ancients

The script was assuming that ancients would have been migrated and was
considering the numAncients-1 to be the next block to migrate but when
numAncients is zero that's a problem.

Also remved logic for  picking up where db migration left of for the
level db since it was complicating the logic and that process takes a
few seconds, which is nothing compared with the minutes taken to migrate
the ancients.

* Ensure that we set gas limit if migrating at pre-gingerbread point

Fix migration script gap in migrated blocks (#189)

* Fix migration script gap in migrated blocks

The range of ancient blocks to remove from the non ancients database was
off by one and resulted in a gap between ancients and non ancients.

Also corrected some log statements that were off by one.

Add pre-migration command to migration script (#192)

* add pre-migration command, rsync and ancients run in parallel, remove onlyAncients flag

* remove block and state migration sub-commands

* make non ancient migration its own step, add flag to measure time

* add more granular timers

* open db without freezer in state migration, remove clearAll

* fix error

* remove update flag from rsync command, add rsync comments

* delete commented out versions of checkForPrevFullMigration

* remove aliases

* remove clearNonAncients flag

* remove measureTime flag, always log time measurements

* remove logging from help text

* remove db reset

* move scan for extra ancients into pre-migration

* update README

* rename extraAncientNumHashes to strayAncientBlocks

state-migration: Fail if account would be overwritten (#202)

* state-migration: Fail if account would be overwritten

* Review changes

* Review changes 2

* Fail in unclear state

* more changes

* Use whitelist to decide if nonce and state are overwritten

Cosmetic changes to the migration script

- Use more lists for added readability
- Capitalize Alfajores and Celo
- Reorder scripting instructions to fit the actual order or operations
- Use GitHub callouts

migration: Add tests (#217)

* migration: Add tests for state migration

* migration: Fix issues shown by tests

* migration: pass allowlist into state migration

Allows for easier testing

* migration: Add test with allowlist

* Correct overwrite counter

* Use in memory DB

migration: Add working allowlist for Alfajores (#220)

* migration: Simplify tests

* migration: Add working allowlist for Alfajores

Adapt migration code to changes in StateDB

StateDB.CreateAccount used to copy existing balance, now it does not any
more.

migration: Set fields correctly for migration block (#212)

migration: Enable Granite (#226)

Write genesis file in state migration (#219)

* squash of #167

* add writeGenesis

* open old freezer in readonly mode, fix locking error

* remove devAlloc

* Revert "open old freezer in readonly mode, fix locking error"

This reverts commit e3fddea.

* fix locking error

* fix lint error, check errors, add comment

* remove comment

* filter extra genesis fields

* fix issue with genesis extra data

* update testdata

---------

Co-authored-by: Javier Cortejoso <javier.cortejoso@gmail.com>

migration: Overwrite create2deployer code (#233)

migration: Allow 'createx' preinstall (#238)

The code already exists on Alfajores and matches the one that would be
deployed, therefore we just allow this address.

add migration-block-number flag (#245)

* add migration-block-number flag

* address feedback

* move migration-block-number flag out of state migration options

Fixes for re-running migration script on same destination db  (#246)

* add reset flag

* add --checksum to rsync options
karlb pushed a commit that referenced this pull request Jan 13, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 15, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 15, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 20, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 27, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 29, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 29, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Jan 31, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Feb 4, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Feb 4, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Feb 12, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Feb 12, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
karlb pushed a commit that referenced this pull request Feb 12, 2025
* Revert to using time.Now() for migration block

Instead of simply adding 5 to the parent block time.

We really do need a deterministic time for the migration block so that
all parties that run the migration arrive at the same migration block
but the problem is that op-geth requires that the L2 migration block
(aka l2 origin) occurs after the l1 origin (I guess the point where you
deploy the bridge contracts to the l1). When we migrate a partially
synced datadir the block before the transition block will be very old,
up to 4 years old! So of course it occurs before the l1 origin. So a fix
just to get things working is to use time.Now(), but probably we should
make this a configurable parameter.

* add flag to specify timestamp

* Update op-chain-ops/cmd/celo-migrate/main.go

---------

Co-authored-by: piersy <pierspowlesland@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants