Skip to content

native: implement HF-based update#3402

Merged
roman-khimov merged 12 commits intomasterfrom
native-upd
May 10, 2024
Merged

native: implement HF-based update#3402
roman-khimov merged 12 commits intomasterfrom
native-upd

Conversation

@AnnaShaleva
Copy link
Member

@AnnaShaleva AnnaShaleva commented Apr 4, 2024

Let our natives be updated! Close #3213.

TODO:

  • Implement Oracle service cache update.
  • Make lazy MD cache not so lazy and remove generic Methods/Events.
  • Use cached HF-specific MD for new HF-specific MD initialization if no changes are required.
  • Enable Cockatrice with all dependent logic.
  • Tests (unit and state compatbility).

@codecov
Copy link

codecov bot commented Apr 5, 2024

Codecov Report

Attention: Patch coverage is 84.52012% with 50 lines in your changes are missing coverage. Please review.

Project coverage is 86.08%. Comparing base (83fdcc8) to head (8995f11).
Report is 18 commits behind head on master.

❗ Current head 8995f11 differs from pull request most recent head 9b8d579. Consider uploading reports for the commit 9b8d579 to get more accurate results

Files Patch % Lines
pkg/core/interop/context.go 84.61% 9 Missing and 5 partials ⚠️
pkg/core/native/oracle.go 65.21% 8 Missing ⚠️
pkg/core/native/management.go 88.33% 4 Missing and 3 partials ⚠️
pkg/config/hardfork.go 71.42% 3 Missing and 1 partial ⚠️
pkg/core/native/native_nep17.go 86.66% 2 Missing and 2 partials ⚠️
pkg/config/hardfork_string.go 33.33% 2 Missing ⚠️
pkg/core/blockchain.go 93.54% 1 Missing and 1 partial ⚠️
pkg/core/native/designate.go 71.42% 1 Missing and 1 partial ⚠️
pkg/core/native/native_gas.go 50.00% 1 Missing and 1 partial ⚠️
pkg/core/native/notary.go 50.00% 1 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3402      +/-   ##
==========================================
- Coverage   86.12%   86.08%   -0.05%     
==========================================
  Files         331      332       +1     
  Lines       38114    38339     +225     
==========================================
+ Hits        32827    33005     +178     
- Misses       3776     3804      +28     
- Partials     1511     1530      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AnnaShaleva AnnaShaleva force-pushed the native-upd branch 3 times, most recently from 02adea3 to a51a62d Compare April 9, 2024 11:50
@AnnaShaleva AnnaShaleva requested a review from roman-khimov April 9, 2024 11:50
@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Apr 9, 2024

@roman-khimov, the implementation seems to be in the final stage, I'm going to test it against C# node states for mainnet/testnet, and if everything is OK, then write a couple of unit tests. Need your opinion on the implementaiton structure, it differs from C# a bit, but behaviour is compatible.

@AnnaShaleva
Copy link
Member Author

With the latest C# node fixes mainnet states match perfectly (with Cockatrace enabled at 5210001 height):

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ go run scripts/compare-dumps/compare-dumps.go ./dumps/GoDump_mainnet_5224362/ ./dumps/Storage_334f454e_mainnet_5225043_cockatrice/
Processing directory BlockStorage_0
Processing directory BlockStorage_100000
Processing directory BlockStorage_200000
Processing directory BlockStorage_300000
Processing directory BlockStorage_400000
Processing directory BlockStorage_500000
Processing directory BlockStorage_600000
Processing directory BlockStorage_700000
Processing directory BlockStorage_800000
Processing directory BlockStorage_900000
Processing directory BlockStorage_1000000
Processing directory BlockStorage_1100000
Processing directory BlockStorage_1200000
Processing directory BlockStorage_1300000
Processing directory BlockStorage_1400000
Processing directory BlockStorage_1500000
Processing directory BlockStorage_1600000
Processing directory BlockStorage_1700000
Processing directory BlockStorage_1800000
Processing directory BlockStorage_1900000
Processing directory BlockStorage_2000000
Processing directory BlockStorage_2100000
Processing directory BlockStorage_2200000
Processing directory BlockStorage_2300000
Processing directory BlockStorage_2400000
Processing directory BlockStorage_2500000
Processing directory BlockStorage_2600000
Processing directory BlockStorage_2700000
Processing directory BlockStorage_2800000
Processing directory BlockStorage_2900000
Processing directory BlockStorage_3000000
Processing directory BlockStorage_3100000
Processing directory BlockStorage_3200000
Processing directory BlockStorage_3300000
Processing directory BlockStorage_3400000
Processing directory BlockStorage_3500000
Processing directory BlockStorage_3600000
Processing directory BlockStorage_3700000
Processing directory BlockStorage_3800000
Processing directory BlockStorage_3900000
Processing directory BlockStorage_4000000
Processing directory BlockStorage_4100000
Processing directory BlockStorage_4200000
Processing directory BlockStorage_4300000
Processing directory BlockStorage_4400000
Processing directory BlockStorage_4500000
Processing directory BlockStorage_4600000
Processing directory BlockStorage_4700000
Processing directory BlockStorage_4800000
Processing directory BlockStorage_4900000
Processing directory BlockStorage_5000000
Processing directory BlockStorage_5100000
Processing directory BlockStorage_5200000
Processing directory BlockStorage_5300000
file BlockStorage_5300000/dump-block-5225000.json: dump files differ in size: 362 vs 1000
compare-dumps dumpDirA dumpDirB
exit status 1

Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be OK.

// MethodAndPrice is a generic hardfork-independent native contract method descriptor.
type MethodAndPrice struct {
HFSpecificMethodAndPrice
ActiveFrom *config.Hardfork
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why are you keeping all of them this way. You can have the one and only proper contract version at the current height and then just switch it when HF happens.

Historic calls maybe? These do require some dances.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have the one and only proper contract version at the current height

That was an early version of this PR. And yes, due to the historic calls. To manage single contract version at a time, we need to initialize it every time when new historic call is processed (which I suppose is far from the best solution) or we need some kind of cache. So if we're to implement this cache anyway, then it's better to initialize it once during the node start in order not to affect the node performance during blocks processing. And, as discussed, we don't have a lot of natives/hardforks, so the cache size is not a problem for us (and we store pointers in cache anyway, as a result the most part of CS structures are shared between hardforks).

But that's just my thoughts. We can get back to one and only proper contract version.

// data in the storage.
func (s *Designate) Initialize(ic *interop.Context) error {
func (s *Designate) Initialize(ic *interop.Context, hf *config.Hardfork) error {
if hf != s.ActiveIn() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking here...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm against of this change because I assume (from C# implementation and neo-project/neo#3200) that in future we still might want to (e.g.) put something into the contract storage on the next hardfork. And this code will be contract-dependent.

That's the whole point of neo-project/neo#3200, otherwise if we do call Initialize only on deploy, then this issue is completely useless.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think?

storedJ, _ := json.Marshal(storedCS)
autogenJ, _ := json.Marshal(autogenCS)
return fmt.Errorf("native %s: version mismatch for the latest hardfork %s (stored contract state differs from autogenerated one), "+
"try to resynchronize the node from the genesis: %s vs %s", md.Name, current, string(storedJ), string(autogenJ))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's a friendly error message. Many users may even not notice the real message in these JSON dumps.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These dumps are not indented, so they are quite compact, I've tested it. I may add some indent. Or is it better to remove dumps at all?


// Hashes of all native contracts.
var (
Management util.Uint160
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, can we have them precomputed automatically and saved as consts? //go:generate of some kind. We can combine them then with nativenames probably, consts don't cost a lot.

Copy link
Member Author

@AnnaShaleva AnnaShaleva Apr 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the last commit, it's an interesting experiment, but I still want to keep Uint160 variables to reuse them in other code.

We can combine them then with nativenames probably,

They have different names, is it OK? E.g. nativenames.Gas vs nativehashes.GasToken. If it's OK, then I'll merge them into single package.

A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Ported as a part of neo-project/neo#2942.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
External users should use HF-specific methods and events.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Initialize all necessary HF-specific contract descriptors once during
contract construction.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
With all associated native API changes ported from
neo-project/neo#2925 and
neo-project/neo#3154.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
It will give us a clue on what's wrong with contract states if something
unexpected happen.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Similar to nativenames, instantiate once and then reuse everywhere.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Ensure that Blockchain constructor is able to distinguish empty
Hardforks map (no hardforks should be enabled) from nil hardforks map
(the default value should be used in this case, i.e. all hardforks
should be active from genesis).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
It's useful to keep the ordered set of native contract names.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
AnnaShaleva added a commit that referenced this pull request Apr 25, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@AnnaShaleva
Copy link
Member Author

@roman-khimov, do we need better unit-test coverage for this PR?

AnnaShaleva added a commit that referenced this pull request May 8, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@AnnaShaleva AnnaShaleva requested a review from roman-khimov May 8, 2024 09:05
@roman-khimov roman-khimov merged commit b21db99 into master May 10, 2024
@roman-khimov roman-khimov deleted the native-upd branch May 10, 2024 08:53
AnnaShaleva added a commit that referenced this pull request May 14, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support HF-based native contract update

2 participants