Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Trie simplification.#2815

Merged
gavofyork merged 53 commits intoparitytech:masterfrom
cheme:simple-codec
Aug 2, 2019
Merged

Trie simplification.#2815
gavofyork merged 53 commits intoparitytech:masterfrom
cheme:simple-codec

Conversation

@cheme
Copy link
Contributor

@cheme cheme commented Jun 6, 2019

This PR link substrate code base with trie code base post paritytech/trie#11 .

Currently review and publishing of the change on trie crates are required first.
Yet this is important code to help with the trie review.

I labelled this PR as in progress because it is currently using patched dependancies, but I see the code as reviewable.

This PR switch to the new trie definition (see polkadot re spec https://github.com/w3f/polkadot-re-spec ).
The switch can be de-activated by adding 'legacy-trie' from default features in crates containing this feature, to check former implementation.

Ideally the trie definition should not be local to trie crate but use as a type parameter. I started this refactoring with the trie crate (this explain some of the design), but to change it for substrate it would require replacing the ubiquitous Hasher type parameter by TrieOps.
That is a task that touch all code base and probably something that should be merge quickly, so I did not make it in this PR (the 'legacy-trie' feature is a temporary alternative). More generally an type parameter with TrieOps as associatid trait may be better to avoid this kind of refactoring whenever a new trait need to be accessible almost everywhere like Hasher.

The legacy trie code should probably be dropped, but it seems important to me to test substrate with it because paritytech/trie#11 contains a few api change.

Also a few test are temporary disabled (macros until we stop using patched crate).

@cheme cheme added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jun 6, 2019
@gavofyork gavofyork added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. A4-gotissues labels Jun 25, 2019
@gavofyork
Copy link
Member

@cheme needs resolving, and you'll probably want to bump impl_version of the runtime to avoid the CI from labelling gotissues.

@gavofyork gavofyork added A3-needsresolving and removed A0-please_review Pull request needs code review. labels Jun 25, 2019
@cheme cheme added A0-please_review Pull request needs code review. and removed A3-needsresolving labels Jun 25, 2019
@cheme cheme added this to the 2.0-k milestone Jul 7, 2019
@gavofyork
Copy link
Member

@svyatonik @arkpar any chance you could take a look at the trie logic here and verify it's sensible?

@svyatonik svyatonik self-requested a review July 23, 2019 07:42
//for test_i in 0..10000 {
for test_i in 0..1000 {
if test_i % 50 == 0 {
println!("{:?} of 10000 stress tests done", test_i);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it just 1000 tests now?

BranchWithValue,
}

impl Encode for NodeHeader {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it the same encoding/decoding as ReferenceTrieStreamNoExt? Can't we just re-use that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the same implementation, we got this double implementation situation (reference-test from trie is also problematic for other reason).
It could be possible to move this code in its own crate in trie, but it will not be in substrate.
We could also use this implementation from reference trie but at the cost of a dependency to substrate (maybe with a published crate this won't be such an issue).

Copy link
Contributor Author

@cheme cheme Aug 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking twice, codec implementation needs to be in substrate repo, so to use it for tests in trie repo we could create a new crate core/trie/nodecodec that does not depends on any substrate crate and could easily be published.
That would not solve the issue with trie repo reference trie crate dependencies but would remove this duplicated code.
Do you think it shall be done now or later (with a issue of course)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I guess it is fine as it is

WasmBuilderSource::Crates("1.0.4"),
WasmBuilderSource::CratesOrPath {
path: "../../core/utils/wasm-builder",
version: "1.0.4",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bkchr please check this

Copy link
Contributor Author

@cheme cheme Aug 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be reverted (I need it for patching wasm, but when trie crates will be published this won't be necessary anymore, I should have put a comment).

@gavofyork
Copy link
Member

trie changes merged, so i guess the repo path for that dependency can change.

@cheme
Copy link
Contributor Author

cheme commented Aug 1, 2019

Nice, I will publish and update first thing in the morning tomorrow.

@gavofyork
Copy link
Member

@cheme please ping @arkpar and @svyatonik when this is updated and you've checked all existing comments and typical issues relating to code quality. i'd like to get this in today.

@cheme
Copy link
Contributor Author

cheme commented Aug 2, 2019

Yes I got a small unexpected issue, If anyone can review/approve paritytech/trie#26 (it is just a version update).

@cheme
Copy link
Contributor Author

cheme commented Aug 2, 2019

@arkpar @svyatonik I did update patched dependency, the PR should be ready for review.

About the fact that codec code is duplicated, I think it should remain in substrate repo (and possibly be use by reference trie after getting split in a less dependant crate), but did not do the necessary work in this PR.

CI is failing but it seems to be global to all CI pipelines.

@gavofyork gavofyork merged commit ba281ec into paritytech:master Aug 2, 2019
@gnunicorn gnunicorn modified the milestones: 2.1, Polkadot Mar 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_review Pull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants