Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Implement parameterisable modules#1800

Merged
gavofyork merged 40 commits intomasterfrom
gui-parameterisable-module
Mar 15, 2019
Merged

Implement parameterisable modules#1800
gavofyork merged 40 commits intomasterfrom
gui-parameterisable-module

Conversation

@gui1117
Copy link
Contributor

@gui1117 gui1117 commented Feb 14, 2019

related to #1688

This PR contains currently the modification of Balances (and so srml-support macros) to be able to be used with an instance parameter representing the actual instance. Thus dependant module could make use of two balances at the same time or runtime could use many balances etc...

For module using Balances:

This balance module has been modified but default to a specific instance thus it almost doesn't break API for existing module depending on this one. Also the prefix used for storage of default instance is same as old one. The only difference is GenesisConfig that has a new PhantomData field.

For Balances module:

Not that much change is needed just declarations have new parameters and some inner struct have also an Instance generic.
And also it declares its trait with a generic parameter: Trait<Instance: Instantiable=DefaultInstance> (note: the bound to Instantiable is not required)
use can just write Trait<Instance=DefaultInstance>

Macros:

decl_storage: The whole difficulty is being able to implement fn key() -> &'static [u8].

(how did I get there)
First I thought about implementing storages for some concrete types U0 to U16 for example but that would have required a lot of where clause in decl_storage impl module but also in user impl.
Secondly I thought about make Instance implementing a trait: Trait Instantiable { const PREFIX: &'static str; } using lazy_static to create a final key that is the concatenation of old prefix (cratename + storagename) and a number:

fn key() -> &'static [u8] {
lazy_static!{ static PREFIX: String = { 
   let a = String::from(#prefix);
   a.push_str(Instance::PREFIX);
   a
};
}

but that fails to compile because of static of Instance being used in outer function.

The final solution is derived from it: I actually create the trait Instantiable inside decl_storage with the methods I need and implement it for some structure I make. Expanded it results in

pub trait Instantiable {
  const PREFIX: &'static str:
  build_prefix_once_for_AStorageName(suffix: &'static [u8]) -> &'static [u8];
  build_prefix_once_for_AnotherStorageName(suffix: &'static [u8]) -> &'static [u8];
}
struct DefaultInstance;
impl Instantiable for DefaultInstance {
  const PREFIX: &'static str=""// to be compatible with old prefix
  build_prefix_once_for_AStorageName(suffix: &'static [u8]) -> &'static [u8] {
	static LAZY: #scrate::lazy::Lazy<Vec<u8>> = #scrate::lazy::Lazy::INIT;
	LAZY.get(|| {
		let mut final_prefix = Vec::new();
		final_prefix.extend_from_slice(suffix);
		final_prefix.extend_from_slice(Self::PREFIX.as_bytes());
		final_prefix
	})

  }
  build_prefix_once_for_AnotherStorageName(suffix: &'static [u8]) -> &'static [u8] {
	static LAZY: #scrate::lazy::Lazy<Vec<u8>> = #scrate::lazy::Lazy::INIT;
	LAZY.get(|| {
		let mut final_prefix = Vec::new();
		final_prefix.extend_from_slice(suffix);
		final_prefix.extend_from_slice(Self::PREFIX.as_bytes());
		final_prefix
	})
  }
}

As a note methods couldn't be implemented in a generic way because of how static is build by rust. To understand why see https://github.com/hukumka/generic_static

and then key function is implemented this way:

fn key() -> &'static [u8] {
Instance::build_prefix_once_for_....(#old_prefix)
}

and the key will be computed only once.

decl_storage: modified to be able to have an instance and a default instance on its structure (module, call_type etc...)

decl_event: no that much things changed just being able to make Event generic over two parameters if wanted.

What to do:

  • support linked map.
  • macro call are not very good and inherited from previous attempt. we could simplify things. omit the default parameter as it is not configuration or make it configuratable, make instance just a keyword with the number of instance wanted. I'm on it.
  • code should be rewritten in some parts.
  • documentation.
  • tests

also as a note: a multiple instance module requires now a a decl_storage to actually create the trait.

also we have to keep in mind this might complexify decl_storage in way we don't want. But all in all it seems we always can just make the trait requiring methods we need.

@gui1117 gui1117 added the A3-in_progress Pull request is in progress. No review needed at this stage. label Feb 14, 2019
@tomusdrw
Copy link
Contributor

There is also quite a lot of todos in the code.

transfer_fee: 0,
creation_fee: 0,
vesting: vec![],
_genesis_phantom_data: rstd::marker::PhantomData,
Copy link
Member

@gavofyork gavofyork Feb 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty horrible... is there some way to avoid it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed,
When GenesisConfig needs to access to storage it needs to be generic other the trait. This trait is sometimes contained in GenesisConfig fields and sometimes not, in the latter cases we already add this field _genesis_phantom_data.

Now I changed the condition to requires phantom data also when the trait is generic over instances.
But this condition does actually have false positive. For instance the balances module could also not have a phantom, the Instance generic type is used in fields like T::Balance.
But as we don't access Trait definition inside the macro there is no way to know if T::Balance is actually an associated type from Trait (and so use the Instance generic type parameter) or an associated type from a dependency of Trait like T::AccountId would be (and so doesn't use Instance generic type parameter).

To summarize: This field is not always needed, it is not for balances, but there is no way to know inside the macro. The only solution I see is: we could provide a way for the user to say no_phantom_data (this seems good and would make balances module truly backward compatible)

Also note that user could write instead:

		t.extend(GenesisConfig::<Test>{
			dummy: 42,
			foo: 24,
			.. Default::default()
		}.build_storage().unwrap().0);

@gavofyork
Copy link
Member

There is also quite a lot of todos in the code.

To be fair, it's marked inprogress :)

@gui1117 gui1117 force-pushed the gui-parameterisable-module branch 2 times, most recently from 51aa477 to af12f8e Compare February 18, 2019 16:23
@gui1117
Copy link
Contributor Author

gui1117 commented Feb 18, 2019

small update, also I'll wait for #1809 and then impl for linked_map.

  • user of Balance don't have anything to change now as a new keyword has been added to the decl_storage macro to be able to specify to skip phantom data field when we know it is not necessary.

@gui1117 gui1117 force-pushed the gui-parameterisable-module branch from 7e83447 to 22941b4 Compare February 19, 2019 14:50
@gui1117
Copy link
Contributor Author

gui1117 commented Feb 19, 2019

small update:

  • implemented for linked_map,
  • still have to do documentatio and test and construct_runtime

but otherwise eveything seems fine to me.

@gui1117
Copy link
Contributor Author

gui1117 commented Feb 20, 2019

working the construct_runtime it will probably end up to something like
Example0: example::{Module<example::Instance0>, Event<T, example::Instance0>}

also as a note: I wonder in how many situation this would be useful. If we want to implement it for balance at the end then it worth implementing this way. But if we only expect some user to implement it for a few cases then I don't know if it worth maintaining this way. Because one could simply do a macro that append the storage name and create multiple instance:

multiple_module! {
  pub trait Trait: system::Trait {
  }
  decl_module! {
  ...
}

with multiple_module macro expanding to

mod module1 {
  pub trait Trait: system::Trait {
  }
  decl_module! {
  ...
  storage as Module1
  ...
}

mod module2 {
  pub trait Trait: system::Trait {
  }
  decl_module! {
  ...
  storage as Module2
  ...
}

And this would be equivalent in terms of feature for other modules and runtime.

EDIT: or might be better in build.rs

@gui1117
Copy link
Contributor Author

gui1117 commented Feb 20, 2019

@gavofyork actually I wonder about your opinions on this feature, mine are mixed.
The good part is that it is entirely backward compatible it only extend definiton of modules. the bad part is that it add a bit of complexity on our side for a need I don't know how to estimate.
But if the need is here. It seems doable

@xlc
Copy link
Contributor

xlc commented Feb 20, 2019

Some feedback from a parachain developer.
We are working on a chain with multiple tokens and we have a module implements all the tokens. It provides multiple implementation of the Currency trait so we can pass different Currency trait to different modules.
We don't need multiple instantiations of a module, we need multiple implementations of a trait, which is already supported.
The general trend of runtime modules should be depending on traits, instead of depending on a particular module.

@gui1117
Copy link
Contributor Author

gui1117 commented Feb 20, 2019

thanks @xlc
as a note I could imagine a use case like a module requiring two currency:

trait Trait {
  type Currency1: Currency;
  type Currency2: Currency;
}

a user of this module could use balance for each currency but with two different instance

impl Trait for Runtime {
  type Currency1 = balance::Module<Self, balance::Instance1>;
  type Currency2 = balance::Module<Self, balance::Instance2>;
}

@gui1117 gui1117 force-pushed the gui-parameterisable-module branch 3 times, most recently from c7f2500 to 3e5a3a5 Compare February 22, 2019 10:13
@gui1117 gui1117 force-pushed the gui-parameterisable-module branch from 3e5a3a5 to 331d4a2 Compare February 27, 2019 14:03
@gui1117
Copy link
Contributor Author

gui1117 commented Feb 27, 2019

small update: I am more confident in the code now, still have to check Log part and inherent part.
Also the new syntax is implemented you can take a look at node/runtime as I temporarly pushed the test:

construct_runtime!(
	pub enum Runtime with Log(InternalLog: DigestItem<Hash, SessionKey>) where
		Block = Block,
		NodeBlock = node_primitives::Block,
		UncheckedExtrinsic = UncheckedExtrinsic
	{
		System: system::{default, Log(ChangesTrieRoot)},
....
		Balances1: balances::<Instance1>::{Event<T, I>},
		Example: example::<Instance1>::{Module, Call, Storage, Event<T, I>, Config<T, I>, Origin<T, I>},
	}
);

gui1117 added 3 commits March 1, 2019 14:55
this requires parity codec implements codec for core::marker::PhantomData
@gui1117
Copy link
Contributor Author

gui1117 commented Mar 1, 2019

full example builds, inherent, origin and log are implemented.

it requires parity-codec to implement Codec for core::marker::Phantom

@gui1117 gui1117 removed A3-in_progress Pull request is in progress. No review needed at this stage. A4-gotissues labels Mar 11, 2019
@gavofyork
Copy link
Member

@thiolliere is the minor release something you can take care of?

@gavofyork
Copy link
Member

i'l like to push forward and get this in now.

@gavofyork gavofyork added A7-looksgoodcantmerge and removed A0-please_review Pull request needs code review. A4-gotissues labels Mar 13, 2019
@bkchr
Copy link
Member

bkchr commented Mar 13, 2019

Wouldn't it be easier and much less inversive to add the instance as associated type?

@gui1117
Copy link
Contributor Author

gui1117 commented Mar 14, 2019

@bkchr we can't implement multiple times the same trait with a different associated type, either we make the trait generic or we provide multiple traits.

note: indeed providing multiple traits could be a solution, I would implement it with a global procedural macro like

// expand to multiple modules or a single module but different trait like TraitInstance2, TraitInstance3
// rename storage name for each implementation as $(ModuleName)Instance2, ModuleNameInstance3
decl_multiple_instance! {
// whole module except imports
}

@gui1117 gui1117 added A0-please_review Pull request needs code review. and removed A7-looksgoodcantmerge labels Mar 14, 2019
@gui1117
Copy link
Contributor Author

gui1117 commented Mar 15, 2019

ok codec published, dependency updated, and master merged

@gavofyork gavofyork merged commit 7c95fb0 into master Mar 15, 2019
@gavofyork gavofyork deleted the gui-parameterisable-module branch March 15, 2019 18:25
@tomusdrw tomusdrw mentioned this pull request Mar 20, 2019
MTDK1 pushed a commit to bdevux/substrate that referenced this pull request Apr 12, 2019
* first implementation

* remove done comment

* origin done

* impl log for instance

* impl inherent for instance

* Fix wasm build + full example build

this requires parity codec implements codec for core::marker::PhantomData

* patch parity-codec link to github branch

* improve internal names and fix instance prefix

* Fix in macros

* add test modules for support

this allow to test for construct_runtime as well.

The reason to have put that in another crate is:
* if we put test in `tests/` dir of srml/support then decl_storage fails to get
  srml-support access because it believes it is inside srml-support
  crate and so derive access to `quote!{ crate }` but this is wrong
  (and I don't see any way to prevent that, and it only bother us so I
  don't think that matters that much)
* if we put test inside lib.rs then contruct_runtime cannot be used
  because it call some macros that are defined with macros
  (decl_outer_event and decl_outer_origin) and thus rustc complains.

* defaultinstance to its own struct to avoid errors

* enforce <T, I> for Event and Config, impl test

* add origin, log, inherent to test

* test more code generation

* basic storage test

* fix typo

* rename a few imports and field

* delete wip test in example and runtime

* change default prefix to make it backward compatible with test

* rename Instance to I and Instantiable to Instance

note: the name of generic parameter I is only enforce by decl_module!
and this could be rewritten

* doc

* clean old TODOs

* update parity-codec to 3.2

* update node impl version + builds

* fix warning

* fix unrelated grandpa test

* refactor code
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_review Pull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants