CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

I have read and hereby sign the Contributor License Agreement.

Thanks a lot for the application. Are you aware of the following grant application: #1950? Potentially both teams could work together on the project or at least on separate tasks initially. Do you also want to use Zombienet for the PoC Tests? We could potentially put you in touch with the Zombienet team.

Given that our primary objective is that of aligning with specifications, our initial focus is on Conformance Testing at the individual node level. Additionally, Integration testing plays a pivotal role, ensuring that interactions and compatibility within system components are seamless.

Zombienet is more adequate system tests or detailed network simulations that involve multiple nodes, and its introduction at this point results in some problems/challenges:

• Noise: Introducing a full chain with various nodes increases the variables in play. This heightened complexity can hinder our efforts to isolate specific issues or confirm the conformance of individual nodes.

• State Management: Achieving the precise state needed for effective testing becomes more challenging within Zombienet’s multi-node framework.
  Precision: Although natural language can be intuitive, some specifications are deeply technical. Our concern is that Zombienet might lack the expressiveness required to evaluate such nuanced conformance, especially when identifying spec gaps or areas that are underspecified.

Zombienet will undoubtedly be of great value in subsequent testing phases. However, our immediate priority is targeted and meticulous testing, ensuring the resilience of each node independently and laying a firm foundation for broader evaluations later on.

We definitely want to use Zombienet but we don't find it adequate in this early phase.

With respect to @bhargavbh questions:

Could you please elaborate on the test fixtures and how you decide the abstraction level for the interfaces so that the tests can be integrated with multiple hosts implementations?

Each node implementation consists of distinct components, each specified individually. Beyond just exposing interfaces, it's crucial to have adequate introspection and instrumentation for these components.

Our focus on elements such as Host API, trie proof, and scale codes is due to their varied levels of exposure. While it's essential to verify each group of specs, we aim to do so without exposing bare all implementation specifics. Therefore, we plan to prepare adapters tailored for each subset of components. These adapters will adhere to a future SpecVerificationAPI, ensuring each component is instrumented tested appropriately.

there is already comprehensive tests available for SCALE codec here, which could be integrated in your testing framework to convince the feasibility of the architecture.

While we plan to use these tests as a reference, they are unit tests that are being much tied to the code itself. In many cases, they depend on implementation details that are not part of the specs that we intend to confirm. We initially don't think that this specific source code can be directly repurposed to test other implementations.

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at grantsPR@web3.foundation and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀