Add cache to native contract executions V2 by shargon · Pull Request #1766 · neo-project/neo

shargon · 2020-07-14T17:13:38Z

Close #1764
Other version for share the snapshot

This cache avoid the second deserialization process

Benchmarks:

[TestMethod]
        public void TestGetValidators1()
        {
            var snapshot = Blockchain.Singleton.GetSnapshot();

            using (ScriptBuilder sb = new ScriptBuilder())
            {
                sb.EmitAppCall(NativeContract.NEO.Hash, "getNextBlockValidators", new object[0]);

                // Fake validators

                StorageItem storage = snapshot.Storages.GetAndChange(CreateStorageKey(14), () => new StorageItem());
                storage.Value = NativeContract.NEO.GetValidators(snapshot).ToByteArray();

                // Start benchmark

                Stopwatch sw = Stopwatch.StartNew();

                for (int x = 0; x < 1_000; x++)
                {
                    using var engine = ApplicationEngine.Run(sb.ToArray(), snapshot, testMode: true);

                    var result = engine.ResultStack.Peek();
                    result.GetType().Should().Be(typeof(VM.Types.Array));
                    ((VM.Types.Array)result).Count.Should().Be(7);
                    ((VM.Types.Array)result)[0].GetSpan().ToHexString().Should().Be("02486fd15702c4490a26703112a5cc1d0923fd697a33406bd5a1c00e0013b09a70");
                    ((VM.Types.Array)result)[1].GetSpan().ToHexString().Should().Be("024c7b7fb6c310fccf1ba33b082519d82964ea93868d676662d4a59ad548df0e7d");
                    ((VM.Types.Array)result)[2].GetSpan().ToHexString().Should().Be("02aaec38470f6aad0042c6e877cfd8087d2676b0f516fddd362801b9bd3936399e");
                    ((VM.Types.Array)result)[3].GetSpan().ToHexString().Should().Be("03b209fd4f53a7170ea4444e0cb0a6bb6a53c2bd016926989cf85f9b0fba17a70c");
                    ((VM.Types.Array)result)[4].GetSpan().ToHexString().Should().Be("03b8d9d5771d8f513aa0869b9cc8d50986403b78c6da36890638c3d46a5adce04a");
                    ((VM.Types.Array)result)[5].GetSpan().ToHexString().Should().Be("02ca0e27697b9c248f6f16e085fd0061e26f44da85b58ee835c110caa5ec3ba554");
                    ((VM.Types.Array)result)[6].GetSpan().ToHexString().Should().Be("02df48f60e8f3e01c48ff40b9b7f1310d7a8b2a193188befe1c2e3df740e895093");
                }

                sw.Stop();
                Console.WriteLine(sw.Elapsed);
            }
        }

Without this PR: 00:00:02.1273432
With this PR: 00:00:00.1614797

shargon · 2020-07-14T17:39:00Z

Also it was detected that in policy we used ToByteArray for store BigInteger instead of ToByteArrayStandard

Tommo-L · 2020-07-15T02:54:43Z

This cache avoid the second deserialization process

Transaction also have this problem, when nodes start to relay a transaction after verification.

roman-khimov · 2020-07-15T08:49:34Z

src/neo/SmartContract/Native/Tokens/NeoToken.cs

            base.OnPersist(engine);
            StorageItem storage = engine.Snapshot.Storages.GetAndChange(CreateStorageKey(Prefix_NextValidators), () => new StorageItem());
-            storage.Value = GetValidators(engine.Snapshot).ToByteArray();
+            storage.Set(GetValidators(engine.Snapshot));


I think it'd be nice to optimize this one also while we're here.

If you're to look at preview2 state change dump, you'll see something like

{ "block": 3, "size": 1, "storage": [ { "state": "Changed", "key": "ffffffff0e00000000000000000000000000000001", "value": "e807023e9b32ea89b94d066e649b124fd50e396ee91369e8e2a6ae1b11c170d022256d03009b7540e10f2562e5fd8fac9eaec25166a58b26e412348ff5a86927bfac22a202ba2c70f5996f357a43198705859fae2cfea13e1172962800772b3d588a9d4abd03408dcd416396f64783ac587ea1e1593c57d9fea880c8a6a1920e92a25947780602a7834be9b32e2981d157cb5bbd3acb42cfd11ea5c3b10224d7a44e98c5910f1b0214baf0ceea3a66f17e7e1e839ea25fd8bed6cd82e6bb6e68250189065f44ff01030205e9cefaea5a1dfc580af20c8d5aa2468bb0148f1a5e4605fc622c80e604ba00" } ] },

for every block and this value obviously doesn't change often. I guess it would be same even with an updated (post-preview2) governance system. But we shouldn't update storage at all if this value doesn't change, it's not hard to do, actually the only reason we didn't do it immediately in neo-go is that it would break state change dumps compatibility for us (even though technically storage contents would be the same for each block).

The problem is that TrackeableState is outside StorageItem, I know what you say, I will try to improve it in other PR.

src/neo/Ledger/StorageItem.cs

shargon · 2020-07-17T07:47:57Z

I don't know why coveralls still failing

shargon · 2020-07-17T08:00:41Z

Wait for #1775

* Allow call from native contract (#1700) * Fix MethodCallback (#1723) * Fix #1714 (#1715) * Add base64 SYSCALLs (#1717) * Neo.VM.3.0.0-CI00230 (#1725) * Allow to iterate buffer (#1726) * Buffer iterator * Rename * Remove using Co-authored-by: Erik Zhang <erik@neo.org> * Update StorageContext.cs (#1728) StorageContext needs to be public so it can be accessed by Neo Debugger Co-authored-by: Erik Zhang <erik@neo.org> * Fix return value check (#1730) Co-authored-by: Shargon <shargon@gmail.com> * Fix ContractEventDescriptor (#1733) * fix enumerator (#1744) Co-authored-by: Tommo-L <luchuan@neo.org> * Change json fields to all lower case for consistency (#1736) * Replace DataCache.Find by DataCache.Seek (#1740) * replace DataCache.Find by DataCache.Seek * fix order * optimize * Update ByteArrayComparer.cs * Update ByteArrayComparer.cs * Update DataCache.cs * fix comments * fix comments * fix comments * Update DataCache.cs * Update DataCache.cs * Reorder methods Co-authored-by: Tommo-L <luchuan@neo.org> Co-authored-by: erikzhang <erik@neo.org> Co-authored-by: Shargon <shargon@gmail.com> * Add MaxVerificationGas (#1745) Co-authored-by: Shargon <shargon@gmail.com> * Create KeyBuilder (#1748) * Speed up the initialization of ApplicationEngine. (#1749) * Change nef checksum to double SHA256 (#1751) * Change to double SHA256 * Clean code * Revert change * Clean code * Fix checksum * Update NefFile.cs * Fix compile * Clean code Clean code Fix checksum Update NefFile.cs Fix compile * Optimize * Optimize * Fix Co-authored-by: erikzhang <erik@neo.org> * Check witnesses on isStandard (#1754) * Check witness on isStandard * Add IsDeployed * Add IsPayable * Fix UT * format * Add coverage * Remove IsPayable, IsDeployed * Move NEP10 to manifest (#1729) * NEP10 abi * To lower case * Move to Manifest * Clean code * Update ContractManifest.cs * Update ContractManifest.cs * Fix native contracts Co-authored-by: Erik Zhang <erik@neo.org> * Capture fault Exception (#1761) * Capture faultException * Update Wallet error * Remove flag check * Clean code * Sender from signers (#1752) * Sender from signers * Remove co- * Move signers outside attributes * Fix UT and remove Signers class * Fix UT * Add FeeOnly scope * Remove orderBy * Remove _signersCache * Fix Signers * Fix WitnessScope * Fix Sender * Update TransactionAttributeType.cs * Update Wallet.cs * Fix Wallet * Rename * Update Wallet.cs * Update Wallet.cs * Partial UT fix * More UT fixes * Fix Sender's WitnessScope * Fix Wallet * Fix UT * Explicit FeeOnly for DeployNativeContracts * Same order as serialization * Test FeeOnly * dotnet format * format Co-authored-by: Erik Zhang <erik@neo.org> * IApplicationEngineProvider (#1758) * Remove the lock from SendersFeeMonitor and rename it to TransactionVerificationContext (#1756) * Add EffectiveVoterTurnout (#1762) * Remove AllowedTriggers from SYSCALLs (#1755) * fix validatorscount (#1770) Co-authored-by: Tommo-L <luchuan@neo.org> * workflows: use checkout action v2 (#1775) * Update git workflow * Update .github/workflows/main.yml Co-authored-by: Erik Zhang <erik@neo.org> Co-authored-by: Erik Zhang <erik@neo.org> * Add cache to native contract executions V2 (#1766) * cache v2 * More optimizations * Policy contract optimization * Remove interporable variable * Two more * Add Set * Optimizations * Optimize Co-authored-by: erikzhang <erik@neo.org> * Optimize attributes (#1774) * Add length before compression data (#1768) * Add length before compression * Optimize * Fix UT * Add UT * Add UT Co-authored-by: erikzhang <erik@neo.org> * Fix VerifyWitnesses (#1776) * Ensure non predictable peers (#1739) * Plugins from List to array * Move false to init * Fix UT * Refactor * Add Exception Message For CreateContract (#1787) * Add Exception Message When Create Contract * Add Exception Message * Code optimization * add UT Co-authored-by: Shargon <shargon@gmail.com> Co-authored-by: Qiao Jin <43407364+Qiao-Jin@users.noreply.github.com> Co-authored-by: Erik Zhang <erik@neo.org> Co-authored-by: Shargon <shargon@gmail.com> Co-authored-by: Harry Pierson <harrypierson@hotmail.com> Co-authored-by: Luchuan <luchuan@ngd.neo.org> Co-authored-by: Tommo-L <luchuan@neo.org> Co-authored-by: joeqian <qianzhuo@ngd.neo.org> Co-authored-by: cloud8little <34291844+cloud8little@users.noreply.github.com> Co-authored-by: Qiao Jin <43407364+Qiao-Jin@users.noreply.github.com>

cache v2

8e160f6

shargon requested a review from erikzhang July 14, 2020 17:13

shargon added 5 commits July 14, 2020 19:17

More optimizations

a6c0354

Policy contract optimization

3f5f07d

Remove interporable variable

a2cc20f

Two more

9f3a1b4

Add Set

554b44b

shargon mentioned this pull request Jul 14, 2020

Add cache to native contract executions #1764

Closed

shargon mentioned this pull request Jul 15, 2020

Optimize IInventory serialization #1767

Closed

roman-khimov reviewed Jul 15, 2020

View reviewed changes

erikzhang reviewed Jul 16, 2020

View reviewed changes

src/neo/Ledger/StorageItem.cs Outdated Show resolved Hide resolved

shargon and others added 3 commits July 16, 2020 10:44

Merge branch 'master' into native-cache-v2

92ab0f5

Optimizations

bcf453e

Optimize

62f1413

erikzhang approved these changes Jul 17, 2020

View reviewed changes

Merge branch 'master' into native-cache-v2

5da2463

Merge branch 'master' into native-cache-v2

95c3494

shargon merged commit da6be43 into neo-project:master Jul 17, 2020

shargon deleted the native-cache-v2 branch July 17, 2020 14:36

roman-khimov mentioned this pull request Aug 18, 2020

Optimize NextValidators writes #1850

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cache to native contract executions V2#1766

Add cache to native contract executions V2#1766
shargon merged 11 commits intoneo-project:masterfrom
shargon:native-cache-v2

shargon commented Jul 14, 2020 •

edited

Loading

Uh oh!

shargon commented Jul 14, 2020 •

edited

Loading

Uh oh!

Tommo-L commented Jul 15, 2020

Uh oh!

roman-khimov Jul 15, 2020

Uh oh!

shargon Jul 15, 2020

Uh oh!

Uh oh!

shargon commented Jul 17, 2020

Uh oh!

shargon commented Jul 17, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shargon commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shargon commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tommo-L commented Jul 15, 2020

Uh oh!

roman-khimov Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

shargon Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shargon commented Jul 17, 2020

Uh oh!

shargon commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shargon commented Jul 14, 2020 •

edited

Loading

shargon commented Jul 14, 2020 •

edited

Loading

shargon commented Jul 17, 2020 •

edited

Loading