test: add the loadtime tool by williambanfield · Pull Request #9342 · tendermint/tendermint

williambanfield · 2022-08-30T22:39:12Z

What does this pull request do?

This pull request adds the loadtime tool. This tool leverages the tm-load-test framework. Using the framework means that the only real logic that needs to be written is the logic for Tx generation. The framework does the rest.

The tool writes a set of metadata into the transaction, including the current transaction rate, number of connections, specified size of the transaction, and the current time.

Notes to reviewers

The payload is specified as proto and the generated definition is checked in. This has been done so that the tooling created for #9331 may leverage the same definition and not need to share or perform any custom logic when reading the metadata. proto also generates a reasonably compact wire representation, much more so than json, allowing more metadata to be able to fit within the 1kb target specified.

Questions to reviewers

It's a bit difficult to judge if this tool currently meets the requirements of #9330, since the documentation on the tm-load-test tool make it a bit confusing what the --rate flag represents. The documentation is as follows:

  -r, --rate int                        The number of transactions to generate each second on each connection, to each endpoint (default 1000)

However, it's a bit difficult to understand how this interacts with send-period

  -p, --send-period int                 The period (in seconds) at which to send batches of transactions (default 1)

does anyone know offhand?

Finally, the original issue lists a question regarding whether or not the data should be contained with the 'key' or the 'value' of the kvstore transaction. Currently, this code does not prepend a <key>=, which results in the transaction being used as both they key and the value in the store. Is the primary concern performance of the kv store?

closes: #9330

PR checklist

Tests written/updated, or no tests needed
CHANGELOG_PENDING.md updated, or no changelog entry needed
Updated relevant documentation (docs/) and code comments, or no
documentation updates needed

sergio-mena · 2022-08-31T08:24:44Z

Is the primary concern performance of the kv store?

In our conversation last week, the concern was that if we don't limit the keys used somehow, we'll end up with a gigantic state.

test/loadtime/Makefile

test/loadtime/main.go

sergio-mena

Cool job! I guess this means we can drop the ad-hoc modifications to the load test tool we did for testnet-testing v0.35.x

williambanfield · 2022-08-31T15:02:55Z

I've updated the transaction generation to prepend the tx with a= so that all transactions overwrite the same key in the kvstore. We don't care about storing these in the kvstore at all, what we care about is storing them in the block chain. Therefore, using the same key every time is fine.

sergio-mena · 2022-08-31T15:19:51Z

Therefore, using the same key every time is fine.

I agree, until a concrete use case justifies using more than one key

thanethomson · 2022-08-31T18:51:56Z

it's a bit difficult to understand how this interacts with send-period

The send period is an integer value whose smallest possible value is 1 (i.e. 1 second) that controls how frequently we send "bursts" of transactions at the transaction rate.

For instance, if you set the send period to 1s and rate to 100, the tool will try to send 100 transactions every 1 second. If you set the send period to 2s and rate to 100, the tool will try to send 100 transactions every 2 seconds.

With this you could simulate "bursty" traffic, by, for example, setting the send period to 60s and rate to 10,000. This would attempt to send a burst of 10k txs once every 60s.

If you want to see the mechanics of how this works under the hood, check out the sendLoop function.

In short, for these tests, simply set the send period to 1 🙂

I've updated the transaction generation to prepend the tx with a= so that all transactions overwrite the same key in the kvstore. We don't care about storing these in the kvstore at all, what we care about is storing them in the block chain. Therefore, using the same key every time is fine.

I'd be concerned that using this approach would potentially cause problems as discovered in #3865. I'd recommend that we rather cycle through a reasonable number of keys, e.g. using hex encoding to run from 0000 through FFFF or something.

williambanfield · 2022-08-31T20:05:05Z

I'd be concerned that using this approach would potentially cause problems as discovered in #3865.

I'm not sure I see how these are related. If a duplicate transaction can be resubmitted, then this can happen irrespective of if the kvstore key is the same or not. That behavior appears more to be a defect in the mempool and mempool cache / recheck interaction than in the store itself.

thanethomson · 2022-08-31T20:28:16Z

I'm not sure I see how these are related. If a duplicate transaction can be resubmitted, then this can happen irrespective of if the kvstore key is the same or not. That behavior appears more to be a defect in the mempool and mempool cache / recheck interaction than in the store itself.

Yeah I suppose I was mainly concerned with how duplicate transactions would be handled, but if we can ensure that at least the values of each key/value pair are unique that'd be fine too. Then we can definitely use the same key 🙂

thanethomson · 2022-09-01T00:39:58Z

Very cool, great work here @williambanfield! I look forward to seeing it in action. Have you tested it out against any running nodes yet?

Also, I think we should backport this to the v0.37.x branch, right?

williambanfield · 2022-09-01T13:57:48Z

@thanethomson I tested it locally against a running node and it seem to perform just fine.

Yes, I'll backport to v0.37 and v0.34

This pull request adds the loadtime tool. This tool leverages the tm-load-test framework. Using the framework means that the only real logic that needs to be written is the logic for Tx generation. The framework does the rest. The tool writes a set of metadata into the transaction, including the current transaction rate, number of connections, specified size of the transaction, and the current time.

* test: add the loadtime tool (#9342) This pull request adds the loadtime tool. This tool leverages the tm-load-test framework. Using the framework means that the only real logic that needs to be written is the logic for Tx generation. The framework does the rest. The tool writes a set of metadata into the transaction, including the current transaction rate, number of connections, specified size of the transaction, and the current time. * lint

…x-rc1 * release/v0.37.0-rc1: QA Process report for v0.37.x (and baseline for v0.34.x) (tendermint#9499) (tendermint#9577) Fix TX payload for DO testnets (tendermint#9540) (tendermint#9542) blocksync: retry requests after timeout (backport tendermint#9518) (tendermint#9533) Extend the load report tool to include transactions' hashes (tendermint#9509) (tendermint#9513) build(deps): Bump styfle/cancel-workflow-action from 0.10.0 to 0.10.1 (tendermint#9502) build(deps): Bump actions/stale from 5 to 6 (tendermint#9494) loadtime: add block time to the data point (tendermint#9484) (tendermint#9489) config: Add missing storage section when generating config (tendermint#9483) (tendermint#9487) Sync Vote.Verify() in spec with implementation (tendermint#9466) (tendermint#9476) fix spec (tendermint#9467) (tendermint#9469) metrics: fix panic because of absent prometheus label (tendermint#9455) (tendermint#9474) Ensure Dockerfile stages use consistent Go version (backport tendermint#9462) (tendermint#9472) build(deps): Bump slackapi/slack-github-action from 1.21.0 to 1.22.0 (tendermint#9432) build(deps): Bump bufbuild/buf-setup-action from 1.7.0 to 1.8.0 (tendermint#9453) state: restore previous error message (tendermint#9435) (tendermint#9440) build(deps): Bump gonum.org/v1/gonum from 0.11.0 to 0.12.0 (tendermint#9411) docs: Update ADRs for v0.37 (tendermint#9399) (tendermint#9418) build(deps): Bump github.com/spf13/viper from 1.12.0 to 1.13.0 (tendermint#9410) build(deps): Bump github.com/lib/pq from 1.10.6 to 1.10.7 (tendermint#9409) feat: support HTTPS inside websocket (tendermint#9416) (tendermint#9422) Removed unused param (tendermint#9394) test: generate uuid on startup for load tool (tendermint#9383) (tendermint#9392) add redirect links (tendermint#9385) (tendermint#9389) refactor: migrate to cosmos/gogoproto (backport tendermint#9356) (tendermint#9381) cmd: print all versions of tendermint and its sub protocols (tendermint#9329) (tendermint#9386) Add missing changes changelog files (backport tendermint#9376) (tendermint#9382) add separated runs by UUID (backport tendermint#9367) (tendermint#9379) spec: abci++ cleanup for v0.37 (backport tendermint#9288) (tendermint#9374) ci: Remove "(WARNING: BETA SOFTWARE)" tagline from all upcoming releases (tendermint#9371) (tendermint#9372) Update rpc client header (tendermint#9276) (tendermint#9349) ci: Pre-release workflows (backport tendermint#9366) (tendermint#9368) test: add the loadtime report tool (tendermint#9351) (tendermint#9364) Update Tendermint version to v0.37.0 (tendermint#9354) test: add the loadtime tool (tendermint#9342) (tendermint#9357) # Conflicts: # version/version.go

initial commit of loadtime

5df843a

williambanfield marked this pull request as ready for review August 30, 2022 22:39

williambanfield requested a review from ebuchman as a code owner August 30, 2022 22:39

williambanfield requested a review from a team August 30, 2022 22:39

sergio-mena reviewed Aug 31, 2022

View reviewed changes

test/loadtime/Makefile Outdated Show resolved Hide resolved

sergio-mena reviewed Aug 31, 2022

View reviewed changes

test/loadtime/main.go Outdated Show resolved Hide resolved

sergio-mena reviewed Aug 31, 2022

View reviewed changes

test/loadtime/main.go Outdated Show resolved Hide resolved

sergio-mena approved these changes Aug 31, 2022

View reviewed changes

williambanfield added 3 commits August 31, 2022 10:50

add payload calculation

17278b9

cr fixes

92492da

prepend kv store prefix to tx

122c6e9

williambanfield changed the title ~~tools: add the loadtime tool~~ test: add the loadtime tool Aug 31, 2022

Merge branch 'main' into wb/loadtime

d9c6010

williambanfield merged commit 70fd943 into main Aug 31, 2022

williambanfield deleted the wb/loadtime branch August 31, 2022 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add the loadtime tool#9342

test: add the loadtime tool#9342
williambanfield merged 5 commits intomainfrom
wb/loadtime

williambanfield commented Aug 30, 2022

Uh oh!

sergio-mena commented Aug 31, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergio-mena left a comment

Uh oh!

williambanfield commented Aug 31, 2022

Uh oh!

sergio-mena commented Aug 31, 2022

Uh oh!

thanethomson commented Aug 31, 2022

Uh oh!

williambanfield commented Aug 31, 2022

Uh oh!

thanethomson commented Aug 31, 2022

Uh oh!

thanethomson commented Sep 1, 2022

Uh oh!

williambanfield commented Sep 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

williambanfield commented Aug 30, 2022

What does this pull request do?

Notes to reviewers

Questions to reviewers

PR checklist

Uh oh!

sergio-mena commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergio-mena left a comment

Choose a reason for hiding this comment

Uh oh!

williambanfield commented Aug 31, 2022

Uh oh!

sergio-mena commented Aug 31, 2022

Uh oh!

thanethomson commented Aug 31, 2022

Uh oh!

williambanfield commented Aug 31, 2022

Uh oh!

thanethomson commented Aug 31, 2022

Uh oh!

thanethomson commented Sep 1, 2022

Uh oh!

williambanfield commented Sep 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sergio-mena commented Aug 31, 2022 •

edited

Loading