e2e: test runner generates loadtime formatted transactions. by williambanfield · Pull Request #9779 · tendermint/tendermint

williambanfield · 2022-11-28T21:33:18Z

This pull request updates the e2e test to use the loadtime transaction format and updates the configuration to allow for parity with the loadtime load generation.

Added configuration parameters

load_tx_size_bytes: This size, in bytes, of each generated transaction. Defaults to 1024 bytes.
load_tx_batch_size: How many transactions to generate every 1 second.
load_tx_connections: How many connections to make to each node.
node.no_send_load: Whether or not to send load to this node.

Difference with `loadtime` tool

Note that there is a slight difference between this configuration scheme and the scheme used in the loadtime tool. The loadtime tool generates batch_size per connection per second. This code generates a total of load_tx_batch_size total every second. To mimic the behavior of the loadtime tool simply multiply by the total number of connections. This was done to maintain the legacy behavior of the tool which generated a total network-wide connection load instead of a per-connection load.

Replicating the v0.37.x load test.

The v0.37.x load tests, as outlined in methods.md, opened a set of connections to a single node instead of to all nodes on the network. The e2e test previously opened a set of connections to all nodes on the network and this default behavior was maintained by this change. To replicate the the v0.37.x load test, set all nodes but one to have a no_send_load parameter of true, setting no_send_load = true on the one node that should receive load. Connection number can be configured exactly as was done in v0.37.x, and batch size should be set to the product of the number of connections with the original batch size. For example, if v0.37.x's load test had a batch size of 200 and a connection number of 4, the batch size should instead be set to 800.

closes: #9778

PR checklist

Tests written/updated, or no tests needed
CHANGELOG_PENDING.md updated, or no changelog entry needed
Updated relevant documentation (docs/) and code comments, or no
documentation updates needed

sergio-mena

Nice work

sergio-mena · 2022-11-30T10:18:30Z

test/e2e/networks/simple.toml

 [node.validator03]
 [node.validator04]

+


[Minor] Needed?

Thought this was removed. Fixed.

sergio-mena · 2022-11-30T10:28:54Z

test/e2e/runner/load.go

+				return
+			}
+		}
+		t.Reset(time.Second)


Not sure about this, but this interval was configurable in tm-load-test. Do we want to do something similar here?

(I'm imagining we may want to increase it to test big TX bursts)

If my comment makes sense but the answer is "not needed now, can do later", I'm OK with that

Yeah, "not needed now, can do later" was ultimately what I was thinking. The current e2e test only uses 1 second as the interval for sending load and the load testing done for v0.37 also only used 1 second. If we want to expand this capability in the future we certainly can but I'm not sure it's necessary to add until we're sure we're going to use it since it would ultimately be very simple to add.

sergio-mena · 2022-11-30T10:33:39Z

test/e2e/runner/load.go

-		case chTx <- tx:
-			time.Sleep(time.Second / time.Duration(multiplier))
-
+		case <-t.C:


Do we want here a mechanism similar to this one in tm-load-test, to avoid the possibility that the batches pile up?

thanethomson

Would it not be preferable to replace the load generation mechanism in the E2E tests with the loadtime tool directly?

williambanfield · 2022-11-30T16:14:10Z

Would it not be preferable to replace the load generation mechanism in the E2E tests with the loadtime tool directly?

I don't believe so. It would be more complicated than preferred. It would either require pulling the code in directly or shelling out to the binary. In the case of pulling the code in, the tm-load-test tool expects to be run as an external process and registers a set of flags for setting the rate and size. Using the code directly would require changing these to also have code API equivalents.
Shelling out to the binary requires adding steps to all uses of the e2e framework to also install the loadtime tool which seems a bit cumbersome.
Both of these options are certainly possible but the task of sending transaction load to a set of servers is ultimately not particularly complex so pulling in and managing this other library or tool for that task seems like overkill.

thanethomson · 2022-11-30T16:49:41Z

In the case of pulling the code in, the tm-load-test tool expects to be run as an external process and registers a set of flags for setting the rate and size. Using the code directly would require changing these to also have code API equivalents.

Technically the CLI scaffolding provided by tm-load-test just calls ExecuteStandalone: https://github.com/informalsystems/tm-load-test/blob/a517a0ac52f5c26e0b1f6a480ff871ee70edba03/pkg/loadtest/loadtest.go#L10

This is the extent of the wiring required to call the tm-load-test machinery programmatically: https://github.com/informalsystems/tm-load-test/blob/a517a0ac52f5c26e0b1f6a480ff871ee70edba03/pkg/loadtest/cli.go#L41-L49

The only thing I can think of that would make using tm-load-test a bit of a pain here is the fact that it doesn't allow you to override its logging right now.

thanethomson

My comments re: tm-load-test aren't critical for this PR. We could consider perhaps reusing it as a library at some point in future.

(cherry picked from commit 21b2801)

(cherry picked from commit 21b2801) # Conflicts: # test/e2e/pkg/manifest.go # test/e2e/pkg/testnet.go

…9799) (cherry picked from commit 21b2801) Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

…9779) (#9800) * e2e: test runner generates loadtime formatted transactions. (#9779) (cherry picked from commit 21b2801) # Conflicts: # test/e2e/pkg/manifest.go # test/e2e/pkg/testnet.go * fixup merge conflicts Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com> Co-authored-by: William Banfield <wbanfield@gmail.com>

williambanfield added 12 commits November 28, 2022 15:58

add configuration fields

3c83afd

builds with load params shimmed into place

0441369

load generator per client

54a6a9c

add configuration for if load should be sent to this node

a9166db

make a client for every node that is indicated to be sent to

1eb4358

one load generator for all processes

5473613

rename load params and add defaults

3833ddd

honor batch size

47c377d

default batch size of 2

d6f6e2b

use defaultConnections variable

857821a

remove whitespace in testnet toml

539ce4f

whitespace

852e816

williambanfield marked this pull request as ready for review November 29, 2022 16:51

williambanfield requested a review from ebuchman as a code owner November 29, 2022 16:51

williambanfield requested a review from a team November 29, 2022 16:51

williambanfield requested review from adizere and lasarojc as code owners November 29, 2022 16:51

sergio-mena approved these changes Nov 30, 2022

View reviewed changes

thanethomson reviewed Nov 30, 2022

View reviewed changes

williambanfield and others added 4 commits November 30, 2022 11:38

allow batch to exit if next batch is about to begin.

d875711

add worker pool to createTxBatch

abeef3a

whitespace

10be61c

Merge branch 'main' into wb/e2e-runner-loadtime

e426359

thanethomson approved these changes Nov 30, 2022

View reviewed changes

williambanfield self-assigned this Nov 30, 2022

williambanfield added S:backport-to-v0.34.x Tell mergify to backport the PR to v0.34.x S:backport-to-v0.37.x Tell mergify to backport the PR to v0.37.x labels Nov 30, 2022

williambanfield merged commit 21b2801 into main Nov 30, 2022

williambanfield deleted the wb/e2e-runner-loadtime branch November 30, 2022 18:36

mergify bot mentioned this pull request Nov 30, 2022

e2e: test runner generates loadtime formatted transactions. (backport #9779) #9799

Merged

mergify bot pushed a commit that referenced this pull request Nov 30, 2022

e2e: test runner generates loadtime formatted transactions. (#9779)

9daa9fe

(cherry picked from commit 21b2801)

mergify bot mentioned this pull request Nov 30, 2022

e2e: test runner generates loadtime formatted transactions. (backport #9779) #9800

Merged

mergify bot pushed a commit that referenced this pull request Nov 30, 2022

e2e: test runner generates loadtime formatted transactions. (#9779)

379a23f

(cherry picked from commit 21b2801) # Conflicts: # test/e2e/pkg/manifest.go # test/e2e/pkg/testnet.go

williambanfield added a commit that referenced this pull request Nov 30, 2022

e2e: test runner generates loadtime formatted transactions. (#9779) (#…

cacc2f4

…9799) (cherry picked from commit 21b2801) Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>

moul mentioned this pull request Mar 6, 2025

dev/moul/docs ci #10143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e: test runner generates loadtime formatted transactions.#9779

e2e: test runner generates loadtime formatted transactions.#9779
williambanfield merged 16 commits intomainfrom
wb/e2e-runner-loadtime

williambanfield commented Nov 28, 2022 •

edited

Loading

Uh oh!

sergio-mena left a comment

Uh oh!

sergio-mena Nov 30, 2022

Uh oh!

williambanfield Nov 30, 2022

Uh oh!

sergio-mena Nov 30, 2022

Uh oh!

williambanfield Nov 30, 2022

Uh oh!

sergio-mena Nov 30, 2022

Uh oh!

williambanfield Nov 30, 2022

Uh oh!

thanethomson left a comment

Uh oh!

williambanfield commented Nov 30, 2022

Uh oh!

thanethomson commented Nov 30, 2022

Uh oh!

thanethomson left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

williambanfield commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Added configuration parameters

Difference with loadtime tool

Replicating the v0.37.x load test.

PR checklist

Uh oh!

sergio-mena left a comment

Choose a reason for hiding this comment

Uh oh!

sergio-mena Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

williambanfield Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

sergio-mena Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

williambanfield Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

sergio-mena Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

williambanfield Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

thanethomson left a comment

Choose a reason for hiding this comment

Uh oh!

williambanfield commented Nov 30, 2022

Uh oh!

thanethomson commented Nov 30, 2022

Uh oh!

thanethomson left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

williambanfield commented Nov 28, 2022 •

edited

Loading

Difference with `loadtime` tool