Skip to content

Fix TX payload for DO testnets#9540

Merged
sergio-mena merged 10 commits intomainfrom
sergio/9539-fix-testnet-payload
Oct 12, 2022
Merged

Fix TX payload for DO testnets#9540
sergio-mena merged 10 commits intomainfrom
sergio/9539-fix-testnet-payload

Conversation

@sergio-mena
Copy link
Contributor

Closes #9539

This PR changes the way the payload is encoded and decoded in the loadtime tool.
To make sure only one "=" character appears in the transaction's payload, we:

  • First we use protobufs to encode the Payload structure
  • Then we encode those bytes in (uppercase) hex

As this encoding is less efficient, we need to adjust the padding length in order to obtain the desired transaction length.

The tradeoff here is that the minimum transaction payload is doubled (the space required to store a transaction without padding); but we are not likely to hit this limit anytime soon.


PR checklist

  • Tests written/updated, or no tests needed
  • CHANGELOG_PENDING.md updated, or no changelog entry needed
  • Updated relevant documentation (docs/) and code comments, or no
    documentation updates needed

@sergio-mena sergio-mena requested a review from a team October 12, 2022 10:50
@sergio-mena sergio-mena self-assigned this Oct 12, 2022
@sergio-mena sergio-mena added S:backport-to-v0.34.x Tell mergify to backport the PR to v0.34.x S:backport-to-v0.37.x Tell mergify to backport the PR to v0.37.x labels Oct 12, 2022
Copy link
Contributor

@thanethomson thanethomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks @sergio-mena! I ran the code on this branch against a localnet and got:

Experiment ID: 8700df95-852e-407b-ac88-9b443ae82007

        Connections: 1
        Rate: 1000
        Size: 1024

        Total Valid Tx: 9000
        Total Negative Latencies: 0
        Minimum Latency: 577.637343ms
        Maximum Latency: 2.222080172s
        Average Latency: 1.232116008s
        Standard Deviation: 422.259864ms

Total Invalid Tx: 0

The stats output from the loadtime tool:

Parameter,Value,Units
total_time,10.001,seconds
total_txs,9000,count
total_bytes,9216000,bytes
avg_tx_rate,899.939669,transactions per second
avg_data_rate,921538.221460,bytes per second

Seems like every tx that was accepted by broadcast_tx_sync was found by the reporting tool.

Have you tested this with the E2E app yet? (i.e. in a testnet)

@sergio-mena
Copy link
Contributor Author

I ran the code on this branch against a localnet and got:

Thanks a lot for testing that!

Have you tested this with the E2E app yet? (i.e. in a testnet)

Yes... stingy as I am, it has one validator and one seed node :-D; and, with the code in this PR, 99% of the sent transactions were reported by both Prometheus and the reporting part of loadtime.

@sergio-mena sergio-mena merged commit b42c439 into main Oct 12, 2022
@sergio-mena sergio-mena deleted the sergio/9539-fix-testnet-payload branch October 12, 2022 17:58
mergify bot pushed a commit that referenced this pull request Oct 12, 2022
* Added print

* Fix unmarshall

* Fix unmarshalling

* Simplified steps to unmarshall

* minor

* Use 'encoding/hex'

* Forget about C, this is Go!

* gosec warning

* Set maximum payload size

* nosec annotation

(cherry picked from commit b42c439)
mergify bot pushed a commit that referenced this pull request Oct 12, 2022
* Added print

* Fix unmarshall

* Fix unmarshalling

* Simplified steps to unmarshall

* minor

* Use 'encoding/hex'

* Forget about C, this is Go!

* gosec warning

* Set maximum payload size

* nosec annotation

(cherry picked from commit b42c439)
sergio-mena added a commit that referenced this pull request Oct 12, 2022
* Added print

* Fix unmarshall

* Fix unmarshalling

* Simplified steps to unmarshall

* minor

* Use 'encoding/hex'

* Forget about C, this is Go!

* gosec warning

* Set maximum payload size

* nosec annotation

(cherry picked from commit b42c439)

Co-authored-by: Sergio Mena <sergio@informal.systems>
sergio-mena added a commit that referenced this pull request Oct 12, 2022
* Added print

* Fix unmarshall

* Fix unmarshalling

* Simplified steps to unmarshall

* minor

* Use 'encoding/hex'

* Forget about C, this is Go!

* gosec warning

* Set maximum payload size

* nosec annotation

(cherry picked from commit b42c439)

Co-authored-by: Sergio Mena <sergio@informal.systems>
james-chf added a commit to heliaxdev/tendermint that referenced this pull request Nov 25, 2022
…x-rc1

* release/v0.37.0-rc1:
  QA Process report for v0.37.x (and baseline for v0.34.x) (tendermint#9499) (tendermint#9577)
  Fix TX payload for DO testnets (tendermint#9540) (tendermint#9542)
  blocksync: retry requests after timeout (backport tendermint#9518) (tendermint#9533)
  Extend the load report tool to include transactions' hashes (tendermint#9509) (tendermint#9513)
  build(deps): Bump styfle/cancel-workflow-action from 0.10.0 to 0.10.1 (tendermint#9502)
  build(deps): Bump actions/stale from 5 to 6 (tendermint#9494)
  loadtime: add block time to the data point (tendermint#9484) (tendermint#9489)
  config: Add missing storage section when generating config (tendermint#9483) (tendermint#9487)
  Sync Vote.Verify() in spec with implementation (tendermint#9466) (tendermint#9476)
  fix spec (tendermint#9467) (tendermint#9469)
  metrics: fix panic because of absent prometheus label (tendermint#9455) (tendermint#9474)
  Ensure Dockerfile stages use consistent Go version (backport tendermint#9462) (tendermint#9472)
  build(deps): Bump slackapi/slack-github-action from 1.21.0 to 1.22.0 (tendermint#9432)
  build(deps): Bump bufbuild/buf-setup-action from 1.7.0 to 1.8.0 (tendermint#9453)
  state: restore previous error message (tendermint#9435) (tendermint#9440)
  build(deps): Bump gonum.org/v1/gonum from 0.11.0 to 0.12.0 (tendermint#9411)
  docs: Update ADRs for v0.37 (tendermint#9399) (tendermint#9418)
  build(deps): Bump github.com/spf13/viper from 1.12.0 to 1.13.0 (tendermint#9410)
  build(deps): Bump github.com/lib/pq from 1.10.6 to 1.10.7 (tendermint#9409)
  feat: support HTTPS inside websocket (tendermint#9416) (tendermint#9422)
  Removed unused param (tendermint#9394)
  test: generate uuid on startup for load tool (tendermint#9383) (tendermint#9392)
  add redirect links (tendermint#9385) (tendermint#9389)
  refactor: migrate to cosmos/gogoproto (backport tendermint#9356) (tendermint#9381)
  cmd: print all versions of tendermint and its sub protocols  (tendermint#9329) (tendermint#9386)
  Add missing changes changelog files (backport tendermint#9376) (tendermint#9382)
  add separated runs by UUID (backport tendermint#9367) (tendermint#9379)
  spec: abci++ cleanup for v0.37 (backport tendermint#9288) (tendermint#9374)
  ci: Remove "(WARNING: BETA SOFTWARE)" tagline from all upcoming releases (tendermint#9371) (tendermint#9372)
  Update rpc client header (tendermint#9276) (tendermint#9349)
  ci: Pre-release workflows (backport tendermint#9366) (tendermint#9368)
  test: add the loadtime report tool (tendermint#9351) (tendermint#9364)
  Update Tendermint version to v0.37.0 (tendermint#9354)
  test: add the loadtime tool (tendermint#9342) (tendermint#9357)

# Conflicts:
#	version/version.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S:backport-to-v0.34.x Tell mergify to backport the PR to v0.34.x S:backport-to-v0.37.x Tell mergify to backport the PR to v0.37.x

Projects

Status: Done/Merged

Development

Successfully merging this pull request may close these issues.

Most transactions in DO testnets get refused

2 participants