Skip to content

Tendermint throughput, can't achieve more than 2K.  #3002

@tulioalberton

Description

@tulioalberton

Hello guys, I hope this message finds you well.

I could not find another way to ask this question, but could you inform me about the configurations to achieve the throughput of 10K operations/second?

"speed: Tendermint blocks can commit to finality in the order of 1 second. Tendermint can handle transaction volume at the rate of 10,000 transactions per second for 250byte transactions. The bottleneck is in the application."

I am using four (4) persistent_peers but I can't reach the 10K throughput.
I barely achieve 2K, I am getting this throughput through tm-monitor.

I tried many variants of tm-bench command (with r=1000, r=10K, c=100, etc. ) through 6 machines:
./tm-bench -v -c 50 -T 60 -r 100 -s 300 192.168.2.29:26657 -broadcast-tx-method commit
Even send the tx to all endpoints, I can't reach more than 2k.

I believe that the batch is getting full because of the logs says during several outputs:
" I[1126-12-01|16:34:31.285] Executed block module=state height=992 validTxs=36514 invalidTxs=0
I[1126-12-01|16:34:31.299] Committed state module=state height=992 txs=36514 appHash=F8CF380000000000"

Thanks in advance!!!

##########################################################
################ Detailed machine configuration ################

I have four nodes with the configuration below running the persistent_peers:

Linux s9 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Stepping: 5
CPU MHz: 2261.099
BogoMIPS: 4521.86
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15

Each one has this config.toml:

#### This is a TOML config file.
#### For more information, see https://github.com/toml-lang/toml

### main base config options #####

#### TCP or UNIX socket address of the ABCI application,
#### or the name of an ABCI application compiled in with the Tendermint binary
proxy_app = "tcp://192.168.2.29:26658"

#### A custom human readable name for this node
moniker = "s9"

#### If this node is many blocks behind the tip of the chain, FastSync
#### allows them to catchup quickly by downloading blocks in parallel
#### and verifying their commits
fast_sync = true

#### Database backend: leveldb | memdb | cleveldb
db_backend = "leveldb"

#### Database directory
db_dir = "data"

#### Output level for logging, including package level options
log_level = "main:info,state:info,*:error"

#### Output format: 'plain' (colored text) or 'json'
log_format = "plain"

### additional base config options #####

#### Path to the JSON file containing the initial validator set and other meta data
genesis_file = "config/genesis.json"

#### Path to the JSON file containing the private key to use as a validator in the consensus protocol
priv_validator_file = "config/priv_validator.json"

#### TCP or UNIX socket address for Tendermint to listen on for
#### connections from an external PrivValidator process
priv_validator_laddr = ""

#### Path to the JSON file containing the private key to use for node authentication in the p2p protocol
node_key_file = "config/node_key.json"

#### Mechanism to connect to the ABCI application: socket | grpc
abci = "socket"

#### TCP or UNIX socket address for the profiling server to listen on
prof_laddr = ""

#### If true, query the ABCI app on connecting to a new peer
#### so the app can decide if we should keep the connection or not
filter_peers = false

### advanced configuration options #####

#### rpc server configuration options #####
[rpc]

#### TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://192.168.2.29:26657"

#### A list of origins a cross-domain request can be executed from
#### Default value '[]' disables cors support
#### Use '["*"]' to allow any origin
cors_allowed_origins = "[*]"

#### A list of methods the client is allowed to use with cross-domain requests
cors_allowed_methods = "[HEAD GET POST]"

#### A list of non simple headers the client is allowed to use with cross-domain requests
cors_allowed_headers = "[Origin Accept Content-Type X-Requested-With X-Server-Time]"

#### TCP or UNIX socket address for the gRPC server to listen on
#### NOTE: This server only supports /broadcast_tx_commit
grpc_laddr = ""

#### Maximum number of simultaneous connections.
#### Does not include RPC (HTTP&WebSocket) connections. See max_open_connections
#### If you want to accept more significant number than the default, make sure
#### you increase your OS limits.
#### 0 - unlimited.
#### Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
#### 1024 - 40 - 10 - 50 = 924 = ~900
grpc_max_open_connections = 1024

#### Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool
unsafe = false

#### Maximum number of simultaneous connections (including WebSocket).
#### Does not include gRPC connections. See grpc_max_open_connections
#### If you want to accept more significant number than the default, make sure
#### you increase your OS limits.
#### 0 - unlimited.
#### Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
#### 1024 - 40 - 10 - 50 = 924 = ~900
max_open_connections = 8192

### peer to peer configuration options #####
[p2p]

#### Address to listen for incoming connections
laddr = "tcp://192.168.2.29:26656"

#### Address to advertise to peers for them to dial
#### If empty, will use the same port as the laddr,
#### and will introspect on the listener or use UPnP
#### to figure out the address.
external_address = ""

#### Comma separated list of seed nodes to connect to
seeds = ""

#### Comma separated list of nodes to keep persistent connections to
persistent_peers = "33d312fe0853b9882227dc4238f46ae697ae47cf@192.168.2.30:26656,19cc59c4cfd17cf300cdef05cd86a9b6b3fbda0d@192.168.2.31:26656,87c0a035944c168af9b55d0f13eb22e724224fd2@192.168.2.32:26656"

#### UPNP port forwarding
upnp = false

#### Path to address book
addr_book_file = "config/addrbook.json"

#### Set true for strict address routability rules
#### Set false for private or local networks
addr_book_strict = false

#### Maximum number of inbound peers
max_num_inbound_peers = 500

#### Maximum number of outbound peers to connect to, excluding persistent peers
max_num_outbound_peers = 500

#### Time to wait before flushing messages out on the connection
flush_throttle_timeout = "100ms"

#### Maximum size of a message packet payload, in bytes
max_packet_msg_payload_size = 1024

#### Rate at which packets can be sent, in bytes/second
send_rate = 5120000

#### Rate at which packets can be received, in bytes/second
recv_rate = 5120000

#### Set true to enable the peer-exchange reactor
pex = true

#### Seed mode, in which node constantly crawls the network and looks for
#### peers. If another node asks it for addresses, it responds and disconnects.
#### Does not work if the peer-exchange reactor is disabled.
seed_mode = false

#### Comma separated list of peer IDs to keep private (will not be gossiped to other peers)
private_peer_ids = ""

#### Toggle to disable guard against peers connecting from the same ip.
allow_duplicate_ip = true

#### Peer connection configuration.
handshake_timeout = "20s"
dial_timeout = "10s"

### mempool configuration options #####
[mempool]

recheck = true
broadcast = true
wal_dir = ""

#### size of the mempool
size = 500000

#### size of the cache (used to filter transactions we saw earlier)
cache_size = 200000

### consensus configuration options #####
[consensus]

wal_file = "data/cs.wal/wal"

timeout_propose = "15s"
timeout_propose_delta = "5000ms"
timeout_prevote = "15s"
timeout_prevote_delta = "5000ms"
timeout_precommit = "15s"
timeout_precommit_delta = "5000ms"
timeout_commit = "30s"

#### Make progress as soon as we have all the precommits (as if TimeoutCommit = 0)
skip_timeout_commit = false

#### EmptyBlocks mode and possible interval between empty blocks
create_empty_blocks = true
create_empty_blocks_interval = "1s"

#### Reactor sleep duration parameters
peer_gossip_sleep_duration = "200ms"
peer_query_maj23_sleep_duration = "4s"

#### Block time parameters. Corresponds to the minimum time increment between consecutive blocks.
blocktime_iota = "1s"

### transactions indexer configuration options #####
[tx_index]

#### What indexer to use for transactions
#### Options:
####   1) "null" (default)
####   2) "kv" - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend).
indexer = "kv"

#### Comma-separated list of tags to index (by default the only tag is "tx.hash")
#### You can also index transactions by height by adding "tx.height" tag here.
#### It's recommended to index only a subset of tags due to possible memory
#### bloat. This is, of course, depends on the indexer's DB and the volume of
#### transactions.
index_tags = ""

#### When set to true, tells indexer to index all tags (predefined tags:
#### "tx.hash", "tx.height" and all tags from DeliverTx responses).
#### Note this may be not desirable (see the comment above). IndexTags has a
#### precedence over IndexAllTags (i.e. when given both, IndexTags will be
#### indexed).
index_all_tags = false

### instrumentation configuration options #####
[instrumentation]

#### When true, Prometheus metrics are served under /metrics on
#### PrometheusListenAddr.
#### Check out the documentation for the list of available metrics.
prometheus = false

#### Address to listen for Prometheus collector(s) connections
prometheus_listen_addr = ":26660"

#### Maximum number of simultaneous connections.
#### If you want to accept more significant number than the default, make sure
#### you increase your OS limits.
#### 0 - unlimited.
max_open_connections = 3

#### Instrumentation namespace
namespace = "tendermint"

Metadata

Metadata

Assignees

No one assigned

    Labels

    T:perfType: Performance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions