A basic benchmark against the e2e simple network with load 50-75 txs/block before getting degraded. Eventually, the network is only able to handle 15-20 txs/block...not sure why, but I doubt its p2p related. From this, we see that reading off of peer queues before sending on the TCP connection takes the most time.

Codecov Report

Merging #6126 (fc97795) into master (2ceb816) will decrease coverage by 0.00%.
The diff coverage is 72.47%.

@@            Coverage Diff             @@
##           master    #6126      +/-   ##
==========================================
- Coverage   60.76%   60.76%   -0.01%     
==========================================
  Files         279      281       +2     
  Lines       26321    26614     +293     
==========================================
+ Hits        15993    16171     +178     
- Misses       8651     8760     +109     
- Partials     1677     1683       +6     

@erikgrinaker would love your thoughts on the following.

I've spent quite some time researching network and process queuing disciplines and their various tradeoffs. I have no means come close to understanding all the disciplines and their tradeoffs, nor do I really want to spend too much time on doing so.

I've arrived at different options that we can take from here given the nature our now legacy p2p stack (i.e. priorities) for outbound message to peers:

1. Each peer's queue has a simple priority heap as the queue, where the capacity is fixed and envelope priority in the heap is determined by the corresponding channel priority. When the queue is full, messages are dropped. Or,
2. Use (weighted?) deficit round robin [DRR]. This is an adaptation of GPS and is relatively simple to understand and implement. Each peer would have a queue per flow (i.e. channel) and we would either pick a fixed quantum per flow or perhaps a quantum per flow based on that channel's priority. When any given queue is full, messages are dropped.

I definitely agree that we should start out with something simple. Both of these approaches seem viable, but I think I'd prefer DRR over a priority heap -- the heap always picks a higher priority, which can cause high-priority reactors to completely starve low-priority ones, and I think it makes sense to let low-priority reactors at least send some messages during contention.

I was going to suggest WFQ, but DWRR appears roughly equivalent and takes into account payload sizes as well, which is useful since we're scheduling at message granularity and a 10MB state sync chunk shouldn't count the same as a 1KB consensus vote. The plan was originally to schedule at the byte level, but this would require the stream-based transport API which has been deferred -- when/if this is implemented, we should rework the scheduling for this. Also note that because of the large message size variance, queue capacities should probably be in terms of bytes rather than number of messages (or possibly both), using proto.Size() to calculate message sizes.

You may want to have a look at the MConnection queue scheduler as well, which uses priority combined with the time since last packet send:

Thanks for the input @erikgrinaker. I believe I have enough info to get started on this now.

@erikgrinaker I was hoping you could spare a few minutes and take a look at the queue implementation. Specifically in regards to weights, priorities and the individual flow buffer sizes.

I'm having difficulties mapping old priorities and max message sizes to the new queue logic. In particular I see idle block production slow down to ~10s no matter what values I pick.

EDIT: I only see good idle or with load block times when I make it "fair", i.e. same quantum and buffer capacity for all channels/flows. Either my implementation is wrong or picking quantums based on "priority" is going to be very difficult.

Do we need it to be weighted?

So with a few tweaks, I've been able to get good consistent results, even with load, although the TPS tappers off drastically in terms of the # of txs included in blocks:

Tweaks made:

• Single synchronization goroutine (i.e. no mutex).
  • This requires large buffered enqueue and dequeue channels (e.g. 1000) so enqueueing doesn't hold up the dequeue loop and vice versa.
• Single shared buffer that all flows share with a fixed capacity (used 1MB in tests -- should probably be larger?).
• Quanta are based on average message sizes multiplied by priority.

So I've committed the most up-to-date implementation of the queue. I want to leave this PR in draft as-is to get more collaboration on it. The current summary is as follows:

• A single goroutine scheduler is used in the WDRR scheduler/queue. See the XXX/TODO here. It's not clear what kind of contention this causes or if its the ideal approach. Alternatively we could explore separate goroutines with a mutex.
• With the current values and implementation, a bare bones simple e2e network runs fine, but I notice that for the first few blocks (e.g. 5-10), 1/4 nodes lags. After block 10 or so, it runs pretty normal with avg block time of ~2.5s and all nodes in unison.

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.