Opened #77607 to introduce some instability into the TxnIDResolution endpoint. Currently, three failure modes are simulated:

RPC returning errors (15% probability)
RPC hang (15% probability and it hangs anywhere between 5 seconds to 90 seconds)
Corrupted results (5% probability, randomly drop 10% of the result)

I have attached the screenshot of the metrics introduced in #77514 below along with the YCSB-A result.

Normally, when everything runs normally, the resolver queue sees <1k backlogged events (aggregated across all three nodes in the cluster used in the testing) pending resolution. As YCSB-A workload ramps up, the resolver generally is able to keep up with the incoming events in the queue.

However, when artificial instability is introduced, the backlogged queue's size becomes more volatile. At some point, there are >4k backlogged events sitting in the resolver queue. This is very likely due to two factors:

Resolver will retry in the failure mode 1 and 3. This is expected.
RPC hanging can cause the resolver to perform poorly, and it had a hard time keeping up with the incoming contention events.

But overall the result still looks good. The YCSB-A throughput was not impacted at all. The retry budget logic introduced in the resolverQueue did its job and dropped contention events that it deemed "unresolvable" and prevented the resolverQueue from being overwhelmed.

However, one things concerns me here is the mutex in the resolverQueue. When the resolver performs resolution, it locks the mutex for the entire duration of the resolveLocked() call. This means new contention events generated during the resolvedLocked() cannot be enqueued into the resolverQueue.
For the duration of my test, this is mostly fine, since the YCSB-A throughput is not affected much. This is indicating the locking is not causing too much immediate issues, since the eventStore was able to queue up the contention events at different levels through write buffer batch (size of 64) and the golang's buffered channel (24 batches).
This means the eventStore can queue up to of 24 * 64 = 1536 contention events while the resolverQueue is being locked up.

We have observed >4k contention events backlogged during the test across 3 nodes, that is, on average, ~1333 events per node queued up in the eventStore. This is coming pretty close to the maximum queuing capacity of the eventStore. I think if I keep tuning the failure 2's hanging duration to a longer value, it is possible that I can overload the eventStore. Though, I'm not sure how realistic that would be. But it's worth a shot.

What this is indicating is that, we might need a load shedding mechanism here to protect the eventStore (and its resolverQueue) from being overwhelmed. This can be as simple as a cluster setting that filters the contention events with small contention duration. (e.g. a low pass filter).

Of course, this can also be an active mechanism, where we can use the size of the queue as an input variable and the low pass filter as an control variable to build a feedback control loop. (dynamically increase the low pass filter value as the queue size increases, until things returns to normal.)

Appendix

Metrics using unstable binary

Metrics using a normal binary

YCSB-A results, using the same machine and the test configuration as https://roachperf.crdb.dev/?filter=&view=ycsb%2FA%2Fnodes%3D3%2Fcpu%3D32&tab=gce

name	elapsed	ops(total)	ops/sec(cum)	avg(ms)	p50(ms)	p95(ms)	p99(ms)	pMax(ms)	relative perf
baseline	900.0s	25418625	28242.9	5.1	2.5	10.0	71.3	906.0	100.00%
baseline	900.0s	24067158	26741.3	5.4	2.5	10.0	79.7	1040.2	94.68%
enabled-2s-resolution	900.0s	24384584	27094.0	5.3	2.5	9.4	75.5	1744.8	95.93%
enabled-1s-resolution	900.0s	22957011	25507.8	5.6	2.6	10.5	79.7	1409.3	90.32%
enabled-1s-resolution	900.0s	22912593	25458.4	5.7	2.8	10.5	79.7	1073.7	90.14%
enabled-0.5s-resolution	900.0s	23378704	25976.3	5.5	2.8	10.5	75.5	1140.9	91.97%
enabled-30s-resolution	900.0s	23676265	26307.0	5.5	2.4	9.4	79.7	1409.3	93.16%
enabled-30s-resolution	900.0s	24658157	27398.0	5.3	2.5	9.4	71.3	1140.9	97.01%
unstable-2s-resolution	900.0s	24115966	26795.5	5.4	2.6	10.5	71.3	77309.4	94.88%
unstable-2s-resolution	900.0s	23133771	25704.2	5.6	2.5	9.4	75.5	81604.4	91.09%

sql: stability testing with error injection for contention event store #75110

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions