-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Sequence table GC speed is slower than user write, and which have conflict when user write and gc #50194
Description
Describe the problem
I see GC use reverse iterator in newest version, it is wonderful for sequence table.
I merge this PR to v19.1, this very good, OOM never occur.
But, I found another problem, gc default timeout is 1 min, the replica enqueue 7-10 min per time.
sequence table have 1k-2k update per second, but GC can reduce sequence table MVCC key 500/s, run 1min can reduce 30k MVCC key. The reslut is GC is slower than user write when table use Sequence table.
MVCC GG Bytes Age | 7d 20h 53m 37s 450ms | 7d 20h 53m 37s 450ms | 7d 20h 53m 37s 450ms
-- | -- | -- | --
26.0 B / 1 count | 26.0 B / 1 count | 26.0 B / 1 count
454.2 MiB / 1 count | 454.2 MiB / 1 count | 454.2 MiB / 1 count
340.7 MiB / 39692528 count | 340.7 MiB / 39692528 count | 340.7 MiB / 39692528 count
I change the GC queue default to dynamic, hope to GC sequence MVCC key frequently.
func makeQueueGcTimeoutFunc() queueProcessTimeoutFunc {
return func(cs *cluster.Settings, r replicaInQueue) time.Duration {
minimumTimeout := queueGuaranteedProcessingTimeBudget.Get(&cs.SV)
// NB: In production code this will type assertion will always succeed.
// Some tests set up a fake implementation of replicaInQueue in which
// case we fall back to the configured minimum timeout.
repl, ok := r.(interface{ GetMVCCStats() enginepb.MVCCStats })
if !ok {
return minimumTimeout
}
stat := repl.GetMVCCStats()
if stat.LiveCount == 1 && stat.KeyCount == 1 && stat.ValCount > 1 {
timeout := time.Duration(stat.ValCount/500+1) * time.Second
if timeout.Seconds() > 60 {
return timeout
}
}
return minimumTimeout
}
}
But I found there have a conflict between GC Sequence table MVCC key and user write record when primary key is auto_increment, the insert performance is reduce too big when GC Sequence table MVCC key.
Is there any methods to avoid the conflict???
To Reproduce
table primary key is auto_increment, and use Sequence table.
table insert 60 million record.
Sequence table have 40 million MVCC values.
change GC sequence default timeout from 1min to 1hour.
Expected behavior
you will find insert performance is reduce too big when GC Sequence table MVCC key.
Environment:
- CockroachDB version [e.g. 19.1.x]
- Server OS: [e.g. Linux/Distrib]
- Client app [e.g.
cockroach sql, JDBC, ...]

