Skip to content

Sequence table GC speed is slower than user write, and which have conflict when user write and gc #50194

@yangxuanjia

Description

@yangxuanjia

Describe the problem
I see GC use reverse iterator in newest version, it is wonderful for sequence table.
I merge this PR to v19.1, this very good, OOM never occur.
But, I found another problem, gc default timeout is 1 min, the replica enqueue 7-10 min per time.
sequence table have 1k-2k update per second, but GC can reduce sequence table MVCC key 500/s, run 1min can reduce 30k MVCC key. The reslut is GC is slower than user write when table use Sequence table.

MVCC GG Bytes Age | 7d 20h 53m 37s 450ms | 7d 20h 53m 37s 450ms | 7d 20h 53m 37s 450ms
-- | -- | -- | --
26.0 B / 1 count | 26.0 B / 1 count | 26.0 B / 1 count
454.2 MiB / 1 count | 454.2 MiB / 1 count | 454.2 MiB / 1 count
340.7 MiB / 39692528 count | 340.7 MiB / 39692528 count | 340.7 MiB / 39692528 count

I change the GC queue default to dynamic, hope to GC sequence MVCC key frequently.

func makeQueueGcTimeoutFunc() queueProcessTimeoutFunc {
	return func(cs *cluster.Settings, r replicaInQueue) time.Duration {
		minimumTimeout := queueGuaranteedProcessingTimeBudget.Get(&cs.SV)
		// NB: In production code this will type assertion will always succeed.
		// Some tests set up a fake implementation of replicaInQueue in which
		// case we fall back to the configured minimum timeout.
		repl, ok := r.(interface{ GetMVCCStats() enginepb.MVCCStats })
		if !ok {
			return minimumTimeout
		}
		stat := repl.GetMVCCStats()
		if stat.LiveCount == 1 && stat.KeyCount == 1 && stat.ValCount > 1 {
			timeout := time.Duration(stat.ValCount/500+1) * time.Second
			if timeout.Seconds() > 60 {
				return timeout
			}
		}
		return minimumTimeout
	}
}

But I found there have a conflict between GC Sequence table MVCC key and user write record when primary key is auto_increment, the insert performance is reduce too big when GC Sequence table MVCC key.
Is there any methods to avoid the conflict???

To Reproduce
table primary key is auto_increment, and use Sequence table.
table insert 60 million record.
Sequence table have 40 million MVCC values.
change GC sequence default timeout from 1min to 1hour.

Expected behavior
you will find insert performance is reduce too big when GC Sequence table MVCC key.

Additional data / screenshots
Screenshot from 2020-06-15 10-21-04
Screenshot from 2020-06-15 10-21-21

Environment:

  • CockroachDB version [e.g. 19.1.x]
  • Server OS: [e.g. Linux/Distrib]
  • Client app [e.g. cockroach sql, JDBC, ...]

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-storageRelating to our storage engine (Pebble) on-disk storage.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-communityOriginated from the community

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions