Skip to content

storage/engine: cache iterators within read-only batches#16406

Merged
petermattis merged 1 commit intocockroachdb:masterfrom
jonderry:cache-iter
Jun 13, 2017
Merged

storage/engine: cache iterators within read-only batches#16406
petermattis merged 1 commit intocockroachdb:masterfrom
jonderry:cache-iter

Conversation

@jonderry
Copy link
Copy Markdown
Contributor

@jonderry jonderry commented Jun 8, 2017

Read-only batches currently execute on a RocksDB which bypasses the
optimization to reuse iterators across execution of a command that
occurs when using a *rocksDBBatch. We create a minimalist wrapper for
RocksDB called rocksDBReadOnly that caches iterators and does not
have the overhead of state for supporting writes. Here are the results
of microbenchmarks that test the relative performance of iterating over
one key using the underlying engine, versus the read-write batch versus
the read only wrapper:

Engine vs. ReadOnly
name            old time/op  new time/op  delta
IterOn/10-8     1.15us       0.52us        -55.03%  (p=0.000 n=10+10)
IterOn/100-8    1.25us       0.67us        -46.61%  (p=0.000 n=9+10)
IterOn/1000-8   1.42us       0.81us        -43.41%  (p=0.000 n=9+10)
IterOn/10000-8  1.63us       0.96us        -40.83%  (p=0.000 n=10+10)

Batch vs. ReadOnly
name            old time/op  new time/op  delta
IterOn/10-8     568ns        517ns      -9.13%  (p=0.000 n=10+10)
IterOn/100-8    705ns        669ns      -5.04%  (p=0.005 n=10+10)
IterOn/1000-8   855ns        805ns      -5.88%  (p=0.000 n=10+10)
IterOn/10000-8  1.02us       0.96us     -5.60%  (p=0.000 n=9+10)

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Jun 8, 2017

CLA assistant check
All committers have signed the CLA.

@jonderry jonderry changed the title Cache iterators in within read-only batches Cache iterators within read-only batches Jun 8, 2017
@benesch
Copy link
Copy Markdown
Contributor

benesch commented Jun 8, 2017

(Don't mind me—edited the description to align the benchmark results.)

@petermattis
Copy link
Copy Markdown
Collaborator

Looks good overall. I think we want to do this even though the performance benefit over using a normal batch is small. It ensures that read-only operations are truly read-only. Some comments below, the only major one being that Close needs to destroy the iterators.


Review status: 0 of 6 files reviewed at latest revision, 4 unresolved discussions, some commit checks pending.


pkg/storage/replica.go, line 2285 at r1 (raw file):

	var result EvalResult
	rec := ReplicaEvalContext{r, spans}
	br, result, pErr = evaluateBatch(ctx, storagebase.CmdIDKey(""), r.store.Engine().NewReadOnly(), rec, nil, ba)

Per my other comment, we need to Close the read only structure in order to destroy the RocksDB iterators. This will probably look something like:

  readOnly := r.store.Engine().NewReadOnly()
  defer readOnly.Close()

pkg/storage/engine/batch_test.go, line 235 at r1 (raw file):

func TestReadOnlyBasics(t *testing.T) {
	defer leaktest.AfterTest(t)()
	testReadOnlyBasics(t)

There is no need for the indirection here. It is done for some of the other tests where we're passing in different values. Just inline the body of testReadOnlyBasics here.


pkg/storage/engine/engine.go, line 197 at r1 (raw file):

	// them atomically on a call to Commit().
	NewBatch() Batch
	NewReadOnly() ReadWriter

This method needs a documentation comment.


pkg/storage/engine/rocksdb.go, line 649 at r1 (raw file):

}

func (r *rocksDBReadOnly) Close() {}

We do need to destroy the iterators here. Something like:

	if i := &r.prefixIter.rocksDBIterator; i.iter != nil {
		i.destroy()
	}
	if i := &r.normalIter.rocksDBIterator; i.iter != nil {
		i.destroy()
	}

This also means we need to ensure we Close the ReadOnly in Replica.executeReadOnlyBatch.


Comments from Reviewable

@petermattis petermattis changed the title Cache iterators within read-only batches storage/engine: cache iterators within read-only batches Jun 8, 2017
@jonderry
Copy link
Copy Markdown
Contributor Author

jonderry commented Jun 8, 2017

Review status: 0 of 6 files reviewed at latest revision, 4 unresolved discussions, some commit checks pending.


pkg/storage/replica.go, line 2285 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Per my other comment, we need to Close the read only structure in order to destroy the RocksDB iterators. This will probably look something like:

  readOnly := r.store.Engine().NewReadOnly()
  defer readOnly.Close()

Done.


pkg/storage/engine/batch_test.go, line 235 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

There is no need for the indirection here. It is done for some of the other tests where we're passing in different values. Just inline the body of testReadOnlyBasics here.

Done.


pkg/storage/engine/engine.go, line 197 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

This method needs a documentation comment.

Done.


pkg/storage/engine/rocksdb.go, line 649 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

We do need to destroy the iterators here. Something like:

	if i := &r.prefixIter.rocksDBIterator; i.iter != nil {
		i.destroy()
	}
	if i := &r.normalIter.rocksDBIterator; i.iter != nil {
		i.destroy()
	}

This also means we need to ensure we Close the ReadOnly in Replica.executeReadOnlyBatch.

changed Close to do this as well as set a boolean flag.


Comments from Reviewable

@jonderry
Copy link
Copy Markdown
Contributor Author

jonderry commented Jun 8, 2017

Review status: 0 of 6 files reviewed at latest revision, 1 unresolved discussion, some commit checks pending.


pkg/storage/engine/rocksdb.go, line 649 at r1 (raw file):

Previously, jonderry wrote…

changed Close to do this as well as set a boolean flag.

Done.


Comments from Reviewable

@petermattis
Copy link
Copy Markdown
Collaborator

:lgtm:

One final bit is to tie the PR to the issue. Add the following to the end of the commit message:

Fixes #6803

Review status: 0 of 6 files reviewed at latest revision, all discussions resolved, some commit checks pending.


Comments from Reviewable

@petermattis
Copy link
Copy Markdown
Collaborator

@jonderry There are a few test failures (mostly style issues) that need to be fixed up before this can be merged.

Read-only batches currently execute on a RocksDB which bypasses the
optimization to reuse iterators across execution of a command that
occurs when using a *rocksDBBatch. We create a minimalist wrapper for
RocksDB called rocksDBReadOnly that caches iterators and does not
have the overhead of state for supporting writes. Here are the results
of microbenchmarks that test the relative performance of iterating over
one key using the underlying engine, versus the read-write batch versus
the read only wrapper:

Engine vs. ReadOnly
name            old time/op  new time/op  delta
IterOn/10-8     1.15us       0.51us       -55.60%  (p=0.000 n=10+9)
IterOn/100-8    1.25us       0.66us       -47.44%  (p=0.000 n=9+10)
IterOn/1000-8   1.42us       0.81us       -42.99%  (p=0.000 n=9+9)
IterOn/10000-8  1.63us       0.99us       -39.40%  (p=0.000 n=10+10)

Batch vs. ReadOnly
name            old time/op  new time/op  delta
IterOn/10-8     568ns        510ns       -10.27%  (p=0.000 n=10+9)
IterOn/100-8    705ns        659ns       -6.51%  (p=0.000 n=10+10)
IterOn/1000-8   855ns        811ns       -5.18%  (p=0.000 n=10+9)
IterOn/10000-8  1.02us       0.99us      -3.33%  (p=0.000 n=9+10)

Fixes cockroachdb#6803
@petermattis
Copy link
Copy Markdown
Collaborator

@jonderry I fixed the style issues. I'll merge this when green.

@petermattis petermattis merged commit 5662c6a into cockroachdb:master Jun 13, 2017
}

func (r *rocksDBReadOnly) NewTimeBoundIterator(start, end hlc.Timestamp) Iterator {
panic("not implemented")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function gets called from sqlccl.BenchmarkClusterEmptyIncrementalBackup, which now panics. @petermattis

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then it's likely this broke backups when enterprise.kv.timebound_iterator.enabled is true, too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants