This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

81062: storage,kvserver: Improve SST collision checking for wide SSTs r=sumeerbhola,nicktrav a=itsbilal

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in #73514
and then reverted later).
2) Write bloom filters in L6 by default, to help speed up 1.
Also thread a new IterOption to only optionally read these
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes #80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.

This change includes 4 changes to significantly improve
the performance of CheckSSTCollisions (and by extension,
AddSSTable) for cases where we add very wide sstables
relative to the engine:

1) Check if we're adding an sstable that has a small number of
keys, or has a 100x greater overlap with the engine relative
to its own size. In that case, switch to doing prefix seeks
inside CheckSSTCollisions, similar to what we did in cockroachdb#73514
and then reverted later).
2) Thread a new IterOption to only optionally read L6
filter blocks in prefix iteration, defaulting to not reading
them. This iterator option hooks into the one added in
Pebble in cockroachdb/pebble#1685 ).
3) Add a ParseHook to allow command-line pebble options to
allow configuring the writing of bloom filters. This allows
L6 filter block writing to be turned on through a command
line argument.
4) Update intentInterleavingIter to not prefix-seek the
intent/lock table iterator if we are in prefix seek mode and
the MVCC iterator returned nothing.

Fixes cockroachdb#80980.

Release note (performance improvement): Significantly improve
performance of IMPORTs when the source is producing data
not sorted by the destination table's primary key, especially
if the destination table has a very large primary key with
lots of columns.