importccl: Added row limit option for importing bundle formats#56587
importccl: Added row limit option for importing bundle formats#56587craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
|
This change is |
b3f16b4 to
34b690a
Compare
e1f4e91 to
ffa5da5
Compare
adityamaru
left a comment
There was a problem hiding this comment.
Reviewed 1 of 8 files at r1, 1 of 4 files at r2.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @adityamaru and @mokaixu)
pkg/ccl/importccl/import_stmt.go, line 538 at r1 (raw file):
} format.Format = roachpb.IOFileFormat_Mysqldump
nit: can we undo this change
pkg/ccl/importccl/import_stmt.go, line 549 at r1 (raw file):
format.MysqlDump.RowLimit = int64(rowLimit) }
nit: ditto
pkg/ccl/importccl/import_stmt_test.go, line 1598 at r2 (raw file):
Quoted 4 lines of code…
res := sqlDB.QueryStr(t, "SELECT * FROM second") if actualRowCount := len(res); expectedRowLimit != actualRowCount { t.Fatalf("expected %d, got %d", expectedRowLimit, actualRowCount) }
I think you can do something like:
var numRows int
sqlDB.QueryRow("SELECT count(*) FROM second").Scan(&numRows)
require.Equal(t, expectedRowLimit, numRows)
Here and elsewhere in the tests.
pkg/roachpb/io-formats.proto, line 110 at r1 (raw file):
CSV file
I think you need to update the comment.
pkg/roachpb/io-formats.proto, line 117 at r1 (raw file):
// Indicates the number of rows to import per CSV file. // Must be a non-zero positive number.
ditto as above
mokaixu
left a comment
There was a problem hiding this comment.
Reviewable status:
pkg/ccl/importccl/import_stmt.go, line 538 at r1 (raw file):
Previously, adityamaru (Aditya Maru) wrote…
nit: can we undo this change
like separate the mysqldump logic? If I undo this change, all the mysqldump imports with rowlimits fail
pkg/ccl/importccl/import_stmt_test.go, line 1598 at r2 (raw file):
Previously, adityamaru (Aditya Maru) wrote…
res := sqlDB.QueryStr(t, "SELECT * FROM second") if actualRowCount := len(res); expectedRowLimit != actualRowCount { t.Fatalf("expected %d, got %d", expectedRowLimit, actualRowCount) }I think you can do something like:
var numRows int
sqlDB.QueryRow("SELECT count(*) FROM second").Scan(&numRows)
require.Equal(t, expectedRowLimit, numRows)Here and elsewhere in the tests.
done
pkg/roachpb/io-formats.proto, line 110 at r1 (raw file):
Previously, adityamaru (Aditya Maru) wrote…
CSV fileI think you need to update the comment.
oops! done changed to per table
pkg/roachpb/io-formats.proto, line 117 at r1 (raw file):
Previously, adityamaru (Aditya Maru) wrote…
// Indicates the number of rows to import per CSV file. // Must be a non-zero positive number.ditto as above
done, changed to per table
85b53a1 to
e941479
Compare
|
This LGTM! Feel free to merge over the weekend once CI is green 🥳 |
77a3071 to
73db851
Compare
adityamaru
left a comment
There was a problem hiding this comment.
Reviewed 1 of 8 files at r1, 1 of 4 files at r2, 2 of 4 files at r3, 4 of 4 files at r4.
Reviewable status:
|
bors r=adityamaru |
|
Build failed: |
A row limit option allows users to specify a fixed number of rows
that they want to import. This functionality is only available for
CSV/DELIMITED/AVRO formats.
With this code change, users can limit the number of rows they want
to import for PGDUMP and MYSQLDUMP bundle data formats. That is, they can
specify: `WITH row_limit="{$num}"` to only import $num rows.
Release note (sql change): Added WITH row_limit="{$num}" option for importing
bundle formats to allow users to do a quick test run on an import of $num rows.
Ex. IMPORT ... WITH row_limit="3";
|
bors r=adityamaru |
|
Build succeeded: |
A row limit option allows users to specify a fixed number of rows
that they want to import. This functionality is only available for
CSV/DELIMITED/AVRO formats.
With this code change, users can limit the number of rows they want
to import for PGDUMP and MYSQLDUMP bundle data formats. That is, they can
specify:
WITH row_limit="{$num}"to only import $num rows.Release note (sql change): Added WITH row_limit="{$num}" option for importing
bundle formats to allow users to do a quick test run on an import of $num rows.
Ex. IMPORT ... WITH row_limit="3";
Resolves: #26493