Skip to content

importccl: verify number of columns during IMPORT PGDUMP#27345

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
madelynnblue:import-copy-cols
Jul 12, 2018
Merged

importccl: verify number of columns during IMPORT PGDUMP#27345
craig[bot] merged 1 commit intocockroachdb:masterfrom
madelynnblue:import-copy-cols

Conversation

@madelynnblue
Copy link
Copy Markdown
Contributor

Also make error messages more consistent between PGDUMP and PGCOPY.

Release note (bug fix): Correctly verify number of COPY columns during
IMPORT PGDUMP.

Also make error messages more consistent between PGDUMP and PGCOPY.

Release note (bug fix): Correctly verify number of COPY columns during
IMPORT PGDUMP.
@madelynnblue madelynnblue requested review from a team and dt July 10, 2018 19:11
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@madelynnblue
Copy link
Copy Markdown
Contributor Author

bors r+

craig bot pushed a commit that referenced this pull request Jul 12, 2018
27341: importccl: add oversample option to configure oversampling r=mjibson a=mjibson

On clusters with many nodes and smallish disks, doing an IMPORT that is
within an order of magnitude of the free space of the disks can lead to
disk fullness. This happens because the sampling algorithm (by design)
has a relatively high standard deviation in its error rate. We currently
target split points at a few hundred megs, but the standard deviation
on sampling means that a single node could easily be a few of those
away from the target mean, resulting in overscheduling data to a node
during shuffle.

Introduce an oversample option that can be set to some higher number. This
reduces the standard deviation of the error, resulting in each node having
more similar portion of the data, but does not have a major impact on
the rest of the performance.

Release note (sql change): Add an `oversample` WITH option to IMPORT to
decrease variance in data distributionduring processing.

27345: importccl: verify number of columns during IMPORT PGDUMP r=mjibson a=mjibson

Also make error messages more consistent between PGDUMP and PGCOPY.

Release note (bug fix): Correctly verify number of COPY columns during
IMPORT PGDUMP.

27438: mkrelease: statically link windows release binaries r=mberhault a=benesch

This got lost in 38899a8. Static linking is necessary to bundle MinGW-only
libraries into the Windows binary. The binary is otherwise only
executable from within a MinGW environment.

Fix #27435.

Release note: None

---

At this point I deserve to win an award for most broken refactor.

Co-authored-by: Matt Jibson <matt.jibson@gmail.com>
Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 12, 2018

Build succeeded

@craig craig bot merged commit 749a7b2 into cockroachdb:master Jul 12, 2018
@madelynnblue madelynnblue deleted the import-copy-cols branch July 12, 2018 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants