importccl: verify number of columns during IMPORT PGDUMP#27345
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Jul 12, 2018
madelynnblue:import-copy-cols
Merged
importccl: verify number of columns during IMPORT PGDUMP#27345craig[bot] merged 1 commit intocockroachdb:masterfrom madelynnblue:import-copy-cols
craig[bot] merged 1 commit intocockroachdb:masterfrom
madelynnblue:import-copy-cols
Conversation
Also make error messages more consistent between PGDUMP and PGCOPY. Release note (bug fix): Correctly verify number of COPY columns during IMPORT PGDUMP.
Member
dt
approved these changes
Jul 12, 2018
Contributor
Author
|
bors r+ |
craig bot
pushed a commit
that referenced
this pull request
Jul 12, 2018
27341: importccl: add oversample option to configure oversampling r=mjibson a=mjibson On clusters with many nodes and smallish disks, doing an IMPORT that is within an order of magnitude of the free space of the disks can lead to disk fullness. This happens because the sampling algorithm (by design) has a relatively high standard deviation in its error rate. We currently target split points at a few hundred megs, but the standard deviation on sampling means that a single node could easily be a few of those away from the target mean, resulting in overscheduling data to a node during shuffle. Introduce an oversample option that can be set to some higher number. This reduces the standard deviation of the error, resulting in each node having more similar portion of the data, but does not have a major impact on the rest of the performance. Release note (sql change): Add an `oversample` WITH option to IMPORT to decrease variance in data distributionduring processing. 27345: importccl: verify number of columns during IMPORT PGDUMP r=mjibson a=mjibson Also make error messages more consistent between PGDUMP and PGCOPY. Release note (bug fix): Correctly verify number of COPY columns during IMPORT PGDUMP. 27438: mkrelease: statically link windows release binaries r=mberhault a=benesch This got lost in 38899a8. Static linking is necessary to bundle MinGW-only libraries into the Windows binary. The binary is otherwise only executable from within a MinGW environment. Fix #27435. Release note: None --- At this point I deserve to win an award for most broken refactor. Co-authored-by: Matt Jibson <matt.jibson@gmail.com> Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
Contributor
Build succeeded |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also make error messages more consistent between PGDUMP and PGCOPY.
Release note (bug fix): Correctly verify number of COPY columns during
IMPORT PGDUMP.