Skip to content

sql: the Insert planNode runs everything in one humongous batch #16180

@andreimatei

Description

@andreimatei

Currently the tableWriter (and I assume also tableUpdater) does everything in one batch. This can lead to large Raft proposals (and it can run into the kv.raft.command.max_size limit). If the Insert is configured with autocommit, then the batch is first checked against the kv.transaction.max_intents limit.

As a fun fact, it's currently possible to not hit the max_size limit, but subsequently hit the kv.transaction.max_intents limit at commit time. I think this is not ideal since, if all the intents that EndTransaction needs to carry were produced by a single batch, and that batch was applied successfully, the EndTransaction would probably apply successfully too.

I think it'd be a significant improvement if the Insert node would chunk up its writes so that large batches and proposals are avoided.
If we do this, and if we allow parallel execution of batches, there might be sequencing concerns to keep in mind for the UPDATE case - if a single UPDATE stmt updates a row twice, is it specified which one has to win?

Also see #15713

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions