Skip to content

kv: Add API to allow for split/scatter behavior in multi-tenant environments #74389

@ajstorm

Description

@ajstorm

Bulk jobs currently send AdminSplit and AdminScatter requests to KV when it thinks it has sent a lot of data to one range, before it keeps sending, to give kv a chance to make a new empty range and move it somewhere. Perhaps most importantly, it gives kv a chance to split that new range off and move it to a new node before the bulk job proceeds to go and send it a bunch more data, that would then all need to be rebalanced over if we instead waited for the range / allocator to decide it is over-full and split and move on its own. If we can get it split and moved before we fill it, it's much cheaper.

But we don't do any of this when a tenant is the one generating and sending data. Of course, deciding a range is too full and should be split, or a node has too much on it and should rebalance replicas away, is obviously kv/allocator work, but the bulk job has information that the range and allocator doesn't here, which is that it is going to be sending more data so it might be advantageous to split and rebalance it now, while it is cheap.

One idea that was proposed was to create an advisory request that bulk could send to KV to indicate when a proactive split may be beneficial. The thinking is that if bulk could send some indication that it's about to send a large number of sequential keys to a given sst, KV could use that info to determine if/when to split. See parent Epic for the full discussion.

Initially, for 22.1, however we want to just open the existing AdminSplit and AdminScatter APIs to tenants, while still blocking them at the SQL statement level, and add monitoring on number of spans per tenant, so existing code will work with party between tenant and dedicated clusters.

Epic: CRDB-10720

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-distributionRelating to rebalancing and leasing.A-multitenancyRelated to multi-tenancyC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV Team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions