Skip to content

[Feature] (broker-load)support nereids load #24221

@wsjz

Description

@wsjz

Search before asking

  • I had searched in the issues and found no similar issues.

Description

(step1)
add load grammar: #23485

  • rename broker load to bulk load.
  • add load grammar to nereids optimizer.
  • convert to insert into clause with table value function.

(step2)
load job schedule #26356
Now support:

  1. load data by tvf insert into sql, but just for simple load
  2. show load stmt
  • job id, label name, job state, time info
  • simple progress
  1. cancel load from db
  2. support enable new load by Config.enable_nereids_load
  3. TODO:
  • support partition insert job
  • support show statistics from BE
  • need add ut case

(step3)

  • support txn for multi data desc load to insert
  • support multi location by union
  • support multiple task and collect task statistic
  • support transactional task
  • check file path by s3 fs list status, support pattern path

Use case

No response

Related issues

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stalekind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions