Enable Partition Discovery for Broker Load #1569
Conversation
…#1517) Doris support deploy multi BE on one host. So when allocating BE for replicas of a tablet, we should select different host. But there is a bug in tablet scheduler that same host may be selected for one tablet. This patch will fix this problem. There are some places related to this problem: 1. Create Table There is no bug in Create Table process. 2. Tablet Scheduler Fixed when selecting BE for REPLICA_MISSING and REPLICA_RELOCATING. Fixed when balance the tablet. 3. Colocate Table Balancer Fixed when selecting BE for repairing colocate backend sequence. Not fix in colocate group balance. Leave it to colocate repairing. 4. Tablet report Tablet report may add replica to catalog. But I did not check the host here, Tablet Scheduler will fix it.
|
Firstly, thanks for your improvement. I have some questions about this PR.
|
@imay
|
The TabletQuorumFailedException will be thrown in commitTxn while the success replica num of tablet is less then quorom replica num. The Hadoop load does not handle this exception because the push task will retry it later. The streaming broker, insert, stream and mini load will catch this exception and abort the txn after that.
|
Add Partition Discover as a feature of Broker Load. For usage of this feature see diffs of related docs. |
| 1. 不指定Partition Discovery的基础路径（BASE_PATH） | ||
| LOAD LABEL example_db.label10 | ||
| ( | ||
| DATA INFILE("hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir") |
There was a problem hiding this comment.
这样直接指定目录就可以导入，跟我们之前的默认行为是有冲突的。有的时候用户写错了，可能会将整个目录导入。
为了避免这种情况，可以不使用DATA INFILE，可是使用类似DATA INDIR的方式来指定导入一个目录
| INTO TABLE `my_table` | ||
| FORMAT AS "csv" | ||
| BASE_PATH AS "hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/" | ||
| (k1, k2, k3, utc_date，city) |
There was a problem hiding this comment.
这里columns表示的一般是文件中包含的列名，如果加上utc_date, city可能会让用户感到困惑。
建议不要再这个地方声明partition列名
| DATA INFILE("hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/city=beijing/utc_date=2019-06-26") | ||
| INTO TABLE `my_table` | ||
| FORMAT AS "csv" | ||
| BASE_PATH AS "hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/" |
There was a problem hiding this comment.
是不是可以增加partition column 声明，这样能够让用户清晰的知道自己在做什么。
@yuanlihan |
@imay |
|
@imay I have added an issue about this Enable Partition Discovery for Broker Load |
When creating table with OLAP engine, use can specify multi parition columns.
eg:
PARTITION BY RANGE(`date`, `id`)
(
PARTITION `p201701_1000` VALUES LESS THAN ("2017-02-01", "1000"),
PARTITION `p201702_2000` VALUES LESS THAN ("2017-03-01", "2000"),
PARTITION `p201703_all` VALUES LESS THAN ("2017-04-01")
)
Notice that load by hadoop cluster does not support multi parition column table.
We create a new segment format for BetaRowset. New format merge data file and index file into one file. And we create a new format for short key index. In origin code index is stored in format like RowCusor which is not efficient to compare. Now we encode multiple column into binary, and we assure that this binary is sorted same with the key columns.
Help document collation (integration of help and documentation documents)
commit f5d6201 Author: morrySnow <101034200+morrySnow@users.noreply.github.com> Date: Wed Mar 22 10:53:15 2023 +0800 [fix](planner) should always execute projection plan (apache#17885) 1. should always execute projection plan, whatever the statement it is. 2. should always execute projection plan, since we only have vectorized engine now commit ad61c84 Author: mch_ucchi <41606806+sohardforaname@users.noreply.github.com> Date: Mon Mar 27 17:50:52 2023 +0800 [fix](planner) fix conjunct planned on exchange node (apache#18042) sql like: select k5, k6, SUM(k3) AS k3 from ( select k5, date_format(k6, '%Y-%m-%d') as k6, count(distinct k3) as k3 from t group by k5, k6 ) AS temp where 1=1 group by k5, k6; will throw exception since conjuncts planned on exchange node, because exchange node cannot handle conjuncts, now we skip exchange node when planning conjuncts, which fixes the bug. notice: the bug occurs iff the conjunct is always true like 1=1 above.
Currently, we support loading data from parquet file, but can not parse partition columns in the path of parquet file and can not recursively list all files under base path of input.
This patch is able to discover and infer partitioning information under the base path of input like in Spark. It recursively list all the files under the base path and parse partition columns base on the base path if needed.
This patch parse partition columns at BrokerScanNode.java and save parsing result of each file path as a property of TBrokerRangeDesc, then parquet_reader of BE can read the value of specified partition column.