Add bitmap agg type and udaf #1610

kangkaisen · 2019-08-08T08:27:19Z

The change for this PR:

1 Add a bitmap_union agg type

2 Add a bitmap udaf

3 Add a bitmap_inin function, convert a int value to bitmap

4 Remove aligned_free method in port.h, because which is conflict with RoaringBitmap portability.h.

5 Remove merge method from aggregate_func, becuase all agg input is aggregated intermediate data. So the merge method and update method we only need one.

6 Remove unused Field constructor method

7 Use the Field polymorphism eliminate if-else.

8 why change the AggregateInfo::init api?

because key column and none agg value column no update semantic，so we need fill the dst with src by init method

How to use:

1 creata table with bitmap_union:

CREATE TABLE `bitmap_test` (
  `id` int(11) NULL COMMENT "",
  `id2` varchar(100) bitmap_count NULL
) ENGINE=OLAP
AGGREGATE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 10;

2 load the data by stream load:

seq 10 13 | awk '{OFS="\t"}{print $1, $1 * 10}' | curl --location-trusted -u xxx:xxx  -T - http://hostname:8410/api/test/bitmap_test/_stream_load

3 query:

select bitmap_union(id2) from bitmap_test;

select id, bitmap_union(id2) from bitmap_test group by id ;

Todo:

add document for bitmap_union
add test for bitmap_union
rewrite count distinct to bitmap_union for bitmap_union agg type column
bitmap_union support insert into select
bitmap_union support broker load
avoid serialize and deserialize from Scannode to Aggnode
add a new type for hll_union and bitmap_union

fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java

fe/src/main/java/org/apache/doris/analysis/CreateTableStmt.java

kangkaisen · 2019-08-18T09:13:18Z

This update contains the following change:

1 Rebase the code
2 Add UT for bitmap function
3 Add doc about bitmap
4 Rewrite count distinct to bitmap_union for bitmap_union agg type
5 Add a version for bitmap serialize result

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap_union.md

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md

be/src/olap/row.h

kangkaisen · 2019-08-20T08:14:26Z

This update contains the following change:

1 Release row_cursor arena memory immediately
2 Move bitmap functions from AggregateFunctions to BitmapFunctions
3 Add BITMAP_COUNT function, rename bitmap_init function to TO_BITMAP, rename bitmap function to BITMAP_UNION_INT
4 Update the doc

be/src/exprs/bitmap_function.cpp

be/src/olap/row.h

be/src/olap/row_cursor.cpp

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md

be/src/olap/schema.h

be/src/exec/olap_scanner.cpp

kangkaisen · 2019-08-22T11:00:49Z

After tested with real data, I found the bitmap load is very slower than sum. For about one million rows, stream load with sum only need 3 seconds, but stream load with bitmap need 60 seconds.

So I did a special improve for empty bitmap and single int bitmap, After the improvement, stream load with bitmap only need 4 seconds.

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md

be/src/util/bitmap.h

fe/src/main/java/org/apache/doris/catalog/Type.java

fe/src/main/java/org/apache/doris/analysis/AggregateInfoBase.java

be/src/olap/row_cursor.cpp

be/src/olap/schema.cpp

kangkaisen · 2019-08-23T05:09:40Z

Update for @imay comment.

kangkaisen · 2019-08-23T07:12:09Z

Rebase the code and Fix UT.

be/src/olap/row_cursor.cpp

be/src/olap/field.h

be/src/olap/wrapper_field.cpp

imay

LGTM

JackyYangPassion · 2020-01-06T02:59:53Z

when use the case, there is some error, version 0.11
ERROR 1064 (HY000): Syntax error at: CREATE TABLE bitmap_test( idint(11) NULL COMMENT "ID", id2 varchar(100) bitmap_count NULL ) ENGINE=OLAP AGGREGATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 10 ^ Encountered: IDENTIFIER Expected: COMMA

kangkaisen · 2020-01-06T03:33:53Z

when use the case, there is some error, version 0.11
ERROR 1064 (HY000): Syntax error at: CREATE TABLE bitmap_test( idint(11) NULL COMMENT "ID", id2 varchar(100) bitmap_count NULL ) ENGINE=OLAP AGGREGATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 10 ^ Encountered: IDENTIFIER Expected: COMMA

please refer to the bitmap doc:
https://github.com/apache/incubator-doris/blob/master/docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md

JackyYangPassion · 2020-01-06T07:37:00Z

I use mysql client with the bitmap doc, but there is error
Encountered: IDENTIFIER
create table sql :
CREATE TABLE pv_bitmap(dtint(11) NULL COMMENT "",pagevarchar(10) NULL COMMENT "",user_id bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(dt, page) COMMENT "OLAP" DISTRIBUTED BY HASH(dt) BUCKETS 2;

if there need some set in client?

Server version: 5.1.0 Doris version DORIS-0.11.14-release

JackyYangPassion · 2020-01-06T07:52:35Z

fe log
`Encountered: IDENTIFIER
Expected

at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:362) ~[palo-fe.jar:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:213) [palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:168) [palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:259) [palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:393) [palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.loop(ConnectProcessor.java:403) [palo-fe.jar:?]
at org.apache.doris.qe.ConnectScheduler$LoopHandler.run(ConnectScheduler.java:172) [palo-fe.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_211]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]

Caused by: org.apache.doris.common.AnalysisException: Syntax error
at org.apache.doris.analysis.SqlParser.unrecovered_syntax_error(SqlParser.java:1459) ~[palo-fe.jar:?]
at java_cup.runtime.lr_parser.parse(lr_parser.java:616) ~[jflex-1.4.3.jar:?]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:350) ~[palo-fe.jar:?]
... 11 more`

imay · 2020-01-06T08:37:00Z

I use mysql client with the bitmap doc, but there is error
Encountered: IDENTIFIER
create table sql :
CREATE TABLE pv_bitmap(dtint(11) NULL COMMENT "",pagevarchar(10) NULL COMMENT "",user_id bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(dt, page) COMMENT "OLAP" DISTRIBUTED BY HASH(dt) BUCKETS 2;

if there need some set in client?

Server version: 5.1.0 Doris version DORIS-0.11.14-release

you can try

`user_id` varchar(0) BITMAP_UNION NULL COMMENT ""

JackyYangPassion · 2020-01-06T10:28:09Z

I use mysql client with the bitmap doc, but there is error
Encountered: IDENTIFIER
create table sql :
CREATE TABLE pv_bitmap(dtint(11) NULL COMMENT "",pagevarchar(10) NULL COMMENT "",user_id bitmap BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(dt, page) COMMENT "OLAP" DISTRIBUTED BY HASH(dt) BUCKETS 2;
if there need some set in client?
Server version: 5.1.0 Doris version DORIS-0.11.14-release

you can try
`user_id` varchar(0) BITMAP_UNION NULL COMMENT ""

when i create table with sql it works
CREATE TABLE bitmap_test(idint(11) NULL COMMENT "",id2 varchar(100) BITMAP_UNION NULL COMMENT "" ) ENGINE=OLAP AGGREGATE KEY(id) COMMENT "OLAP" DISTRIBUTED BY HASH(id) BUCKETS 10

but when insert value there is some error

insert into bitmap_test values (10,bitmap_hash('a100'));

ERROR 1064 (HY000): No matching function with signature: bitmap_hash(varchar(-1)).

imay · 2020-01-06T10:34:57Z

@JackyYangPassion
The version don't include the bitmap_hash function. You can try it with our latest master. Which needs evn-1.2 docker image.

JackyYangPassion · 2020-01-06T13:37:03Z

@JackyYangPassion
The version don't include the bitmap_hash function. You can try it with our latest master. Which needs evn-1.2 docker image.
3Q!
use master it https://github.com/apache/incubator-doris/blob/master/docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md works

…e ori_pred (apache#1610)

morningman reviewed Aug 8, 2019

View reviewed changes

fe/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java Outdated Show resolved Hide resolved

fe/src/main/java/org/apache/doris/analysis/CreateTableStmt.java Outdated Show resolved Hide resolved

kangkaisen force-pushed the bitmap-union branch from d86fe4a to f3cab93 Compare August 18, 2019 09:08

imay reviewed Aug 19, 2019

View reviewed changes

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap_union.md Outdated Show resolved Hide resolved

docs/documentation/cn/sql-reference/sql-functions/aggregate-functions/bitmap.md Outdated Show resolved Hide resolved

be/src/olap/row.h Outdated Show resolved Hide resolved

imay reviewed Aug 20, 2019

View reviewed changes

imay reviewed Aug 23, 2019

View reviewed changes

kangkaisen force-pushed the bitmap-union branch from 9dd5a3b to e8d4422 Compare August 23, 2019 07:11

imay reviewed Aug 23, 2019

View reviewed changes

be/src/olap/row_cursor.cpp Outdated Show resolved Hide resolved

be/src/olap/field.h Outdated Show resolved Hide resolved

kangkaisen force-pushed the bitmap-union branch from 335da0f to cd7e973 Compare August 24, 2019 04:27

kangkaisen added 3 commits August 24, 2019 12:45

Add bitmap agg type and udaf

e141e74

update

485d59c

Fix build

a28bb83

kangkaisen force-pushed the bitmap-union branch from cd7e973 to a28bb83 Compare August 24, 2019 05:09

imay reviewed Aug 26, 2019

View reviewed changes

be/src/olap/wrapper_field.cpp Show resolved Hide resolved

Update WrapperField::create_by_type

ddb99ec

imay approved these changes Aug 26, 2019

View reviewed changes

imay merged commit 1e4dd77 into apache:master Aug 26, 2019

kangkaisen mentioned this pull request Aug 29, 2019

Make bitmap_union agg column support insert into and broker load #1721

Merged

kangkaisen mentioned this pull request Sep 6, 2019

Encapsulate HLL logic #1756

Merged

kangkaisen mentioned this pull request Sep 24, 2019

Avoid SerDe for aggregation query with object pool #1854

Merged

imay mentioned this pull request Sep 26, 2019

Release Notes 0.11.0 #1891

Closed

kangkaisen mentioned this pull request Oct 10, 2019

Add object type #1948

Merged

swjtu-zhanglei pushed a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jul 25, 2023

[Fix](inverted index) Check nullptr for range predicate high/low valu…

e2004f9

…e ori_pred (apache#1610)

Add bitmap agg type and udaf #1610

Add bitmap agg type and udaf #1610

Uh oh!

Conversation

kangkaisen commented Aug 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangkaisen commented Aug 18, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangkaisen commented Aug 20, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangkaisen commented Aug 22, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangkaisen commented Aug 23, 2019

Uh oh!

kangkaisen commented Aug 23, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

imay left a comment

Choose a reason for hiding this comment

Uh oh!

JackyYangPassion commented Jan 6, 2020

Uh oh!

kangkaisen commented Jan 6, 2020

Uh oh!

JackyYangPassion commented Jan 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JackyYangPassion commented Jan 6, 2020

Uh oh!

imay commented Jan 6, 2020

Uh oh!

JackyYangPassion commented Jan 6, 2020

Uh oh!

imay commented Jan 6, 2020

Uh oh!

JackyYangPassion commented Jan 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kangkaisen commented Aug 8, 2019 •

edited

Loading

JackyYangPassion commented Jan 6, 2020 •

edited

Loading

JackyYangPassion commented Jan 6, 2020 •

edited

Loading