Skip to content

[Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (#28916) #29199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 1 commit into from
Dec 28, 2023
Merged

[Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (#28916) #29199

xiaokang merged 1 commit into from
Dec 28, 2023

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Dec 28, 2023

Fix partition name NPE and sample for all table during auto analyze. Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later.

backport #28916

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Jibing-Li Jibing-Li marked this pull request as ready for review December 28, 2023 03:26
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Dec 28, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 28, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.73 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17164913377 Bytes

@Jibing-Li
[Fix](statistics) Fix partition name NPE and sample for all table dur…
a46662a 
…ing auto analyze (apache#28916)

Fix partition name NPE and sample for all table during auto analyze.
Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later.
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Dec 28, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 28, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.62 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17162688343 Bytes

@xiaokang xiaokang merged commit d2dc34e into apache:branch-2.0 Dec 28, 2023
@Jibing-Li Jibing-Li deleted the npe2.0 branch December 28, 2023 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Jibing-Li @doris-robot @xiaokang