Skip to content

Degraded performance of select queries after v22.8 #44500

@Anton-Rodionov

Description

@Anton-Rodionov

Describe the situation
There is still a performance drop between 22.7.7.24 -> 22.8.1.2097 (and up to the latest 22.12)

I'm running a trivial query "select count() from ... where counter_id = ... and ts > ... and ts < .. format Null settings local_filesystem_read_method='read'." (typical "hits" table).

The query pipeline is identical between the versions (22.7.7.24 and 22.8.1.2097). The slowness comes from "AggregatingTransform" step that's processing rows x3-4 times slower.

Trace logs:
22.7 - https://pastila.nl/?02aa875b/fd42b27162d53ca2a79b137e217b9ddd
22.8 - https://pastila.nl/?02aa875b/d0c917e0b0166c5f8dd3af2d1cc96035

The QPS drop is x2-3 times regardless of the concurrency level used in clickhouse-benchmark tool (same results for -c 1, 8, 16)

How to reproduce
I was not able to reproduce the issue when creating new tables and importing fake data. On the real machine I can upgrade/downgrade between the versions and clearly see the performance difference. I went to the point where one replica in the cluster is on 22.7.7.24 and another is 22.8.1.2097. Practically all queries are slower on 22.8 about x1.5-3 times.

Additional context
I checked all versions after 22.8 up to the latest 22.12, the issue persists.

Perf of 22.7
Screenshot 2022-12-21 at 21 54 44

Perf of 22.8
Screenshot 2022-12-21 at 21 51 51

The query in this case is x1.5 times slower.

Also, the code difference: v22.8.1.2097-lts...v22.7.7.24-stable

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions