Performance degradation caused by redundant pack reads during building bitmap filter for stable only #9817
Description
Bug Report
When I did some performance tests, I found that the query time was significantly slower than before.
For example, when enabling hash join v2, the execution time of TPCH 100 Q9 increased from 3.8 seconds to 4.7 seconds.
mysql> explain analyze select nation, o_year, sum(amount) as sum_profit from ( select n_name as nation, extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount from part, supplier, lineitem, partsupp, orders, nation where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey = l_orderkey and s_nationkey = n_nationkey and p_name like '%dim%' ) as profit group by nation, o_year order by nation, o_year desc;
+-----------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+-----------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| Sort_79 | 2406.00 | 175 | root | | time:4.73s, open:7.99ms, close:15µs, loops:2, RU:772290.169591 | tpch100.nation.n_name, Column#51:desc | 17.5 KB | 0 Bytes |
| └─TableReader_275 | 2406.00 | 175 | root | | time:4.73s, open:7.99ms, close:13.2µs, loops:2, cop_task: {num: 23, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00} | MppVersion: 2, data:ExchangeSender_274 | 6.86 KB | N/A |
| └─ExchangeSender_274 | 2406.00 | 175 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:4.72s, avg: 4.72s, p80:4.72s, p95:4.72s, iters:21, tasks:2, threads:72} | ExchangeType: PassThrough | N/A | N/A |
| └─Projection_82 | 2406.00 | 175 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:4.72s, avg: 4.72s, p80:4.72s, p95:4.72s, iters:21, tasks:2, threads:72} | tpch100.nation.n_name, Column#51, Column#53 | N/A | N/A |
| └─Projection_270 | 2406.00 | 175 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:4.72s, avg: 4.72s, p80:4.72s, p95:4.72s, iters:21, tasks:2, threads:72} | Column#53, tpch100.nation.n_name, Column#51 | N/A | N/A |
| └─HashAgg_271 | 2406.00 | 175 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:4.72s, avg: 4.72s, p80:4.72s, p95:4.72s, iters:21, tasks:2, threads:72} | group by:Column#64, tpch100.nation.n_name, funcs:sum(Column#65)->Column#53, funcs:firstrow(tpch100.nation.n_name)->tpch100.nation.n_name, funcs:firstrow(Column#64)->Column#51, stream_count: 36 | N/A | N/A |
| └─ExchangeReceiver_273 | 2406.00 | 350 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:4.72s, avg: 4.72s, p80:4.72s, p95:4.72s, iters:21, tasks:2, threads:72} | stream_count: 36 | N/A | N/A |
| └─ExchangeSender_272 | 2406.00 | 350 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:0s, avg: 2.36s, p80:4.72s, p95:4.72s, iters:2, tasks:2, threads:2} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.nation.n_name, collate: utf8mb4_bin], stream_count: 36 | N/A | N/A |
| └─HashAgg_268 | 2406.00 | 350 | mpp[tiflash] | | tiflash_task:{proc max:4.72s, min:0s, avg: 2.36s, p80:4.72s, p95:4.72s, iters:2, tasks:2, threads:2} | group by:Column#69, Column#70, funcs:sum(Column#68)->Column#65 | N/A | N/A |
| └─Projection_278 | 59788256.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.7s, min:0s, avg: 2.35s, p80:4.7s, p95:4.7s, iters:4017, tasks:2, threads:72} | minus(mul(tpch100.lineitem.l_extendedprice, minus(1, tpch100.lineitem.l_discount)), mul(tpch100.partsupp.ps_supplycost, tpch100.lineitem.l_quantity))->Column#68, tpch100.nation.n_name->Column#69, extract(YEAR, tpch100.orders.o_orderdate)->Column#70 | N/A | N/A |
| └─Projection_257 | 59788256.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.68s, min:0s, avg: 2.34s, p80:4.68s, p95:4.68s, iters:4017, tasks:2, threads:72} | tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.orders.o_orderdate, tpch100.nation.n_name | N/A | N/A |
| └─Projection_247 | 59788256.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.68s, min:0s, avg: 2.34s, p80:4.68s, p95:4.68s, iters:4017, tasks:2, threads:72} | tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.orders.o_orderdate, tpch100.nation.n_name, tpch100.orders.o_orderkey | N/A | N/A |
| └─HashJoin_246 | 59788256.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.68s, min:0s, avg: 2.34s, p80:4.68s, p95:4.68s, iters:4089, tasks:2, threads:72} | inner join, equal:[eq(tpch100.lineitem.l_suppkey, tpch100.supplier.s_suppkey)] | N/A | N/A |
| ├─ExchangeReceiver_122(Build) | 999999.00 | 2000000 | mpp[tiflash] | | tiflash_task:{proc max:284.1ms, min:0s, avg: 142ms, p80:284.1ms, p95:284.1ms, iters:244, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 28ms} | | N/A | N/A |
| │ └─ExchangeSender_121 | 999999.00 | 1000000 | mpp[tiflash] | | tiflash_task:{proc max:239.5ms, min:0s, avg: 119.7ms, p80:239.5ms, p95:239.5ms, iters:123, tasks:2, threads:36} | ExchangeType: Broadcast, Compression: FAST | N/A | N/A |
| │ └─Projection_120 | 999999.00 | 1000000 | mpp[tiflash] | | tiflash_task:{proc max:186.5ms, min:0s, avg: 93.2ms, p80:186.5ms, p95:186.5ms, iters:123, tasks:2, threads:36} | tpch100.supplier.s_suppkey, tpch100.nation.n_name | N/A | N/A |
| │ └─HashJoin_115 | 999999.00 | 1000000 | mpp[tiflash] | | tiflash_task:{proc max:186.5ms, min:0s, avg: 93.2ms, p80:186.5ms, p95:186.5ms, iters:124, tasks:2, threads:36} | inner join, equal:[eq(tpch100.supplier.s_nationkey, tpch100.nation.n_nationkey)] | N/A | N/A |
| │ ├─ExchangeReceiver_119(Build) | 25.00 | 25 | mpp[tiflash] | | tiflash_task:{proc max:24.5ms, min:0s, avg: 12.2ms, p80:24.5ms, p95:24.5ms, iters:1, tasks:2, threads:36}, tiflash_wait: {pipeline_queue_wait: 5ms} | | N/A | N/A |
| │ │ └─ExchangeSender_118 | 25.00 | 25 | mpp[tiflash] | | tiflash_task:{proc max:8.73ms, min:0s, avg: 4.37ms, p80:8.73ms, p95:8.73ms, iters:1, tasks:2, threads:36} | ExchangeType: Broadcast, Compression: FAST | N/A | N/A |
| │ │ └─TableFullScan_117 | 25.00 | 25 | mpp[tiflash] | table:nation | tiflash_task:{proc max:8.73ms, min:0s, avg: 4.37ms, p80:8.73ms, p95:8.73ms, iters:1, tasks:2, threads:36}, tiflash_scan:{mvcc_input_rows:25, mvcc_input_bytes:425, mvcc_output_rows:25, lm_skip_rows:0, local_regions:1, remote_regions:0, tot_learner_read:3ms, region_balance:{instance_num: 1, max/min: 1/1=1.000000}, delta_rows:25, delta_bytes:3334, segments:1, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:0ms, tot_build_inputstream:0ms, min_local_stream:0ms, max_local_stream:3ms, dtfile:{data_scanned_rows:25, data_skipped_rows:0, mvcc_scanned_rows:25, mvcc_skipped_rows:0, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:0ms, tot_read:0ms}} | keep order:false | N/A | N/A |
| │ └─TableFullScan_116(Probe) | 1000000.00 | 1000000 | mpp[tiflash] | table:supplier | tiflash_task:{proc max:185.5ms, min:0s, avg: 92.7ms, p80:185.5ms, p95:185.5ms, iters:16, tasks:2, threads:36}, tiflash_wait: {pipeline_breaker_wait: 31ms, pipeline_queue_wait: 148ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:1, remote_regions:0, tot_learner_read:2ms, region_balance:{instance_num: 1, max/min: 1/1=1.000000}, delta_rows:0, delta_bytes:0, segments:1, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:0ms, tot_build_inputstream:1ms, min_local_stream:0ms, max_local_stream:180ms, dtfile:{data_scanned_rows:1000000, data_skipped_rows:0, mvcc_scanned_rows:0, mvcc_skipped_rows:0, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:0ms, tot_read:59ms}} | keep order:false | N/A | N/A |
| └─Projection_114(Probe) | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.64s, min:0s, avg: 2.32s, p80:4.64s, p95:4.64s, iters:4017, tasks:2, threads:72} | tpch100.lineitem.l_suppkey, tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.orders.o_orderdate, tpch100.orders.o_orderkey | N/A | N/A |
| └─HashJoin_94 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:4.64s, min:0s, avg: 2.32s, p80:4.64s, p95:4.64s, iters:8790, tasks:2, threads:72} | inner join, equal:[eq(tpch100.lineitem.l_orderkey, tpch100.orders.o_orderkey)] | N/A | N/A |
| ├─ExchangeReceiver_110(Build) | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:3.92s, min:0s, avg: 1.96s, p80:3.92s, p95:3.92s, iters:2684, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 565ms} | | N/A | N/A |
| │ └─ExchangeSender_109 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:3.94s, min:0s, avg: 1.97s, p80:3.94s, p95:3.94s, iters:4015, tasks:2, threads:72} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.lineitem.l_orderkey, collate: binary] | N/A | N/A |
| │ └─Projection_108 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:3.84s, min:0s, avg: 1.92s, p80:3.84s, p95:3.84s, iters:4015, tasks:2, threads:72} | tpch100.lineitem.l_orderkey, tpch100.lineitem.l_suppkey, tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount, tpch100.partsupp.ps_supplycost, tpch100.partsupp.ps_suppkey, tpch100.partsupp.ps_partkey | N/A | N/A |
| │ └─HashJoin_95 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:3.84s, min:0s, avg: 1.92s, p80:3.84s, p95:3.84s, iters:6521, tasks:2, threads:72} | inner join, equal:[eq(tpch100.lineitem.l_suppkey, tpch100.partsupp.ps_suppkey) eq(tpch100.lineitem.l_partkey, tpch100.partsupp.ps_partkey)] | N/A | N/A |
| │ ├─ExchangeReceiver_104(Build) | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:2.94s, min:0s, avg: 1.47s, p80:2.94s, p95:2.94s, iters:2707, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 599ms} | | N/A | N/A |
| │ │ └─ExchangeSender_103 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:3.16s, min:0s, avg: 1.58s, p80:3.16s, p95:3.16s, iters:4014, tasks:2, threads:72} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.lineitem.l_suppkey, collate: binary], [name: tpch100.lineitem.l_partkey, collate: binary] | N/A | N/A |
| │ │ └─Projection_102 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:2.7s, min:0s, avg: 1.35s, p80:2.7s, p95:2.7s, iters:4014, tasks:2, threads:72} | tpch100.lineitem.l_orderkey, tpch100.lineitem.l_partkey, tpch100.lineitem.l_suppkey, tpch100.lineitem.l_quantity, tpch100.lineitem.l_extendedprice, tpch100.lineitem.l_discount | N/A | N/A |
| │ │ └─HashJoin_96 | 59708020.55 | 32571554 | mpp[tiflash] | | tiflash_task:{proc max:2.7s, min:0s, avg: 1.35s, p80:2.7s, p95:2.7s, iters:13584, tasks:2, threads:72} | inner join, equal:[eq(tpch100.part.p_partkey, tpch100.lineitem.l_partkey)] | N/A | N/A |
| │ │ ├─ExchangeReceiver_100(Build) | 2000000.00 | 2171278 | mpp[tiflash] | | tiflash_task:{proc max:400.1ms, min:0s, avg: 200ms, p80:400.1ms, p95:400.1ms, iters:223, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 43ms} | | N/A | N/A |
| │ │ │ └─ExchangeSender_99 | 2000000.00 | 1085639 | mpp[tiflash] | | tiflash_task:{proc max:402ms, min:0s, avg: 201ms, p80:402ms, p95:402ms, iters:319, tasks:2, threads:72} | ExchangeType: Broadcast, Compression: FAST | N/A | N/A |
| │ │ │ └─TableFullScan_97 | 2000000.00 | 1085639 | mpp[tiflash] | table:part | tiflash_task:{proc max:401ms, min:0s, avg: 200.5ms, p80:401ms, p95:401ms, iters:319, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 6ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:14, remote_regions:0, tot_learner_read:37ms, region_balance:{instance_num: 2, max/min: 7/7=1.000000}, delta_rows:0, delta_bytes:0, segments:27, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:1814ms, tot_build_inputstream:2004ms, min_local_stream:0ms, max_local_stream:395ms, dtfile:{data_scanned_rows:20052329, data_skipped_rows:23752597, mvcc_scanned_rows:9040958, mvcc_skipped_rows:8937634, lm_filter_scanned_rows:20052329, lm_filter_skipped_rows:8937634, tot_rs_index_check:2ms, tot_read:5205ms}} | pushed down filter:like(tpch100.part.p_name, "%dim%", 92), keep order:false | N/A | N/A |
| │ │ └─TableFullScan_101(Probe) | 600037902.00 | 600037902 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:1.82s, min:0s, avg: 910.2ms, p80:1.82s, p95:1.82s, iters:9570, tasks:2, threads:72}, tiflash_wait: {pipeline_breaker_wait: 223ms, pipeline_queue_wait: 567ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:487, remote_regions:0, tot_learner_read:190ms, region_balance:{instance_num: 2, max/min: 244/243=1.004115}, delta_rows:0, delta_bytes:0, segments:776, stale_read_regions:0, tot_build_snapshot:26ms, tot_build_bitmap:55160ms, tot_build_inputstream:56222ms, min_local_stream:0ms, max_local_stream:1735ms, dtfile:{data_scanned_rows:601849646, data_skipped_rows:213267916, mvcc_scanned_rows:233990167, mvcc_skipped_rows:213267916, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:124ms, tot_read:138893ms}} | keep order:false | N/A | N/A |
| │ └─ExchangeReceiver_107(Probe) | 80000000.00 | 80000000 | mpp[tiflash] | | tiflash_task:{proc max:3.68s, min:0s, avg: 1.84s, p80:3.68s, p95:3.68s, iters:2506, tasks:2, threads:72}, tiflash_wait: {pipeline_breaker_wait: 3252ms, pipeline_queue_wait: 394ms} | | N/A | N/A |
| │ └─ExchangeSender_106 | 80000000.00 | 80000000 | mpp[tiflash] | | tiflash_task:{proc max:3.93s, min:0s, avg: 1.97s, p80:3.93s, p95:3.93s, iters:1275, tasks:2, threads:72} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.partsupp.ps_suppkey, collate: binary], [name: tpch100.partsupp.ps_partkey, collate: binary] | N/A | N/A |
| │ └─TableFullScan_105 | 80000000.00 | 80000000 | mpp[tiflash] | table:partsupp | tiflash_task:{proc max:763.3ms, min:0s, avg: 381.6ms, p80:763.3ms, p95:763.3ms, iters:1275, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 653ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:62, remote_regions:0, tot_learner_read:27ms, region_balance:{instance_num: 2, max/min: 31/31=1.000000}, delta_rows:0, delta_bytes:0, segments:105, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:10040ms, tot_build_inputstream:10213ms, min_local_stream:0ms, max_local_stream:736ms, dtfile:{data_scanned_rows:80277931, data_skipped_rows:29752164, mvcc_scanned_rows:35577087, mvcc_skipped_rows:29752164, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:31ms, tot_read:15133ms}} | keep order:false | N/A | N/A |
| └─ExchangeReceiver_113(Probe) | 150000000.00 | 150000000 | mpp[tiflash] | | tiflash_task:{proc max:4.54s, min:0s, avg: 2.27s, p80:4.54s, p95:4.54s, iters:4774, tasks:2, threads:72}, tiflash_wait: {pipeline_breaker_wait: 4016ms, pipeline_queue_wait: 480ms} | | N/A | N/A |
| └─ExchangeSender_112 | 150000000.00 | 150000000 | mpp[tiflash] | | tiflash_task:{proc max:4.71s, min:0s, avg: 2.35s, p80:4.71s, p95:4.71s, iters:2418, tasks:2, threads:72} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: tpch100.orders.o_orderkey, collate: binary] | N/A | N/A |
| └─TableFullScan_111 | 150000000.00 | 150000000 | mpp[tiflash] | table:orders | tiflash_task:{proc max:803.3ms, min:0s, avg: 401.6ms, p80:803.3ms, p95:803.3ms, iters:2418, tasks:2, threads:72}, tiflash_wait: {pipeline_queue_wait: 697ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:97, remote_regions:0, tot_learner_read:149ms, region_balance:{instance_num: 2, max/min: 49/48=1.020833}, delta_rows:0, delta_bytes:0, segments:202, stale_read_regions:0, tot_build_snapshot:1ms, tot_build_bitmap:8434ms, tot_build_inputstream:8732ms, min_local_stream:0ms, max_local_stream:776ms, dtfile:{data_scanned_rows:150476097, data_skipped_rows:53379297, mvcc_scanned_rows:54323283, mvcc_skipped_rows:53379297, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:46ms, tot_read:15235ms}} | keep order:false | N/A | N/A |
+-----------------------------------------------------------------------------------+--------------+-----------+--------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
41 rows in set (4.74 sec)FlameGraph
Before
After
According to flame graph, the part of building bitmap filter for stable only became much larger than before.
Also, the mvcc_scanned_rows is much greater than before. (for lineitem, 3699235 vs 233990167)
Before
| │ │ └─TableFullScan_101(Probe) | 600037902.00 | 600037902 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:1.28s, min:0s, avg: 637.9ms, p80:1.28s, p95:1.28s, iters:9560, tasks:2, threads:72}, tiflash_wait: {pipeline_breaker_wait: 177ms, pipeline_queue_wait: 713ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:487, remote_regions:0, tot_learner_read:149ms, region_balance:{instance_num: 2, max/min: 244/243=1.004115}, delta_rows:0, delta_bytes:0, segments:784, stale_read_regions:0, tot_build_snapshot:37ms, tot_build_bitmap:3627ms, tot_build_inputstream:4554ms, min_local_stream:474ms, max_local_stream:1158ms, dtfile:{data_scanned_rows:601825537, data_skipped_rows:221636406, mvcc_scanned_rows:3699235, mvcc_skipped_rows:470565855, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:269ms, tot_read:115738ms}} | keep order:false | N/A | N/A |
After
| │ │ └─TableFullScan_101(Probe) | 600037902.00 | 600037902 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:1.82s, min:0s, avg: 910.2ms, p80:1.82s, p95:1.82s, iters:9570, tasks:2, threads:72}, tiflash_wait: {pipeline_breaker_wait: 223ms, pipeline_queue_wait: 567ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, lm_skip_rows:0, local_regions:487, remote_regions:0, tot_learner_read:190ms, region_balance:{instance_num: 2, max/min: 244/243=1.004115}, delta_rows:0, delta_bytes:0, segments:776, stale_read_regions:0, tot_build_snapshot:26ms, tot_build_bitmap:55160ms, tot_build_inputstream:56222ms, min_local_stream:0ms, max_local_stream:1735ms, dtfile:{data_scanned_rows:601849646, data_skipped_rows:213267916, mvcc_scanned_rows:233990167, mvcc_skipped_rows:213267916, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:124ms, tot_read:138893ms}} | keep order:false | N/A | N/A |After some investigation, I found that this bug was introduced by #9738. This PR aims to reuse the pack filter result for MVCC, Build Bitmap, and Query. However, for building a bitmap filter, the pack with RSResult::All does not need to read at all. Reusing the same pack filter result caused redundant pack reads, significantly degrading performance.