Skip to content

Optimize sort settings default providers#138711

Merged
jordan-powers merged 4 commits intoelastic:mainfrom
jordan-powers:optimize-sort-settings-default-providers-2
Dec 4, 2025
Merged

Optimize sort settings default providers#138711
jordan-powers merged 4 commits intoelastic:mainfrom
jordan-powers:optimize-sort-settings-default-providers-2

Conversation

@jordan-powers
Copy link
Copy Markdown
Contributor

@jordan-powers jordan-powers commented Nov 27, 2025

This PR optimizes the default providers for the sort settings index.sort.field, index.sort.order, index.sort.mode, and index.sort.missing to fix a performance regression in the many-shards-quantitative benchmark associated with the introduction of these default providers in #135886.

@jordan-powers
Copy link
Copy Markdown
Contributor Author

Benchmark results running the elastic/logs many-shards-quantitative benchmark in esbench.

Baseline ref: 22a7491
Contender ref: 397cd92

Results:

|                                                        Metric |                                   Task |         Baseline |        Contender |        Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|---------------------------------------:|-----------------:|-----------------:|------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |                                        |    459.276       |    470.93        |    11.6535  |    min |   +2.54% |
|             Min cumulative indexing time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|          Median cumulative indexing time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |     0       |    min |    0.00% |
|             Max cumulative indexing time across primary shard |                                        |    276.031       |    285.264       |     9.23297 |    min |   +3.34% |
|           Cumulative indexing throttle time of primary shards |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |                                        |    232.969       |    184.357       |   -48.6121  |    min |  -20.87% |
|                      Cumulative merge count of primary shards |                                        |   1638           |   1715           |    77       |        |   +4.70% |
|                Min cumulative merge time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |                                        |    173.972       |    132.698       |   -41.2737  |    min |  -23.72% |
|              Cumulative merge throttle time of primary shards |                                        |    105.254       |     74.8442      |   -30.4093  |    min |  -28.89% |
|       Min cumulative merge throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |                                        |     78.3887      |     54.2814      |   -24.1073  |    min |  -30.75% |
|                     Cumulative refresh time of primary shards |                                        |     32.7415      |     32.5988      |    -0.1427  |    min |   -0.44% |
|                    Cumulative refresh count of primary shards |                                        | 419699           | 420964           |  1265       |        |   +0.30% |
|              Min cumulative refresh time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|           Median cumulative refresh time across primary shard |                                        |      6.66667e-05 |      8.33333e-05 |     2e-05   |    min |  +25.00% |
|              Max cumulative refresh time across primary shard |                                        |      9.42203     |      9.12148     |    -0.30055 |    min |   -3.19% |
|                       Cumulative flush time of primary shards |                                        |      6.75512     |      5.92762     |    -0.8275  |    min |  -12.25% |
|                      Cumulative flush count of primary shards |                                        |  50746           |  50748           |     2       |        |    0.00% |
|                Min cumulative flush time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |     0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |                                        |      5e-05       |      5e-05       |     0       |    min |    0.00% |
|                Max cumulative flush time across primary shard |                                        |      1.43217     |      1.25625     |    -0.17592 |    min |  -12.28% |
|                                       Total Young Gen GC time |                                        |    136.701       |    138.914       |     2.213   |      s |   +1.62% |
|                                      Total Young Gen GC count |                                        |   3538           |   3721           |   183       |        |   +5.17% |
|                                         Total Old Gen GC time |                                        |      0           |      0           |     0       |      s |    0.00% |
|                                        Total Old Gen GC count |                                        |      0           |      0           |     0       |        |    0.00% |
|                                                  Dataset size |                                        |     38.5072      |     36.096       |    -2.41113 |     GB |   -6.26% |
|                                                    Store size |                                        |     38.5072      |     36.096       |    -2.41113 |     GB |   -6.26% |
|                                                 Translog size |                                        |      0.00512289  |      0.0051232   |     0       |     GB |    0.01% |
|                                        Heap used for segments |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                      Heap used for doc values |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                           Heap used for terms |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                           Heap used for norms |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                          Heap used for points |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                   Heap used for stored fields |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                                 Segment count |                                        |  50361           |  50455           |    94       |        |   +0.19% |
|                                   Total Ingest Pipeline count |                                        |      2.10812e+08 |      2.10812e+08 |     0       |        |    0.00% |
|                                    Total Ingest Pipeline time |                                        |      1.3458e+07  |      1.3484e+07  | 26040       |     ms |   +0.19% |
|                                  Total Ingest Pipeline failed |                                        |      0           |      0           |     0       |        |    0.00% |
|                                                Min Throughput |                       insert-pipelines |      8.65798     |      8.61799     |    -0.04    |  ops/s |   -0.46% |
|                                               Mean Throughput |                       insert-pipelines |      8.65798     |      8.61799     |    -0.04    |  ops/s |   -0.46% |
|                                             Median Throughput |                       insert-pipelines |      8.65798     |      8.61799     |    -0.04    |  ops/s |   -0.46% |
|                                                Max Throughput |                       insert-pipelines |      8.65798     |      8.61799     |    -0.04    |  ops/s |   -0.46% |
|                                      100th percentile latency |                       insert-pipelines |   1729.14        |   1737.24        |     8.09741 |     ms |   +0.47% |
|                                 100th percentile service time |                       insert-pipelines |   1729.14        |   1737.24        |     8.09741 |     ms |   +0.47% |
|                                                    error rate |                       insert-pipelines |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             insert-ilm |     20.0165      |     19.0825      |    -0.93399 |  ops/s |   -4.67% |
|                                               Mean Throughput |                             insert-ilm |     20.0165      |     19.0825      |    -0.93399 |  ops/s |   -4.67% |
|                                             Median Throughput |                             insert-ilm |     20.0165      |     19.0825      |    -0.93399 |  ops/s |   -4.67% |
|                                                Max Throughput |                             insert-ilm |     20.0165      |     19.0825      |    -0.93399 |  ops/s |   -4.67% |
|                                      100th percentile latency |                             insert-ilm |     48.6596      |     51.4192      |     2.75956 |     ms |   +5.67% |
|                                 100th percentile service time |                             insert-ilm |     48.6596      |     51.4192      |     2.75956 |     ms |   +5.67% |
|                                                    error rate |                             insert-ilm |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput | validate-package-template-installation |     35.3818      |     32.1921      |    -3.18977 |  ops/s |   -9.02% |
|                                               Mean Throughput | validate-package-template-installation |     35.3818      |     32.1921      |    -3.18977 |  ops/s |   -9.02% |
|                                             Median Throughput | validate-package-template-installation |     35.3818      |     32.1921      |    -3.18977 |  ops/s |   -9.02% |
|                                                Max Throughput | validate-package-template-installation |     35.3818      |     32.1921      |    -3.18977 |  ops/s |   -9.02% |
|                                      100th percentile latency | validate-package-template-installation |     27.8724      |     30.6741      |     2.80172 |     ms |  +10.05% |
|                                 100th percentile service time | validate-package-template-installation |     27.8724      |     30.6741      |     2.80172 |     ms |  +10.05% |
|                                                    error rate | validate-package-template-installation |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |        update-custom-package-templates |     23.383       |     22.4174      |    -0.96564 |  ops/s |   -4.13% |
|                                               Mean Throughput |        update-custom-package-templates |     23.383       |     22.4174      |    -0.96564 |  ops/s |   -4.13% |
|                                             Median Throughput |        update-custom-package-templates |     23.383       |     22.4174      |    -0.96564 |  ops/s |   -4.13% |
|                                                Max Throughput |        update-custom-package-templates |     23.383       |     22.4174      |    -0.96564 |  ops/s |   -4.13% |
|                                      100th percentile latency |        update-custom-package-templates |    512.8         |    534.922       |    22.1224  |     ms |   +4.31% |
|                                 100th percentile service time |        update-custom-package-templates |    512.8         |    534.922       |    22.1224  |     ms |   +4.31% |
|                                                    error rate |        update-custom-package-templates |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    add-many-shards-ilm |     22.7746      |     22.1564      |    -0.61814 |  ops/s |   -2.71% |
|                                               Mean Throughput |                    add-many-shards-ilm |     22.7746      |     22.1564      |    -0.61814 |  ops/s |   -2.71% |
|                                             Median Throughput |                    add-many-shards-ilm |     22.7746      |     22.1564      |    -0.61814 |  ops/s |   -2.71% |
|                                                Max Throughput |                    add-many-shards-ilm |     22.7746      |     22.1564      |    -0.61814 |  ops/s |   -2.71% |
|                                      100th percentile latency |                    add-many-shards-ilm |     40.6528      |     44.2149      |     3.56211 |     ms |   +8.76% |
|                                 100th percentile service time |                    add-many-shards-ilm |     40.6528      |     44.2149      |     3.56211 |     ms |   +8.76% |
|                                                    error rate |                    add-many-shards-ilm |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        initial-indices |     80.4095      |     70.295       |   -10.1145  | docs/s |  -12.58% |
|                                               Mean Throughput |                        initial-indices |   1330.25        |   1122.83        |  -207.422   | docs/s |  -15.59% |
|                                             Median Throughput |                        initial-indices |   1190.46        |    966.798       |  -223.659   | docs/s |  -18.79% |
|                                                Max Throughput |                        initial-indices |   2368.7         |   2029.5         |  -339.203   | docs/s |  -14.32% |
|                                       50th percentile latency |                        initial-indices |   3346.26        |   4006.66        |   660.397   |     ms |  +19.74% |
|                                       90th percentile latency |                        initial-indices |   5895.69        |   6286.06        |   390.373   |     ms |   +6.62% |
|                                       99th percentile latency |                        initial-indices |   6435.26        |   6807.08        |   371.82    |     ms |   +5.78% |
|                                     99.9th percentile latency |                        initial-indices |   7391.12        |   7032.88        |  -358.234   |     ms |   -4.85% |
|                                    99.99th percentile latency |                        initial-indices |   7879.23        |   7106.08        |  -773.145   |     ms |   -9.81% |
|                                      100th percentile latency |                        initial-indices |   7883.95        |   7111.03        |  -772.921   |     ms |   -9.80% |
|                                  50th percentile service time |                        initial-indices |   3348.81        |   4015.34        |   666.524   |     ms |  +19.90% |
|                                  90th percentile service time |                        initial-indices |   5897.76        |   6278.06        |   380.296   |     ms |   +6.45% |
|                                  99th percentile service time |                        initial-indices |   6439.89        |   6810.55        |   370.659   |     ms |   +5.76% |
|                                99.9th percentile service time |                        initial-indices |   7397.52        |   7033.54        |  -363.981   |     ms |   -4.92% |
|                               99.99th percentile service time |                        initial-indices |   7879.23        |   7106.08        |  -773.145   |     ms |   -9.81% |
|                                 100th percentile service time |                        initial-indices |   7883.95        |   7111.03        |  -772.921   |     ms |   -9.80% |
|                                                    error rate |                        initial-indices |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        discover/search |      0.00228704  |      0.00228591  |    -0       |  ops/s |   -0.05% |
|                                               Mean Throughput |                        discover/search |      0.00759078  |      0.00767796  |     9e-05   |  ops/s |   +1.15% |
|                                             Median Throughput |                        discover/search |      0.00804437  |      0.00808595  |     4e-05   |  ops/s |   +0.52% |
|                                                Max Throughput |                        discover/search |      0.009853    |      0.00996214  |     0.00011 |  ops/s |   +1.11% |
|                                       50th percentile latency |                        discover/search |    501.631       |    447.622       |   -54.0088  |     ms |  -10.77% |
|                                       90th percentile latency |                        discover/search |   2164.05        |   2498.7         |   334.646   |     ms |  +15.46% |
|                                      100th percentile latency |                        discover/search |   5054.01        |   7493.44        |  2439.43    |     ms |  +48.27% |
|                                  50th percentile service time |                        discover/search |    424.471       |    431.024       |     6.5529  |     ms |   +1.54% |
|                                  90th percentile service time |                        discover/search |    835.619       |    653.342       |  -182.277   |     ms |  -21.81% |
|                                 100th percentile service time |                        discover/search |   5053.24        |   7491.9         |  2438.66    |     ms |  +48.26% |
|                                                    error rate |                        discover/search |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                     discover/visualize |      0.00231463  |      0.0023152   |     0       |  ops/s |   +0.02% |
|                                               Mean Throughput |                     discover/visualize |      0.00829714  |      0.00829794  |     0       |  ops/s |    0.01% |
|                                             Median Throughput |                     discover/visualize |      0.0079041   |      0.00790462  |     0       |  ops/s |    0.01% |
|                                                Max Throughput |                     discover/visualize |      0.014204    |      0.0142064   |     0       |  ops/s |   +0.02% |
|                                       50th percentile latency |                     discover/visualize |    407.759       |    407.425       |    -0.33482 |     ms |   -0.08% |
|                                       90th percentile latency |                     discover/visualize |    698.916       |   1370.36        |   671.445   |     ms |  +96.07% |
|                                      100th percentile latency |                     discover/visualize |   2994.83        |   4464.7         |  1469.87    |     ms |  +49.08% |
|                                  50th percentile service time |                     discover/visualize |    405.559       |    404.604       |    -0.95467 |     ms |   -0.24% |
|                                  90th percentile service time |                     discover/visualize |    477.164       |    436.753       |   -40.411   |     ms |   -8.47% |
|                                 100th percentile service time |                     discover/visualize |   2994.52        |   4463.85        |  1469.33    |     ms |  +49.07% |
|                                                    error rate |                     discover/visualize |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  kafka |      0.00502374  |      0.00502366  |    -0       |  ops/s |   -0.00% |
|                                               Mean Throughput |                                  kafka |      0.0129604   |      0.0130498   |     9e-05   |  ops/s |   +0.69% |
|                                             Median Throughput |                                  kafka |      0.0134311   |      0.013441    |     1e-05   |  ops/s |   +0.07% |
|                                                Max Throughput |                                  kafka |      0.0187264   |      0.0187262   |    -0       |  ops/s |   -0.00% |
|                                       50th percentile latency |                                  kafka |    408.485       |    407.818       |    -0.66705 |     ms |   -0.16% |
|                                       90th percentile latency |                                  kafka |    563.111       |    515.802       |   -47.3091  |     ms |   -8.40% |
|                                      100th percentile latency |                                  kafka |   3529.2         |   3004.35        |  -524.853   |     ms |  -14.87% |
|                                  50th percentile service time |                                  kafka |    406.878       |    405.563       |    -1.31564 |     ms |   -0.32% |
|                                  90th percentile service time |                                  kafka |    516.008       |    426.462       |   -89.5462  |     ms |  -17.35% |
|                                 100th percentile service time |                                  kafka |   3528.47        |   3003.75        |  -524.724   |     ms |  -14.87% |
|                                                    error rate |                                  kafka |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  nginx |      0.00627563  |      0.00637389  |     0.0001  |  ops/s |   +1.57% |
|                                               Mean Throughput |                                  nginx |      0.0218074   |      0.0218535   |     5e-05   |  ops/s |   +0.21% |
|                                             Median Throughput |                                  nginx |      0.0211443   |      0.0211628   |     2e-05   |  ops/s |   +0.09% |
|                                                Max Throughput |                                  nginx |      0.0345796   |      0.0346688   |     9e-05   |  ops/s |   +0.26% |
|                                       50th percentile latency |                                  nginx |   1822.27        |   1805.82        |   -16.4543  |     ms |   -0.90% |
|                                       90th percentile latency |                                  nginx |   3817.53        |   4097.84        |   280.305   |     ms |   +7.34% |
|                                      100th percentile latency |                                  nginx |   6164.35        |  10622.1         |  4457.72    |     ms |  +72.31% |
|                                  50th percentile service time |                                  nginx |   1797.21        |   1781.77        |   -15.4429  |     ms |   -0.86% |
|                                  90th percentile service time |                                  nginx |   1971.45        |   1904.64        |   -66.8134  |     ms |   -3.39% |
|                                 100th percentile service time |                                  nginx |   4425.57        |  10117.4         |  5691.87    |     ms | +128.61% |
|                                                    error rate |                                  nginx |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                 apache |      0.0233558   |      0.0233906   |     3e-05   |  ops/s |   +0.15% |
|                                               Mean Throughput |                                 apache |      0.0529463   |      0.05315     |     0.0002  |  ops/s |   +0.38% |
|                                             Median Throughput |                                 apache |      0.029454    |      0.0294395   |    -1e-05   |  ops/s |   -0.05% |
|                                                Max Throughput |                                 apache |      0.153354    |      0.158288    |     0.00493 |  ops/s |   +3.22% |
|                                       50th percentile latency |                                 apache |    402.697       |    398.509       |    -4.18781 |     ms |   -1.04% |
|                                       90th percentile latency |                                 apache |    527.587       |    568.676       |    41.0894  |     ms |   +7.79% |
|                                       99th percentile latency |                                 apache |   1063.42        |   1176.43        |   113.009   |     ms |  +10.63% |
|                                      100th percentile latency |                                 apache |   1197.8         |   1657.37        |   459.564   |     ms |  +38.37% |
|                                  50th percentile service time |                                 apache |    399.323       |    396.22        |    -3.10315 |     ms |   -0.78% |
|                                  90th percentile service time |                                 apache |    478.489       |    471.908       |    -6.58146 |     ms |   -1.38% |
|                                  99th percentile service time |                                 apache |   1062.43        |   1128.32        |    65.8967  |     ms |   +6.20% |
|                                 100th percentile service time |                                 apache |   1072.94        |   1178.17        |   105.228   |     ms |   +9.81% |
|                                                    error rate |                                 apache |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                            system/auth |      0.0427313   |      0.0427313   |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                            system/auth |      0.0845312   |      0.0843797   |    -0.00015 |  ops/s |   -0.18% |
|                                             Median Throughput |                            system/auth |      0.0742458   |      0.0734513   |    -0.00079 |  ops/s |   -1.07% |
|                                                Max Throughput |                            system/auth |      0.152622    |      0.152624    |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                            system/auth |    425.621       |    422.003       |    -3.61781 |     ms |   -0.85% |
|                                       90th percentile latency |                            system/auth |    564.255       |    554.736       |    -9.51891 |     ms |   -1.69% |
|                                       99th percentile latency |                            system/auth |    745.554       |    808.064       |    62.5097  |     ms |   +8.38% |
|                                      100th percentile latency |                            system/auth |   2069.43        |   1828.25        |  -241.176   |     ms |  -11.65% |
|                                  50th percentile service time |                            system/auth |    420.992       |    415.029       |    -5.96263 |     ms |   -1.42% |
|                                  90th percentile service time |                            system/auth |    515.911       |    505.076       |   -10.8346  |     ms |   -2.10% |
|                                  99th percentile service time |                            system/auth |    635.143       |    709.371       |    74.2283  |     ms |  +11.69% |
|                                 100th percentile service time |                            system/auth |   2067.88        |   1826.62        |  -241.262   |     ms |  -11.67% |
|                                                    error rate |                            system/auth |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                system/syslog/dashboard |      0.0242005   |      0.0242006   |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                system/syslog/dashboard |      0.0382994   |      0.0383863   |     9e-05   |  ops/s |   +0.23% |
|                                             Median Throughput |                system/syslog/dashboard |      0.0316978   |      0.0318483   |     0.00015 |  ops/s |   +0.47% |
|                                                Max Throughput |                system/syslog/dashboard |      0.0694738   |      0.0694744   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                system/syslog/dashboard |    414.382       |    413.607       |    -0.77515 |     ms |   -0.19% |
|                                       90th percentile latency |                system/syslog/dashboard |    710.115       |    665.609       |   -44.5056  |     ms |   -6.27% |
|                                      100th percentile latency |                system/syslog/dashboard |   2363.52        |   6580.07        |  4216.55    |     ms | +178.40% |
|                                  50th percentile service time |                system/syslog/dashboard |    406.338       |    405.815       |    -0.52301 |     ms |   -0.13% |
|                                  90th percentile service time |                system/syslog/dashboard |    466.415       |    465.453       |    -0.96168 |     ms |   -0.21% |
|                                 100th percentile service time |                system/syslog/dashboard |   2362.42        |   6579.33        |  4216.91    |     ms | +178.50% |
|                                                    error rate |                system/syslog/dashboard |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                     system/syslog/lens |      0.0113572   |      0.0113584   |     0       |  ops/s |    0.01% |
|                                               Mean Throughput |                     system/syslog/lens |      0.0240039   |      0.0239369   |    -7e-05   |  ops/s |   -0.28% |
|                                             Median Throughput |                     system/syslog/lens |      0.0234542   |      0.0232861   |    -0.00017 |  ops/s |   -0.72% |
|                                                Max Throughput |                     system/syslog/lens |      0.0583249   |      0.0583588   |     3e-05   |  ops/s |   +0.06% |
|                                       50th percentile latency |                     system/syslog/lens |   1207.61        |   1268.55        |    60.9388  |     ms |   +5.05% |
|                                       90th percentile latency |                     system/syslog/lens |   6158.28        |   6631.36        |   473.08    |     ms |   +7.68% |
|                                      100th percentile latency |                     system/syslog/lens |  10520.8         |   9581.16        |  -939.663   |     ms |   -8.93% |
|                                  50th percentile service time |                     system/syslog/lens |    433.978       |    430.35        |    -3.62863 |     ms |   -0.84% |
|                                  90th percentile service time |                     system/syslog/lens |   3987.15        |   4136.49        |   149.343   |     ms |   +3.75% |
|                                 100th percentile service time |                     system/syslog/lens |   8202.82        |   6584.58        | -1618.24    |     ms |  -19.73% |
|                                                    error rate |                     system/syslog/lens |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        mysql/dashboard |      0.00224979  |      0.00336261  |     0.00111 |  ops/s |  +49.46% |
|                                               Mean Throughput |                        mysql/dashboard |      0.0163243   |      0.0165892   |     0.00026 |  ops/s |   +1.62% |
|                                             Median Throughput |                        mysql/dashboard |      0.0176932   |      0.0177524   |     6e-05   |  ops/s |   +0.33% |
|                                                Max Throughput |                        mysql/dashboard |      0.0245867   |      0.0245871   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                        mysql/dashboard |    401.275       |    396.241       |    -5.03394 |     ms |   -1.25% |
|                                       90th percentile latency |                        mysql/dashboard |    461.174       |    439.077       |   -22.0976  |     ms |   -4.79% |
|                                      100th percentile latency |                        mysql/dashboard |   4248.56        |   4842.73        |   594.166   |     ms |  +13.99% |
|                                  50th percentile service time |                        mysql/dashboard |    398.951       |    394.41        |    -4.54149 |     ms |   -1.14% |
|                                  90th percentile service time |                        mysql/dashboard |    454.797       |    424.371       |   -30.4258  |     ms |   -6.69% |
|                                 100th percentile service time |                        mysql/dashboard |   3782.91        |   4351.51        |   568.599   |     ms |  +15.03% |
|                                                    error rate |                        mysql/dashboard |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  redis |      0.00958383  |      0.00958315  |    -0       |  ops/s |   -0.01% |
|                                               Mean Throughput |                                  redis |      0.0275553   |      0.0275776   |     2e-05   |  ops/s |   +0.08% |
|                                             Median Throughput |                                  redis |      0.0264381   |      0.0264379   |    -0       |  ops/s |   -0.00% |
|                                                Max Throughput |                                  redis |      0.0725947   |      0.0725908   |    -0       |  ops/s |   -0.01% |
|                                       50th percentile latency |                                  redis |    417.702       |    420.65        |     2.94843 |     ms |   +0.71% |
|                                       90th percentile latency |                                  redis |   1723.59        |   2168.25        |   444.657   |     ms |  +25.80% |
|                                       99th percentile latency |                                  redis |   6220.81        |   4499.23        | -1721.58    |     ms |  -27.67% |
|                                      100th percentile latency |                                  redis |   7394.47        |   5167.88        | -2226.59    |     ms |  -30.11% |
|                                  50th percentile service time |                                  redis |    412.661       |    416.616       |     3.95532 |     ms |   +0.96% |
|                                  90th percentile service time |                                  redis |    455.106       |    447.535       |    -7.57099 |     ms |   -1.66% |
|                                  99th percentile service time |                                  redis |   3889.22        |   4159.15        |   269.933   |     ms |   +6.94% |
|                                 100th percentile service time |                                  redis |   7392.12        |   5167.14        | -2224.98    |     ms |  -30.10% |
|                                                    error rate |                                  redis |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             mysql/lens |      0.00680615  |      0.00680619  |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                             mysql/lens |      0.0164084   |      0.0165231   |     0.00011 |  ops/s |   +0.70% |
|                                             Median Throughput |                             mysql/lens |      0.0127808   |      0.0129018   |     0.00012 |  ops/s |   +0.95% |
|                                                Max Throughput |                             mysql/lens |      0.0469888   |      0.0469937   |     0       |  ops/s |   +0.01% |
|                                       50th percentile latency |                             mysql/lens |    404.016       |    403.125       |    -0.89084 |     ms |   -0.22% |
|                                       90th percentile latency |                             mysql/lens |   4300.69        |   4065.11        |  -235.579   |     ms |   -5.48% |
|                                      100th percentile latency |                             mysql/lens |   5224.48        |   6114           |   889.523   |     ms |  +17.03% |
|                                  50th percentile service time |                             mysql/lens |    400.352       |    402.067       |     1.71506 |     ms |   +0.43% |
|                                  90th percentile service time |                             mysql/lens |   2829           |   2974.8         |   145.801   |     ms |   +5.15% |
|                                 100th percentile service time |                             mysql/lens |   5222.35        |   6112.95        |   890.602   |     ms |  +17.05% |
|                                                    error rate |                             mysql/lens |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/overview |      0.00513509  |      0.00513525  |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                    postgresql/overview |      0.061792    |      0.0618311   |     4e-05   |  ops/s |   +0.06% |
|                                             Median Throughput |                    postgresql/overview |      0.0631799   |      0.0632176   |     4e-05   |  ops/s |   +0.06% |
|                                                Max Throughput |                    postgresql/overview |      0.0893931   |      0.0893941   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                    postgresql/overview |   1230.84        |   1156.68        |   -74.1669  |     ms |   -6.03% |
|                                       90th percentile latency |                    postgresql/overview |   2272.46        |   1942.76        |  -329.703   |     ms |  -14.51% |
|                                       99th percentile latency |                    postgresql/overview |   6530.01        |   4529.9         | -2000.1     |     ms |  -30.63% |
|                                      100th percentile latency |                    postgresql/overview |   7323.11        |   5088.54        | -2234.57    |     ms |  -30.51% |
|                                  50th percentile service time |                    postgresql/overview |   1153.96        |   1125.64        |   -28.3221  |     ms |   -2.45% |
|                                  90th percentile service time |                    postgresql/overview |   1547.05        |   1609.11        |    62.0656  |     ms |   +4.01% |
|                                  99th percentile service time |                    postgresql/overview |   4738.04        |   4061.18        |  -676.86    |     ms |  -14.29% |
|                                 100th percentile service time |                    postgresql/overview |   7163.95        |   4743.69        | -2420.26    |     ms |  -33.78% |
|                                                    error rate |                    postgresql/overview |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/duration |      0.0224523   |      0.0224574   |     1e-05   |  ops/s |   +0.02% |
|                                               Mean Throughput |                    postgresql/duration |      0.0365889   |      0.0362369   |    -0.00035 |  ops/s |   -0.96% |
|                                             Median Throughput |                    postgresql/duration |      0.0302364   |      0.0300922   |    -0.00014 |  ops/s |   -0.48% |
|                                                Max Throughput |                    postgresql/duration |      0.0801545   |      0.0802239   |     7e-05   |  ops/s |   +0.09% |
|                                       50th percentile latency |                    postgresql/duration |    416.413       |    425.803       |     9.39038 |     ms |   +2.26% |
|                                       90th percentile latency |                    postgresql/duration |    801.768       |    795.384       |    -6.38394 |     ms |   -0.80% |
|                                      100th percentile latency |                    postgresql/duration |   3985.51        |   4814.76        |   829.251   |     ms |  +20.81% |
|                                  50th percentile service time |                    postgresql/duration |    410.434       |    414.093       |     3.65945 |     ms |   +0.89% |
|                                  90th percentile service time |                    postgresql/duration |    495.168       |    476.637       |   -18.531   |     ms |   -3.74% |
|                                 100th percentile service time |                    postgresql/duration |   3984.41        |   4805.85        |   821.442   |     ms |  +20.62% |
|                                                    error rate |                    postgresql/duration |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             bulk-index |    207.92        |     28.6298      |  -179.29    | docs/s |  -86.23% |
|                                               Mean Throughput |                             bulk-index |  58833.2         |  54708.9         | -4124.34    | docs/s |   -7.01% |
|                                             Median Throughput |                             bulk-index |  59610.4         |  57513           | -2097.34    | docs/s |   -3.52% |
|                                                Max Throughput |                             bulk-index |  60014.6         |  59047.8         |  -966.883   | docs/s |   -1.61% |
|                                       50th percentile latency |                             bulk-index |    323.108       |    323.582       |     0.47308 |     ms |   +0.15% |
|                                       90th percentile latency |                             bulk-index |    559.409       |    561.3         |     1.89136 |     ms |   +0.34% |
|                                       99th percentile latency |                             bulk-index |    947.029       |    887.466       |   -59.5627  |     ms |   -6.29% |
|                                     99.9th percentile latency |                             bulk-index |   3474.96        |   3218.63        |  -256.323   |     ms |   -7.38% |
|                                    99.99th percentile latency |                             bulk-index |   5629.83        |  28112.5         | 22482.6     |     ms | +399.35% |
|                                      100th percentile latency |                             bulk-index |   8841.69        |  34865.6         | 26023.9     |     ms | +294.33% |
|                                  50th percentile service time |                             bulk-index |    323.951       |    323.929       |    -0.0224  |     ms |   -0.01% |
|                                  90th percentile service time |                             bulk-index |    566.043       |    560.432       |    -5.61164 |     ms |   -0.99% |
|                                  99th percentile service time |                             bulk-index |    945.919       |    883.878       |   -62.0415  |     ms |   -6.56% |
|                                99.9th percentile service time |                             bulk-index |   3477.48        |   3217.01        |  -260.476   |     ms |   -7.49% |
|                               99.99th percentile service time |                             bulk-index |   5605.52        |  29089.2         | 23483.6     |     ms | +418.94% |
|                                 100th percentile service time |                             bulk-index |   8841.69        |  34865.6         | 26023.9     |     ms | +294.33% |
|                                                    error rate |                             bulk-index |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             field-caps |      2.42492     |      2.24417     |    -0.18075 |  ops/s |   -7.45% |
|                                               Mean Throughput |                             field-caps |      3.02836     |      2.88439     |    -0.14397 |  ops/s |   -4.75% |
|                                             Median Throughput |                             field-caps |      3.08178     |      2.96612     |    -0.11566 |  ops/s |   -3.75% |
|                                                Max Throughput |                             field-caps |      3.14591     |      3.02231     |    -0.1236  |  ops/s |   -3.93% |
|                                       50th percentile latency |                             field-caps |    606.82        |    624.874       |    18.0546  |     ms |   +2.98% |
|                                       90th percentile latency |                             field-caps |    672.35        |    669.301       |    -3.04961 |     ms |   -0.45% |
|                                       99th percentile latency |                             field-caps |    694.742       |    720.571       |    25.8292  |     ms |   +3.72% |
|                                      100th percentile latency |                             field-caps |    763.887       |    758.324       |    -5.56305 |     ms |   -0.73% |
|                                  50th percentile service time |                             field-caps |    606.82        |    624.874       |    18.0546  |     ms |   +2.98% |
|                                  90th percentile service time |                             field-caps |    672.35        |    669.301       |    -3.04961 |     ms |   -0.45% |
|                                  99th percentile service time |                             field-caps |    694.742       |    720.571       |    25.8292  |     ms |   +3.72% |
|                                 100th percentile service time |                             field-caps |    763.887       |    758.324       |    -5.56305 |     ms |   -0.73% |
|                                                    error rate |                             field-caps |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |        field-caps-exclude-empty-fields |      0.354446    |      0.383768    |     0.02932 |  ops/s |   +8.27% |
|                                               Mean Throughput |        field-caps-exclude-empty-fields |      0.37615     |      0.409491    |     0.03334 |  ops/s |   +8.86% |
|                                             Median Throughput |        field-caps-exclude-empty-fields |      0.377413    |      0.410892    |     0.03348 |  ops/s |   +8.87% |
|                                                Max Throughput |        field-caps-exclude-empty-fields |      0.378146    |      0.412305    |     0.03416 |  ops/s |   +9.03% |
|                                       50th percentile latency |        field-caps-exclude-empty-fields |   5292.47        |   4848.64        |  -443.822   |     ms |   -8.39% |
|                                       90th percentile latency |        field-caps-exclude-empty-fields |   5363.24        |   4911.71        |  -451.528   |     ms |   -8.42% |
|                                       99th percentile latency |        field-caps-exclude-empty-fields |   5404.22        |   4984.53        |  -419.695   |     ms |   -7.77% |
|                                      100th percentile latency |        field-caps-exclude-empty-fields |   5427.63        |   5031.99        |  -395.637   |     ms |   -7.29% |
|                                  50th percentile service time |        field-caps-exclude-empty-fields |   5292.47        |   4848.64        |  -443.822   |     ms |   -8.39% |
|                                  90th percentile service time |        field-caps-exclude-empty-fields |   5363.24        |   4911.71        |  -451.528   |     ms |   -8.42% |
|                                  99th percentile service time |        field-caps-exclude-empty-fields |   5404.22        |   4984.53        |  -419.695   |     ms |   -7.77% |
|                                 100th percentile service time |        field-caps-exclude-empty-fields |   5427.63        |   5031.99        |  -395.637   |     ms |   -7.29% |
|                                                    error rate |        field-caps-exclude-empty-fields |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                field-caps-index-filter |      2.09805     |      2.03072     |    -0.06733 |  ops/s |   -3.21% |
|                                               Mean Throughput |                field-caps-index-filter |      2.20143     |      2.12872     |    -0.07271 |  ops/s |   -3.30% |
|                                             Median Throughput |                field-caps-index-filter |      2.20826     |      2.13725     |    -0.07101 |  ops/s |   -3.22% |
|                                                Max Throughput |                field-caps-index-filter |      2.22311     |      2.1494      |    -0.07371 |  ops/s |   -3.32% |
|                                       50th percentile latency |                field-caps-index-filter |    890.331       |    915.169       |    24.8388  |     ms |   +2.79% |
|                                       90th percentile latency |                field-caps-index-filter |    961.78        |    991.981       |    30.2008  |     ms |   +3.14% |
|                                       99th percentile latency |                field-caps-index-filter |   1098.11        |   1105.46        |     7.35384 |     ms |   +0.67% |
|                                      100th percentile latency |                field-caps-index-filter |   1171.06        |   1195.64        |    24.5818  |     ms |   +2.10% |
|                                  50th percentile service time |                field-caps-index-filter |    890.331       |    915.169       |    24.8388  |     ms |   +2.79% |
|                                  90th percentile service time |                field-caps-index-filter |    961.78        |    991.981       |    30.2008  |     ms |   +3.14% |
|                                  99th percentile service time |                field-caps-index-filter |   1098.11        |   1105.46        |     7.35384 |     ms |   +0.67% |
|                                 100th percentile service time |                field-caps-index-filter |   1171.06        |   1195.64        |    24.5818  |     ms |   +2.10% |
|                                                    error rate |                field-caps-index-filter |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |   field-caps-index-filter-empty-fields |      0.315071    |      0.332959    |     0.01789 |  ops/s |   +5.68% |
|                                               Mean Throughput |   field-caps-index-filter-empty-fields |      0.328892    |      0.343173    |     0.01428 |  ops/s |   +4.34% |
|                                             Median Throughput |   field-caps-index-filter-empty-fields |      0.329896    |      0.343536    |     0.01364 |  ops/s |   +4.13% |
|                                                Max Throughput |   field-caps-index-filter-empty-fields |      0.330953    |      0.347201    |     0.01625 |  ops/s |   +4.91% |
|                                       50th percentile latency |   field-caps-index-filter-empty-fields |   6043.94        |   5773.76        |  -270.182   |     ms |   -4.47% |
|                                       90th percentile latency |   field-caps-index-filter-empty-fields |   6130.38        |   5907.53        |  -222.854   |     ms |   -3.64% |
|                                       99th percentile latency |   field-caps-index-filter-empty-fields |   6232.07        |   6001.03        |  -231.039   |     ms |   -3.71% |
|                                      100th percentile latency |   field-caps-index-filter-empty-fields |   6289.37        |   6099.03        |  -190.342   |     ms |   -3.03% |
|                                  50th percentile service time |   field-caps-index-filter-empty-fields |   6043.94        |   5773.76        |  -270.182   |     ms |   -4.47% |
|                                  90th percentile service time |   field-caps-index-filter-empty-fields |   6130.38        |   5907.53        |  -222.854   |     ms |   -3.64% |
|                                  99th percentile service time |   field-caps-index-filter-empty-fields |   6232.07        |   6001.03        |  -231.039   |     ms |   -3.71% |
|                                 100th percentile service time |   field-caps-index-filter-empty-fields |   6289.37        |   6099.03        |  -190.342   |     ms |   -3.03% |
|                                                    error rate |   field-caps-index-filter-empty-fields |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                 esql_from_all_limit_50 |      0.388699    |      0.372354    |    -0.01635 |  ops/s |   -4.21% |
|                                               Mean Throughput |                 esql_from_all_limit_50 |      0.45163     |      0.436742    |    -0.01489 |  ops/s |   -3.30% |
|                                             Median Throughput |                 esql_from_all_limit_50 |      0.461227    |      0.446201    |    -0.01503 |  ops/s |   -3.26% |
|                                                Max Throughput |                 esql_from_all_limit_50 |      0.484858    |      0.474159    |    -0.0107  |  ops/s |   -2.21% |
|                                       50th percentile latency |                 esql_from_all_limit_50 |   1913.89        |   1897.07        |   -16.8187  |     ms |   -0.88% |
|                                       90th percentile latency |                 esql_from_all_limit_50 |   2073.65        |   2024.85        |   -48.7989  |     ms |   -2.35% |
|                                      100th percentile latency |                 esql_from_all_limit_50 |   2182.14        |   2449.99        |   267.848   |     ms |  +12.27% |
|                                  50th percentile service time |                 esql_from_all_limit_50 |   1913.89        |   1897.07        |   -16.8187  |     ms |   -0.88% |
|                                  90th percentile service time |                 esql_from_all_limit_50 |   2073.65        |   2024.85        |   -48.7989  |     ms |   -2.35% |
|                                 100th percentile service time |                 esql_from_all_limit_50 |   2182.14        |   2449.99        |   267.848   |     ms |  +12.27% |
|                                                    error rate |                 esql_from_all_limit_50 |      0           |      0           |     0       |      % |    0.00% |

@jordan-powers
Copy link
Copy Markdown
Contributor Author

Ok, I ran another benchmark with the optimizations from this PR cherry-picked onto the first commit that introduced the regression. This eliminates any slowdowns due to other changes since the original regression was introduced.

Baseline ref: 22a7491
Contender ref: 687b291

|                                                        Metric |                                   Task |         Baseline |        Contender |          Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|---------------------------------------:|-----------------:|-----------------:|--------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |                                        |    459.276       |    454.589       |      -4.68722 |    min |   -1.02% |
|             Min cumulative indexing time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|          Median cumulative indexing time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |       0       |    min |    0.00% |
|             Max cumulative indexing time across primary shard |                                        |    276.031       |    273.139       |      -2.8925  |    min |   -1.05% |
|           Cumulative indexing throttle time of primary shards |                                        |      0           |      0           |       0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |                                        |    232.969       |    213.911       |     -19.0586  |    min |   -8.18% |
|                      Cumulative merge count of primary shards |                                        |   1638           |   1686           |      48       |        |   +2.93% |
|                Min cumulative merge time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |                                        |    173.972       |    153.567       |     -20.4046  |    min |  -11.73% |
|              Cumulative merge throttle time of primary shards |                                        |    105.254       |     91.7738      |     -13.4797  |    min |  -12.81% |
|       Min cumulative merge throttle time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |                                        |     78.3887      |     65.0141      |     -13.3747  |    min |  -17.06% |
|                     Cumulative refresh time of primary shards |                                        |     32.7415      |     34.0513      |       1.30978 |    min |   +4.00% |
|                    Cumulative refresh count of primary shards |                                        | 419699           | 421359           |    1660       |        |   +0.40% |
|              Min cumulative refresh time across primary shard |                                        |      0           |      0           |       0       |    min |    0.00% |
|           Median cumulative refresh time across primary shard |                                        |      6.66667e-05 |      8.33333e-05 |       2e-05   |    min |  +25.00% |
|              Max cumulative refresh time across primary shard |                                        |      9.42203     |      9.19882     |      -0.22322 |    min |   -2.37% |
|                       Cumulative flush time of primary shards |                                        |      6.75512     |      6.17272     |      -0.5824  |    min |   -8.62% |
|                      Cumulative flush count of primary shards |                                        |  50746           |  50994           |     248       |        |   +0.49% |
|                Min cumulative flush time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |       0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |                                        |      5e-05       |      5e-05       |       0       |    min |    0.00% |
|                Max cumulative flush time across primary shard |                                        |      1.43217     |      1.62473     |       0.19257 |    min |  +13.45% |
|                                       Total Young Gen GC time |                                        |    136.701       |    139.996       |       3.295   |      s |   +2.41% |
|                                      Total Young Gen GC count |                                        |   3538           |   3635           |      97       |        |   +2.74% |
|                                         Total Old Gen GC time |                                        |      0           |      0           |       0       |      s |    0.00% |
|                                        Total Old Gen GC count |                                        |      0           |      0           |       0       |        |    0.00% |
|                                                  Dataset size |                                        |     38.5072      |     38.521       |       0.01378 |     GB |   +0.04% |
|                                                    Store size |                                        |     38.5072      |     38.521       |       0.01378 |     GB |   +0.04% |
|                                                 Translog size |                                        |      0.00512289  |      0.0051232   |       0       |     GB |    0.01% |
|                                        Heap used for segments |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                      Heap used for doc values |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                           Heap used for terms |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                           Heap used for norms |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                          Heap used for points |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                   Heap used for stored fields |                                        |      0           |      0           |       0       |     MB |    0.00% |
|                                                 Segment count |                                        |  50361           |  50439           |      78       |        |   +0.15% |
|                                   Total Ingest Pipeline count |                                        |      2.10812e+08 |      2.10812e+08 |       0       |        |    0.00% |
|                                    Total Ingest Pipeline time |                                        |      1.3458e+07  |      1.3275e+07  | -182988       |     ms |   -1.36% |
|                                  Total Ingest Pipeline failed |                                        |      0           |      0           |       0       |        |    0.00% |
|                                                Min Throughput |                       insert-pipelines |      8.65798     |      8.19218     |      -0.4658  |  ops/s |   -5.38% |
|                                               Mean Throughput |                       insert-pipelines |      8.65798     |      8.19218     |      -0.4658  |  ops/s |   -5.38% |
|                                             Median Throughput |                       insert-pipelines |      8.65798     |      8.19218     |      -0.4658  |  ops/s |   -5.38% |
|                                                Max Throughput |                       insert-pipelines |      8.65798     |      8.19218     |      -0.4658  |  ops/s |   -5.38% |
|                                      100th percentile latency |                       insert-pipelines |   1729.14        |   1827.71        |      98.5752  |     ms |   +5.70% |
|                                 100th percentile service time |                       insert-pipelines |   1729.14        |   1827.71        |      98.5752  |     ms |   +5.70% |
|                                                    error rate |                       insert-pipelines |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                             insert-ilm |     20.0165      |     20.3982      |       0.3817  |  ops/s |   +1.91% |
|                                               Mean Throughput |                             insert-ilm |     20.0165      |     20.3982      |       0.3817  |  ops/s |   +1.91% |
|                                             Median Throughput |                             insert-ilm |     20.0165      |     20.3982      |       0.3817  |  ops/s |   +1.91% |
|                                                Max Throughput |                             insert-ilm |     20.0165      |     20.3982      |       0.3817  |  ops/s |   +1.91% |
|                                      100th percentile latency |                             insert-ilm |     48.6596      |     48.0983      |      -0.56127 |     ms |   -1.15% |
|                                 100th percentile service time |                             insert-ilm |     48.6596      |     48.0983      |      -0.56127 |     ms |   -1.15% |
|                                                    error rate |                             insert-ilm |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput | validate-package-template-installation |     35.3818      |     33.7578      |      -1.62404 |  ops/s |   -4.59% |
|                                               Mean Throughput | validate-package-template-installation |     35.3818      |     33.7578      |      -1.62404 |  ops/s |   -4.59% |
|                                             Median Throughput | validate-package-template-installation |     35.3818      |     33.7578      |      -1.62404 |  ops/s |   -4.59% |
|                                                Max Throughput | validate-package-template-installation |     35.3818      |     33.7578      |      -1.62404 |  ops/s |   -4.59% |
|                                      100th percentile latency | validate-package-template-installation |     27.8724      |     29.2779      |       1.40553 |     ms |   +5.04% |
|                                 100th percentile service time | validate-package-template-installation |     27.8724      |     29.2779      |       1.40553 |     ms |   +5.04% |
|                                                    error rate | validate-package-template-installation |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |        update-custom-package-templates |     23.383       |     23.9903      |       0.60729 |  ops/s |   +2.60% |
|                                               Mean Throughput |        update-custom-package-templates |     23.383       |     23.9903      |       0.60729 |  ops/s |   +2.60% |
|                                             Median Throughput |        update-custom-package-templates |     23.383       |     23.9903      |       0.60729 |  ops/s |   +2.60% |
|                                                Max Throughput |        update-custom-package-templates |     23.383       |     23.9903      |       0.60729 |  ops/s |   +2.60% |
|                                      100th percentile latency |        update-custom-package-templates |    512.8         |    499.829       |     -12.9708  |     ms |   -2.53% |
|                                 100th percentile service time |        update-custom-package-templates |    512.8         |    499.829       |     -12.9708  |     ms |   -2.53% |
|                                                    error rate |        update-custom-package-templates |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                    add-many-shards-ilm |     22.7746      |     20.5979      |      -2.17671 |  ops/s |   -9.56% |
|                                               Mean Throughput |                    add-many-shards-ilm |     22.7746      |     20.5979      |      -2.17671 |  ops/s |   -9.56% |
|                                             Median Throughput |                    add-many-shards-ilm |     22.7746      |     20.5979      |      -2.17671 |  ops/s |   -9.56% |
|                                                Max Throughput |                    add-many-shards-ilm |     22.7746      |     20.5979      |      -2.17671 |  ops/s |   -9.56% |
|                                      100th percentile latency |                    add-many-shards-ilm |     40.6528      |     45.3108      |       4.65806 |     ms |  +11.46% |
|                                 100th percentile service time |                    add-many-shards-ilm |     40.6528      |     45.3108      |       4.65806 |     ms |  +11.46% |
|                                                    error rate |                    add-many-shards-ilm |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                        initial-indices |     80.4095      |     77.8629      |      -2.5466  | docs/s |   -3.17% |
|                                               Mean Throughput |                        initial-indices |   1330.25        |   1306.12        |     -24.1295  | docs/s |   -1.81% |
|                                             Median Throughput |                        initial-indices |   1190.46        |   1230.95        |      40.4963  | docs/s |   +3.40% |
|                                                Max Throughput |                        initial-indices |   2368.7         |   2384.91        |      16.2119  | docs/s |   +0.68% |
|                                       50th percentile latency |                        initial-indices |   3346.26        |   3356.71        |      10.4421  |     ms |   +0.31% |
|                                       90th percentile latency |                        initial-indices |   5895.69        |   5657.66        |    -238.033   |     ms |   -4.04% |
|                                       99th percentile latency |                        initial-indices |   6435.26        |   6410.09        |     -25.1767  |     ms |   -0.39% |
|                                     99.9th percentile latency |                        initial-indices |   7391.12        |  11611.6         |    4220.44    |     ms |  +57.10% |
|                                    99.99th percentile latency |                        initial-indices |   7879.23        |  13263           |    5383.77    |     ms |  +68.33% |
|                                      100th percentile latency |                        initial-indices |   7883.95        |  13268.5         |    5384.5     |     ms |  +68.30% |
|                                  50th percentile service time |                        initial-indices |   3348.81        |   3336.04        |     -12.7758  |     ms |   -0.38% |
|                                  90th percentile service time |                        initial-indices |   5897.76        |   5660.6         |    -237.159   |     ms |   -4.02% |
|                                  99th percentile service time |                        initial-indices |   6439.89        |   6419.87        |     -20.0244  |     ms |   -0.31% |
|                                99.9th percentile service time |                        initial-indices |   7397.52        |  11642.7         |    4245.16    |     ms |  +57.39% |
|                               99.99th percentile service time |                        initial-indices |   7879.23        |  13263           |    5383.77    |     ms |  +68.33% |
|                                 100th percentile service time |                        initial-indices |   7883.95        |  13268.5         |    5384.5     |     ms |  +68.30% |
|                                                    error rate |                        initial-indices |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                        discover/search |      0.00228704  |      0.00228486  |      -0       |  ops/s |   -0.10% |
|                                               Mean Throughput |                        discover/search |      0.00759078  |      0.00771861  |       0.00013 |  ops/s |   +1.68% |
|                                             Median Throughput |                        discover/search |      0.00804437  |      0.00812159  |       8e-05   |  ops/s |   +0.96% |
|                                                Max Throughput |                        discover/search |      0.009853    |      0.00996164  |       0.00011 |  ops/s |   +1.10% |
|                                       50th percentile latency |                        discover/search |    501.631       |    433.79        |     -67.8411  |     ms |  -13.52% |
|                                       90th percentile latency |                        discover/search |   2164.05        |   1703.65        |    -460.4     |     ms |  -21.27% |
|                                      100th percentile latency |                        discover/search |   5054.01        |   3244.5         |   -1809.51    |     ms |  -35.80% |
|                                  50th percentile service time |                        discover/search |    424.471       |    419.261       |      -5.2101  |     ms |   -1.23% |
|                                  90th percentile service time |                        discover/search |    835.619       |    627.4         |    -208.22    |     ms |  -24.92% |
|                                 100th percentile service time |                        discover/search |   5053.24        |   3242.85        |   -1810.4     |     ms |  -35.83% |
|                                                    error rate |                        discover/search |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                     discover/visualize |      0.00231463  |      0.00231566  |       0       |  ops/s |   +0.04% |
|                                               Mean Throughput |                     discover/visualize |      0.00829714  |      0.00829673  |      -0       |  ops/s |   -0.00% |
|                                             Median Throughput |                     discover/visualize |      0.0079041   |      0.00793211  |       3e-05   |  ops/s |   +0.35% |
|                                                Max Throughput |                     discover/visualize |      0.014204    |      0.0142083   |       0       |  ops/s |   +0.03% |
|                                       50th percentile latency |                     discover/visualize |    407.759       |    409.461       |       1.70178 |     ms |   +0.42% |
|                                       90th percentile latency |                     discover/visualize |    698.916       |    668.478       |     -30.4381  |     ms |   -4.36% |
|                                      100th percentile latency |                     discover/visualize |   2994.83        |   1844.86        |   -1149.98    |     ms |  -38.40% |
|                                  50th percentile service time |                     discover/visualize |    405.559       |    405.627       |       0.06799 |     ms |   +0.02% |
|                                  90th percentile service time |                     discover/visualize |    477.164       |    446.575       |     -30.5891  |     ms |   -6.41% |
|                                 100th percentile service time |                     discover/visualize |   2994.52        |   1843.66        |   -1150.85    |     ms |  -38.43% |
|                                                    error rate |                     discover/visualize |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                                  kafka |      0.00502374  |      0.00502315  |      -0       |  ops/s |   -0.01% |
|                                               Mean Throughput |                                  kafka |      0.0129604   |      0.0130494   |       9e-05   |  ops/s |   +0.69% |
|                                             Median Throughput |                                  kafka |      0.0134311   |      0.0134408   |       1e-05   |  ops/s |   +0.07% |
|                                                Max Throughput |                                  kafka |      0.0187264   |      0.0187244   |      -0       |  ops/s |   -0.01% |
|                                       50th percentile latency |                                  kafka |    408.485       |    417.907       |       9.42253 |     ms |   +2.31% |
|                                       90th percentile latency |                                  kafka |    563.111       |    584.924       |      21.813   |     ms |   +3.87% |
|                                      100th percentile latency |                                  kafka |   3529.2         |   4683.15        |    1153.95    |     ms |  +32.70% |
|                                  50th percentile service time |                                  kafka |    406.878       |    411.988       |       5.10942 |     ms |   +1.26% |
|                                  90th percentile service time |                                  kafka |    516.008       |    462.568       |     -53.4403  |     ms |  -10.36% |
|                                 100th percentile service time |                                  kafka |   3528.47        |   4682.43        |    1153.96    |     ms |  +32.70% |
|                                                    error rate |                                  kafka |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                                  nginx |      0.00627563  |      0.00638893  |       0.00011 |  ops/s |   +1.81% |
|                                               Mean Throughput |                                  nginx |      0.0218074   |      0.0218619   |       5e-05   |  ops/s |   +0.25% |
|                                             Median Throughput |                                  nginx |      0.0211443   |      0.0211656   |       2e-05   |  ops/s |   +0.10% |
|                                                Max Throughput |                                  nginx |      0.0345796   |      0.0346823   |       0.0001  |  ops/s |   +0.30% |
|                                       50th percentile latency |                                  nginx |   1822.27        |   1809.92        |     -12.3488  |     ms |   -0.68% |
|                                       90th percentile latency |                                  nginx |   3817.53        |   3816.3         |      -1.23376 |     ms |   -0.03% |
|                                      100th percentile latency |                                  nginx |   6164.35        |   4821.98        |   -1342.37    |     ms |  -21.78% |
|                                  50th percentile service time |                                  nginx |   1797.21        |   1795.94        |      -1.26709 |     ms |   -0.07% |
|                                  90th percentile service time |                                  nginx |   1971.45        |   1970.7         |      -0.75381 |     ms |   -0.04% |
|                                 100th percentile service time |                                  nginx |   4425.57        |   4244.37        |    -181.197   |     ms |   -4.09% |
|                                                    error rate |                                  nginx |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                                 apache |      0.0233558   |      0.0233565   |       0       |  ops/s |    0.00% |
|                                               Mean Throughput |                                 apache |      0.0529463   |      0.0532124   |       0.00027 |  ops/s |   +0.50% |
|                                             Median Throughput |                                 apache |      0.029454    |      0.0294547   |       0       |  ops/s |    0.00% |
|                                                Max Throughput |                                 apache |      0.153354    |      0.153456    |       0.0001  |  ops/s |   +0.07% |
|                                       50th percentile latency |                                 apache |    402.697       |    398.207       |      -4.48929 |     ms |   -1.11% |
|                                       90th percentile latency |                                 apache |    527.587       |    552.996       |      25.4091  |     ms |   +4.82% |
|                                       99th percentile latency |                                 apache |   1063.42        |   1105.07        |      41.6469  |     ms |   +3.92% |
|                                      100th percentile latency |                                 apache |   1197.8         |   1214.45        |      16.6484  |     ms |   +1.39% |
|                                  50th percentile service time |                                 apache |    399.323       |    396.21        |      -3.1138  |     ms |   -0.78% |
|                                  90th percentile service time |                                 apache |    478.489       |    489.486       |      10.9962  |     ms |   +2.30% |
|                                  99th percentile service time |                                 apache |   1062.43        |   1098.04        |      35.6092  |     ms |   +3.35% |
|                                 100th percentile service time |                                 apache |   1072.94        |   1105.54        |      32.5949  |     ms |   +3.04% |
|                                                    error rate |                                 apache |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                            system/auth |      0.0427313   |      0.0427313   |       0       |  ops/s |    0.00% |
|                                               Mean Throughput |                            system/auth |      0.0845312   |      0.0845318   |       0       |  ops/s |    0.00% |
|                                             Median Throughput |                            system/auth |      0.0742458   |      0.074246    |       0       |  ops/s |    0.00% |
|                                                Max Throughput |                            system/auth |      0.152622    |      0.152625    |       0       |  ops/s |    0.00% |
|                                       50th percentile latency |                            system/auth |    425.621       |    426.418       |       0.79652 |     ms |   +0.19% |
|                                       90th percentile latency |                            system/auth |    564.255       |    570.803       |       6.54846 |     ms |   +1.16% |
|                                       99th percentile latency |                            system/auth |    745.554       |    732.609       |     -12.9455  |     ms |   -1.74% |
|                                      100th percentile latency |                            system/auth |   2069.43        |   1937.33        |    -132.095   |     ms |   -6.38% |
|                                  50th percentile service time |                            system/auth |    420.992       |    420.188       |      -0.80348 |     ms |   -0.19% |
|                                  90th percentile service time |                            system/auth |    515.911       |    517.185       |       1.27339 |     ms |   +0.25% |
|                                  99th percentile service time |                            system/auth |    635.143       |    691.062       |      55.9197  |     ms |   +8.80% |
|                                 100th percentile service time |                            system/auth |   2067.88        |   1935.58        |    -132.303   |     ms |   -6.40% |
|                                                    error rate |                            system/auth |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                system/syslog/dashboard |      0.0242005   |      0.0242005   |      -0       |  ops/s |   -0.00% |
|                                               Mean Throughput |                system/syslog/dashboard |      0.0382994   |      0.038298    |      -0       |  ops/s |   -0.00% |
|                                             Median Throughput |                system/syslog/dashboard |      0.0316978   |      0.0316978   |      -0       |  ops/s |   -0.00% |
|                                                Max Throughput |                system/syslog/dashboard |      0.0694738   |      0.0694735   |      -0       |  ops/s |   -0.00% |
|                                       50th percentile latency |                system/syslog/dashboard |    414.382       |    418.063       |       3.68086 |     ms |   +0.89% |
|                                       90th percentile latency |                system/syslog/dashboard |    710.115       |    688.727       |     -21.3875  |     ms |   -3.01% |
|                                      100th percentile latency |                system/syslog/dashboard |   2363.52        |   4750.61        |    2387.09    |     ms | +101.00% |
|                                  50th percentile service time |                system/syslog/dashboard |    406.338       |    405.544       |      -0.79385 |     ms |   -0.20% |
|                                  90th percentile service time |                system/syslog/dashboard |    466.415       |    454.924       |     -11.491   |     ms |   -2.46% |
|                                 100th percentile service time |                system/syslog/dashboard |   2362.42        |   4602.93        |    2240.51    |     ms |  +94.84% |
|                                                    error rate |                system/syslog/dashboard |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                     system/syslog/lens |      0.0113572   |      0.0113582   |       0       |  ops/s |    0.01% |
|                                               Mean Throughput |                     system/syslog/lens |      0.0240039   |      0.0240905   |       9e-05   |  ops/s |   +0.36% |
|                                             Median Throughput |                     system/syslog/lens |      0.0234542   |      0.0234733   |       2e-05   |  ops/s |   +0.08% |
|                                                Max Throughput |                     system/syslog/lens |      0.0583249   |      0.0583532   |       3e-05   |  ops/s |   +0.05% |
|                                       50th percentile latency |                     system/syslog/lens |   1207.61        |    721.159       |    -486.452   |     ms |  -40.28% |
|                                       90th percentile latency |                     system/syslog/lens |   6158.28        |   6292.43        |     134.153   |     ms |   +2.18% |
|                                      100th percentile latency |                     system/syslog/lens |  10520.8         |   8455.07        |   -2065.76    |     ms |  -19.63% |
|                                  50th percentile service time |                     system/syslog/lens |    433.978       |    422.717       |     -11.2613  |     ms |   -2.59% |
|                                  90th percentile service time |                     system/syslog/lens |   3987.15        |   4049.89        |      62.7404  |     ms |   +1.57% |
|                                 100th percentile service time |                     system/syslog/lens |   8202.82        |   6683.21        |   -1519.61    |     ms |  -18.53% |
|                                                    error rate |                     system/syslog/lens |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                        mysql/dashboard |      0.00224979  |      0.00336259  |       0.00111 |  ops/s |  +49.46% |
|                                               Mean Throughput |                        mysql/dashboard |      0.0163243   |      0.0165892   |       0.00026 |  ops/s |   +1.62% |
|                                             Median Throughput |                        mysql/dashboard |      0.0176932   |      0.0177524   |       6e-05   |  ops/s |   +0.33% |
|                                                Max Throughput |                        mysql/dashboard |      0.0245867   |      0.0245871   |       0       |  ops/s |    0.00% |
|                                       50th percentile latency |                        mysql/dashboard |    401.275       |    399.329       |      -1.94601 |     ms |   -0.48% |
|                                       90th percentile latency |                        mysql/dashboard |    461.174       |    446.634       |     -14.5406  |     ms |   -3.15% |
|                                      100th percentile latency |                        mysql/dashboard |   4248.56        |   4709.75        |     461.187   |     ms |  +10.86% |
|                                  50th percentile service time |                        mysql/dashboard |    398.951       |    396.583       |      -2.36778 |     ms |   -0.59% |
|                                  90th percentile service time |                        mysql/dashboard |    454.797       |    429.158       |     -25.6391  |     ms |   -5.64% |
|                                 100th percentile service time |                        mysql/dashboard |   3782.91        |   4275.39        |     492.473   |     ms |  +13.02% |
|                                                    error rate |                        mysql/dashboard |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                                  redis |      0.00958383  |      0.00958337  |      -0       |  ops/s |   -0.00% |
|                                               Mean Throughput |                                  redis |      0.0275553   |      0.0274462   |      -0.00011 |  ops/s |   -0.40% |
|                                             Median Throughput |                                  redis |      0.0264381   |      0.026438    |      -0       |  ops/s |   -0.00% |
|                                                Max Throughput |                                  redis |      0.0725947   |      0.072592    |      -0       |  ops/s |   -0.00% |
|                                       50th percentile latency |                                  redis |    417.702       |    414.127       |      -3.57458 |     ms |   -0.86% |
|                                       90th percentile latency |                                  redis |   1723.59        |   2070.49        |     346.896   |     ms |  +20.13% |
|                                       99th percentile latency |                                  redis |   6220.81        |   8114.34        |    1893.53    |     ms |  +30.44% |
|                                      100th percentile latency |                                  redis |   7394.47        |   9339           |    1944.53    |     ms |  +26.30% |
|                                  50th percentile service time |                                  redis |    412.661       |    409.487       |      -3.17371 |     ms |   -0.77% |
|                                  90th percentile service time |                                  redis |    455.106       |    441.569       |     -13.5377  |     ms |   -2.97% |
|                                  99th percentile service time |                                  redis |   3889.22        |   3807.43        |     -81.79    |     ms |   -2.10% |
|                                 100th percentile service time |                                  redis |   7392.12        |   9338.09        |    1945.97    |     ms |  +26.32% |
|                                                    error rate |                                  redis |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                             mysql/lens |      0.00680615  |      0.00680618  |       0       |  ops/s |    0.00% |
|                                               Mean Throughput |                             mysql/lens |      0.0164084   |      0.0163643   |      -4e-05   |  ops/s |   -0.27% |
|                                             Median Throughput |                             mysql/lens |      0.0127808   |      0.0127895   |       1e-05   |  ops/s |   +0.07% |
|                                                Max Throughput |                             mysql/lens |      0.0469888   |      0.0469927   |       0       |  ops/s |    0.01% |
|                                       50th percentile latency |                             mysql/lens |    404.016       |    404.41        |       0.39471 |     ms |   +0.10% |
|                                       90th percentile latency |                             mysql/lens |   4300.69        |   4170.68        |    -130.014   |     ms |   -3.02% |
|                                      100th percentile latency |                             mysql/lens |   5224.48        |   5512.99        |     288.509   |     ms |   +5.52% |
|                                  50th percentile service time |                             mysql/lens |    400.352       |    400.134       |      -0.21732 |     ms |   -0.05% |
|                                  90th percentile service time |                             mysql/lens |   2829           |   2855.33        |      26.3303  |     ms |   +0.93% |
|                                 100th percentile service time |                             mysql/lens |   5222.35        |   5511.59        |     289.242   |     ms |   +5.54% |
|                                                    error rate |                             mysql/lens |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/overview |      0.00513509  |      0.00513508  |      -0       |  ops/s |   -0.00% |
|                                               Mean Throughput |                    postgresql/overview |      0.061792    |      0.0618523   |       6e-05   |  ops/s |   +0.10% |
|                                             Median Throughput |                    postgresql/overview |      0.0631799   |      0.0632171   |       4e-05   |  ops/s |   +0.06% |
|                                                Max Throughput |                    postgresql/overview |      0.0893931   |      0.0893927   |      -0       |  ops/s |   -0.00% |
|                                       50th percentile latency |                    postgresql/overview |   1230.84        |   1231.89        |       1.04712 |     ms |   +0.09% |
|                                       90th percentile latency |                    postgresql/overview |   2272.46        |   2230.79        |     -41.6715  |     ms |   -1.83% |
|                                       99th percentile latency |                    postgresql/overview |   6530.01        |   5598.76        |    -931.243   |     ms |  -14.26% |
|                                      100th percentile latency |                    postgresql/overview |   7323.11        |   6797.17        |    -525.939   |     ms |   -7.18% |
|                                  50th percentile service time |                    postgresql/overview |   1153.96        |   1155.58        |       1.62109 |     ms |   +0.14% |
|                                  90th percentile service time |                    postgresql/overview |   1547.05        |   1639.04        |      91.9951  |     ms |   +5.95% |
|                                  99th percentile service time |                    postgresql/overview |   4738.04        |   3566.3         |   -1171.74    |     ms |  -24.73% |
|                                 100th percentile service time |                    postgresql/overview |   7163.95        |   6737.01        |    -426.931   |     ms |   -5.96% |
|                                                    error rate |                    postgresql/overview |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/duration |      0.0224523   |      0.0224544   |       0       |  ops/s |    0.01% |
|                                               Mean Throughput |                    postgresql/duration |      0.0365889   |      0.0361791   |      -0.00041 |  ops/s |   -1.12% |
|                                             Median Throughput |                    postgresql/duration |      0.0302364   |      0.0302073   |      -3e-05   |  ops/s |   -0.10% |
|                                                Max Throughput |                    postgresql/duration |      0.0801545   |      0.0801783   |       2e-05   |  ops/s |   +0.03% |
|                                       50th percentile latency |                    postgresql/duration |    416.413       |    425.624       |       9.21109 |     ms |   +2.21% |
|                                       90th percentile latency |                    postgresql/duration |    801.768       |    771.1         |     -30.6675  |     ms |   -3.82% |
|                                      100th percentile latency |                    postgresql/duration |   3985.51        |   5220.41        |    1234.9     |     ms |  +30.98% |
|                                  50th percentile service time |                    postgresql/duration |    410.434       |    407.951       |      -2.48297 |     ms |   -0.60% |
|                                  90th percentile service time |                    postgresql/duration |    495.168       |    486.407       |      -8.76105 |     ms |   -1.77% |
|                                 100th percentile service time |                    postgresql/duration |   3984.41        |   5123.21        |    1138.81    |     ms |  +28.58% |
|                                                    error rate |                    postgresql/duration |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                             bulk-index |    207.92        |    209.706       |       1.78587 | docs/s |   +0.86% |
|                                               Mean Throughput |                             bulk-index |  58833.2         |  60075.7         |    1242.42    | docs/s |   +2.11% |
|                                             Median Throughput |                             bulk-index |  59610.4         |  60908.9         |    1298.49    | docs/s |   +2.18% |
|                                                Max Throughput |                             bulk-index |  60014.6         |  61446.1         |    1431.47    | docs/s |   +2.39% |
|                                       50th percentile latency |                             bulk-index |    323.108       |    316           |      -7.10896 |     ms |   -2.20% |
|                                       90th percentile latency |                             bulk-index |    559.409       |    554.984       |      -4.42548 |     ms |   -0.79% |
|                                       99th percentile latency |                             bulk-index |    947.029       |    891.604       |     -55.4242  |     ms |   -5.85% |
|                                     99.9th percentile latency |                             bulk-index |   3474.96        |   3495.8         |      20.8468  |     ms |   +0.60% |
|                                    99.99th percentile latency |                             bulk-index |   5629.83        |   5631.95        |       2.12321 |     ms |   +0.04% |
|                                      100th percentile latency |                             bulk-index |   8841.69        |  10174.7         |    1333.03    |     ms |  +15.08% |
|                                  50th percentile service time |                             bulk-index |    323.951       |    318.043       |      -5.90779 |     ms |   -1.82% |
|                                  90th percentile service time |                             bulk-index |    566.043       |    557.025       |      -9.01837 |     ms |   -1.59% |
|                                  99th percentile service time |                             bulk-index |    945.919       |    892.909       |     -53.0108  |     ms |   -5.60% |
|                                99.9th percentile service time |                             bulk-index |   3477.48        |   3504.36        |      26.8718  |     ms |   +0.77% |
|                               99.99th percentile service time |                             bulk-index |   5605.52        |   5624.3         |      18.7841  |     ms |   +0.34% |
|                                 100th percentile service time |                             bulk-index |   8841.69        |  10174.7         |    1333.03    |     ms |  +15.08% |
|                                                    error rate |                             bulk-index |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                             field-caps |      2.42492     |      2.39366     |      -0.03125 |  ops/s |   -1.29% |
|                                               Mean Throughput |                             field-caps |      3.02836     |      2.96803     |      -0.06033 |  ops/s |   -1.99% |
|                                             Median Throughput |                             field-caps |      3.08178     |      3.01711     |      -0.06466 |  ops/s |   -2.10% |
|                                                Max Throughput |                             field-caps |      3.14591     |      3.09392     |      -0.05198 |  ops/s |   -1.65% |
|                                       50th percentile latency |                             field-caps |    606.82        |    615.303       |       8.48361 |     ms |   +1.40% |
|                                       90th percentile latency |                             field-caps |    672.35        |    657.275       |     -15.0751  |     ms |   -2.24% |
|                                       99th percentile latency |                             field-caps |    694.742       |    710.999       |      16.2574  |     ms |   +2.34% |
|                                      100th percentile latency |                             field-caps |    763.887       |    786.12        |      22.2323  |     ms |   +2.91% |
|                                  50th percentile service time |                             field-caps |    606.82        |    615.303       |       8.48361 |     ms |   +1.40% |
|                                  90th percentile service time |                             field-caps |    672.35        |    657.275       |     -15.0751  |     ms |   -2.24% |
|                                  99th percentile service time |                             field-caps |    694.742       |    710.999       |      16.2574  |     ms |   +2.34% |
|                                 100th percentile service time |                             field-caps |    763.887       |    786.12        |      22.2323  |     ms |   +2.91% |
|                                                    error rate |                             field-caps |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |        field-caps-exclude-empty-fields |      0.354446    |      0.370474    |       0.01603 |  ops/s |   +4.52% |
|                                               Mean Throughput |        field-caps-exclude-empty-fields |      0.37615     |      0.393918    |       0.01777 |  ops/s |   +4.72% |
|                                             Median Throughput |        field-caps-exclude-empty-fields |      0.377413    |      0.395607    |       0.01819 |  ops/s |   +4.82% |
|                                                Max Throughput |        field-caps-exclude-empty-fields |      0.378146    |      0.397802    |       0.01966 |  ops/s |   +5.20% |
|                                       50th percentile latency |        field-caps-exclude-empty-fields |   5292.47        |   5025.49        |    -266.979   |     ms |   -5.04% |
|                                       90th percentile latency |        field-caps-exclude-empty-fields |   5363.24        |   5112.96        |    -250.275   |     ms |   -4.67% |
|                                       99th percentile latency |        field-caps-exclude-empty-fields |   5404.22        |   5181.48        |    -222.737   |     ms |   -4.12% |
|                                      100th percentile latency |        field-caps-exclude-empty-fields |   5427.63        |   5237.67        |    -189.96    |     ms |   -3.50% |
|                                  50th percentile service time |        field-caps-exclude-empty-fields |   5292.47        |   5025.49        |    -266.979   |     ms |   -5.04% |
|                                  90th percentile service time |        field-caps-exclude-empty-fields |   5363.24        |   5112.96        |    -250.275   |     ms |   -4.67% |
|                                  99th percentile service time |        field-caps-exclude-empty-fields |   5404.22        |   5181.48        |    -222.737   |     ms |   -4.12% |
|                                 100th percentile service time |        field-caps-exclude-empty-fields |   5427.63        |   5237.67        |    -189.96    |     ms |   -3.50% |
|                                                    error rate |        field-caps-exclude-empty-fields |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                field-caps-index-filter |      2.09805     |      2.15672     |       0.05866 |  ops/s |   +2.80% |
|                                               Mean Throughput |                field-caps-index-filter |      2.20143     |      2.23604     |       0.03461 |  ops/s |   +1.57% |
|                                             Median Throughput |                field-caps-index-filter |      2.20826     |      2.24201     |       0.03376 |  ops/s |   +1.53% |
|                                                Max Throughput |                field-caps-index-filter |      2.22311     |      2.25491     |       0.0318  |  ops/s |   +1.43% |
|                                       50th percentile latency |                field-caps-index-filter |    890.331       |    876.606       |     -13.7244  |     ms |   -1.54% |
|                                       90th percentile latency |                field-caps-index-filter |    961.78        |    949.961       |     -11.8188  |     ms |   -1.23% |
|                                       99th percentile latency |                field-caps-index-filter |   1098.11        |   1082.87        |     -15.2397  |     ms |   -1.39% |
|                                      100th percentile latency |                field-caps-index-filter |   1171.06        |   1149.9         |     -21.1577  |     ms |   -1.81% |
|                                  50th percentile service time |                field-caps-index-filter |    890.331       |    876.606       |     -13.7244  |     ms |   -1.54% |
|                                  90th percentile service time |                field-caps-index-filter |    961.78        |    949.961       |     -11.8188  |     ms |   -1.23% |
|                                  99th percentile service time |                field-caps-index-filter |   1098.11        |   1082.87        |     -15.2397  |     ms |   -1.39% |
|                                 100th percentile service time |                field-caps-index-filter |   1171.06        |   1149.9         |     -21.1577  |     ms |   -1.81% |
|                                                    error rate |                field-caps-index-filter |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |   field-caps-index-filter-empty-fields |      0.315071    |      0.338204    |       0.02313 |  ops/s |   +7.34% |
|                                               Mean Throughput |   field-caps-index-filter-empty-fields |      0.328892    |      0.349713    |       0.02082 |  ops/s |   +6.33% |
|                                             Median Throughput |   field-caps-index-filter-empty-fields |      0.329896    |      0.350718    |       0.02082 |  ops/s |   +6.31% |
|                                                Max Throughput |   field-caps-index-filter-empty-fields |      0.330953    |      0.352704    |       0.02175 |  ops/s |   +6.57% |
|                                       50th percentile latency |   field-caps-index-filter-empty-fields |   6043.94        |   5679.7         |    -364.242   |     ms |   -6.03% |
|                                       90th percentile latency |   field-caps-index-filter-empty-fields |   6130.38        |   5774.62        |    -355.763   |     ms |   -5.80% |
|                                       99th percentile latency |   field-caps-index-filter-empty-fields |   6232.07        |   5922.44        |    -309.634   |     ms |   -4.97% |
|                                      100th percentile latency |   field-caps-index-filter-empty-fields |   6289.37        |   6068.5         |    -220.873   |     ms |   -3.51% |
|                                  50th percentile service time |   field-caps-index-filter-empty-fields |   6043.94        |   5679.7         |    -364.242   |     ms |   -6.03% |
|                                  90th percentile service time |   field-caps-index-filter-empty-fields |   6130.38        |   5774.62        |    -355.763   |     ms |   -5.80% |
|                                  99th percentile service time |   field-caps-index-filter-empty-fields |   6232.07        |   5922.44        |    -309.634   |     ms |   -4.97% |
|                                 100th percentile service time |   field-caps-index-filter-empty-fields |   6289.37        |   6068.5         |    -220.873   |     ms |   -3.51% |
|                                                    error rate |   field-caps-index-filter-empty-fields |      0           |      0           |       0       |      % |    0.00% |
|                                                Min Throughput |                 esql_from_all_limit_50 |      0.388699    |      0.388843    |       0.00014 |  ops/s |   +0.04% |
|                                               Mean Throughput |                 esql_from_all_limit_50 |      0.45163     |      0.446811    |      -0.00482 |  ops/s |   -1.07% |
|                                             Median Throughput |                 esql_from_all_limit_50 |      0.461227    |      0.45358     |      -0.00765 |  ops/s |   -1.66% |
|                                                Max Throughput |                 esql_from_all_limit_50 |      0.484858    |      0.482218    |      -0.00264 |  ops/s |   -0.54% |
|                                       50th percentile latency |                 esql_from_all_limit_50 |   1913.89        |   1896.73        |     -17.1597  |     ms |   -0.90% |
|                                       90th percentile latency |                 esql_from_all_limit_50 |   2073.65        |   2016.91        |     -56.7374  |     ms |   -2.74% |
|                                      100th percentile latency |                 esql_from_all_limit_50 |   2182.14        |   2362.99        |     180.85    |     ms |   +8.29% |
|                                  50th percentile service time |                 esql_from_all_limit_50 |   1913.89        |   1896.73        |     -17.1597  |     ms |   -0.90% |
|                                  90th percentile service time |                 esql_from_all_limit_50 |   2073.65        |   2016.91        |     -56.7374  |     ms |   -2.74% |
|                                 100th percentile service time |                 esql_from_all_limit_50 |   2182.14        |   2362.99        |     180.85    |     ms |   +8.29% |
|                                                    error rate |                 esql_from_all_limit_50 |      0           |      0           |       0       |      % |    0.00% |

@jordan-powers
Copy link
Copy Markdown
Contributor Author

Summary

This PR fixes a regression in the initial-indices step of the elastic/logs many-shards-quantitative benchmark.
Our baseline is 22a7491.
The initial regression was introduced in 9fe91ff.
The current fix is 397cd92.
The current fix but cherry-picked on top of the initial regression (to eliminate noise from other changes) is 687b291.

Label Ref initial-indices median throughput % diff from baseline
Baseline 22a7491 1190.46 docs/s -
Regression 9fe91ff 706.34 docs/s -40.67%
Fixed 397cd92 966.8 docs/s -18.79%
Fixed (cherry-picked) 687b291 1230.95 docs/s +3.40%

Since the performance returned to the baseline throughput (within noise) when the fix was cherry-picked on top of the original regression, I believe this fix completely resolves the original regression.
However, there still remains some slowdown due to other changes since that initial regression that likely should be investigated separately.

@jordan-powers jordan-powers marked this pull request as ready for review December 4, 2025 04:53
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Thanks for presenting the results in such a clear way.

However, there still remains some slowdown due to other changes since that initial regression that likely should be investigated separately.

Did flame graphs suggest what the other slowdown(s) could be?

@jordan-powers
Copy link
Copy Markdown
Contributor Author

Did flame graphs suggest what the other slowdown(s) could be?

I'm not certain, but possibly something security related? I'll post more details on the original investigation.

@jordan-powers jordan-powers merged commit c2286e1 into elastic:main Dec 4, 2025
35 checks passed
@jordan-powers jordan-powers deleted the optimize-sort-settings-default-providers-2 branch December 13, 2025 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants