Skip to content

[MOD-13606] CI: Enable LTO on Ubuntu Noble#8092

Merged
gdesmott merged 2 commits intomasterfrom
gd_lto_ci
Mar 16, 2026
Merged

[MOD-13606] CI: Enable LTO on Ubuntu Noble#8092
gdesmott merged 2 commits intomasterfrom
gd_lto_ci

Conversation

@gdesmott
Copy link
Copy Markdown
Collaborator

@gdesmott gdesmott commented Jan 20, 2026

Enable LTO in the CI:

  • Build on Platforms (only Ubuntu Noble atm)
  • PR
  • Build Artifacts
  • Benchmarks

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Release Notes

  • This PR requires release notes
  • This PR does not require release notes

Note

Medium Risk
Medium risk because it changes CI build flags and LLVM toolchain installation/versioning, which can affect build determinism and introduce platform-specific failures.

Overview
Enables Link Time Optimization (LTO) in CI for ubuntu:noble by extending task-get-config.yml with an enable_lto output and setting it to 1 for Noble on both x86_64 and aarch64.

Updates task-test.yml to export LTO=1 only when the platform enables it, and automatically disables LTO for sanitizer and coverage runs.

Introduces centralized LLVM versioning via .install/LLVM_VERSION.sh (set to 21), updates .install/install_llvm.sh to use that version on Linux while pinning macOS to LLVM 18, and ensures Ubuntu 24.04 setup installs LLVM (ubuntu_24.04.sh calls install_llvm.sh).

Written by Cursor Bugbot for commit 793bf88. This will update automatically on new commits. Configure here.

@jit-ci
Copy link
Copy Markdown

jit-ci bot commented Jan 20, 2026

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Jan 20, 2026

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 35 stable tests between versions.

You can check a comparison in detail via the grafana link

##

Details

Tests with No Significant Changes (35 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison gd_lto_ci (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby N/A N/A 0.0%
ftsb-10K-enwiki_abstract-hashes-term-prefix N/A N/A 0.0%
ftsb-10K-enwiki_abstract-hashes-term-suffix 2288 +- 1.0% (20 datapoints) 2234 -2.4% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie N/A N/A 0.0%
ftsb-10K-enwiki_abstract-hashes-term-wildcard N/A N/A 0.0%
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml N/A N/A 0.0%
ftsb-10K-enwiki_pages-hashes-load N/A N/A 0.0%
ftsb-10K-multivalue-numeric-json N/A N/A 0.0%
ftsb-1K-enwiki_abstract-hashes-term-contains N/A N/A 0.0%
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query N/A N/A 0.0% waterline=15.5%.
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable N/A N/A 0.0% waterline=24.3%.
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query N/A N/A 0.0% waterline=9.1%.
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable N/A N/A 0.0% waterline=13.3%.
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query N/A N/A 0.0% waterline=16.9%.
ftsb-1M-enwiki_abstract-hashes-load N/A N/A 0.0%
ftsb-1M-nyc_taxis-ftadd-load N/A N/A 0.0%
ftsb-1M-nyc_taxis-hashes-load N/A N/A 0.0%
search-aggregate-post-filter-simple.yml N/A N/A 0.0%
search-filtering-tag-numeric N/A N/A 0.0% waterline=13.4%.
search-filtering-tag-numeric-filter-pipeline N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-fulltext-aggregate-sortby-limit-0-100 N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie N/A N/A 0.0%
search-ftsb-1700K-docs-union-iterators-q3 N/A N/A 0.0%
search-ftsb-370K-docs-union-iterators-q4 N/A N/A 0.0%
search-ftsb-5200K-docs-union-iterators-q1 N/A N/A 0.0%
search-ftsb-5500K-docs-union-iterators-q2 N/A N/A 0.0%
search-high-cardinality-negation-term-baseline 34 +- 1.5% (20 datapoints) 34 -0.4% No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms N/A N/A 0.0%
search-numeric-optimize N/A N/A 0.0%
search-numeric-sortby-desc N/A N/A 0.0% waterline=33.0%.
search-numeric-sortby-desc-optimize N/A N/A 0.0%
search-numeric-sortby-optimize N/A N/A 0.0%
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 604 +- 2.8% (20 datapoints) 605 0.3% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter N/A N/A 0.0%
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15474 +- 1.4% (20 datapoints) 15570 0.6% No Change

@jit-ci
Copy link
Copy Markdown

jit-ci bot commented Mar 16, 2026

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Mar 16, 2026

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 30 stable tests between versions.

You can check a comparison in detail via the grafana link

##

Details

Tests with No Significant Changes (30 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison gd_lto_ci (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby N/A N/A 0.0%
ftsb-10K-enwiki_abstract-hashes-term-suffix N/A N/A 0.0%
ftsb-10K-enwiki_abstract-hashes-term-wildcard N/A N/A 0.0%
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml N/A N/A 0.0%
ftsb-10K-multivalue-numeric-json N/A N/A 0.0%
ftsb-10K-singlevalue-numeric-json N/A N/A 0.0%
ftsb-1K-enwiki_abstract-hashes-term-contains N/A N/A 0.0%
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query N/A N/A 0.0%
ftsb-1M-enwiki_abstract-hashes-load N/A N/A 0.0%
ftsb-1M-nyc_taxis-ftadd-load N/A N/A 0.0%
ftsb-1M-nyc_taxis-hashes-load N/A N/A 0.0%
search-filtering-tag-numeric-filter-pipeline N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-fulltext-search-sortby-limit-0-100 N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie N/A N/A 0.0%
search-ftsb-1700K-docs-union-iterators-q3 8.2 +- 2.3% (20 datapoints) 8.4 1.8% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable N/A N/A 0.0% waterline=9.8%.
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-one-indexed-field N/A N/A 0.0%
search-ftsb-370K-docs-union-iterators-q4 N/A N/A 0.0%
search-ftsb-5200K-docs-union-iterators-q1 N/A N/A 0.0%
search-ftsb-5500K-docs-union-iterators-q2 N/A N/A 0.0%
search-geo N/A N/A 0.0%
search-high-cardinality-negation-term-baseline N/A N/A 0.0%
search-high-cardinality-negation-term-comparison_union_all_other_terms N/A N/A 0.0%
search-numeric N/A N/A 0.0% waterline=30.3%.
search-numeric-sortby N/A N/A 0.0% waterline=26.4%.
search-numeric-sortby-optimize N/A N/A 0.0%
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 604 +- 2.8% (20 datapoints) 605 0.3% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 153 +- 7.8% (20 datapoints) 161 5.0% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15474 +- 1.4% (20 datapoints) 15570 0.6% No Change

@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Mar 16, 2026

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 35 stable tests between versions.
  • Detected a total of 7 highly unstable benchmarks (7 baseline).
  • Detected a total of 2 regressions bellow the regression water line 8.0%.

You can check a comparison in detail via the grafana link

Performance Regressions and Issues - Comparison between master and gd_lto_ci.

Time Period from a month ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison gd_lto_ci (median obs. +- std.dev) % change (higher-better) Note
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 53 +- 26.8% UNSTABLE (20 datapoints) 40 -24.7% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 23.8% (baseline CV=16.4%); client: client latency stable; only server side confirms regression (client side stable) - insufficient evidence
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2888 +- 9.1% (20 datapoints) 2538 -12.1% waterline=9.1%. REGRESSION
search-ftsb-1M-enwiki_abstract-hashes-gc 153 +- 10.4% UNSTABLE (20 datapoints) 135 -12.0% UNSTABLE (baseline high variance); server: hset p50 increased 9.6% (baseline CV=3.8%); client: Latency increased 13.8% (baseline CV=11.1%); only client side confirms regression (server side stable) - insufficient evidence
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 916 +- 16.9% UNSTABLE (20 datapoints) 809 -11.7% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 14.5% (baseline CV=18.2%); client: OverallQuantiles.allCommands.q50 increased 9.3% (baseline CV=16.4%); only server side confirms regression (client side stable) - insufficient evidence
search-numeric-sortby-desc-optimize 31 +- 7.1% (20 datapoints) 28 -9.5% REGRESSION
search-filtering-tag-numeric 248 +- 13.4% UNSTABLE (20 datapoints) 259 4.3% UNSTABLE (baseline high variance); server: p50 latency stable; client: client latency stable; neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query 346 +- 14.7% UNSTABLE (20 datapoints) 363 4.8% UNSTABLE (baseline high variance); server: p50 latency stable; client: client latency stable; neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1144 +- 13.3% UNSTABLE (20 datapoints) 1297 13.4% UNSTABLE (baseline high variance); server: p50 latency stable; client: OverallQuantiles.allCommands.q50 decreased 11.8% (baseline CV=12.6%); neither server nor client side confirms regression
search-numeric-sortby-desc 2136 +- 33.9% UNSTABLE (20 datapoints) 3782 77.0% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 44.7% (baseline CV=24.9%); client: Latency decreased 43.7% (baseline CV=24.3%); neither server nor client side confirms regression
Tests with No Significant Changes (35 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison gd_lto_ci (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 71 +- 2.9% (20 datapoints) 70 -1.2% No Change
ftsb-10K-enwiki_abstract-hashes-term-prefix 6099 +- 4.7% (20 datapoints) 6223 2.0% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix 2288 +- 1.0% (20 datapoints) 2234 -2.4% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 16195 +- 1.4% (20 datapoints) 15828 -2.3% No Change
ftsb-10K-enwiki_abstract-hashes-term-wildcard 8919 +- 5.7% (20 datapoints) 8483 -4.9% potential REGRESSION
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1021 +- 5.7% (20 datapoints) 963 -5.7% potential REGRESSION
ftsb-10K-enwiki_pages-hashes-load 61484 +- 5.2% (20 datapoints) 59430 -3.3% potential REGRESSION
ftsb-10K-multivalue-numeric-json 984 +- 1.4% (20 datapoints) 997 1.3% No Change
ftsb-10K-singlevalue-numeric-json N/A N/A 0.0%
ftsb-1K-enwiki_abstract-hashes-term-contains 1965 +- 1.2% (20 datapoints) 1962 -0.2% No Change
ftsb-1M-enwiki_abstract-hashes-load 22431 +- 4.4% (20 datapoints) 22806 1.7% No Change
ftsb-1M-nyc_taxis-ftadd-load 29299 +- 2.6% (20 datapoints) 29013 -1.0% No Change
ftsb-1M-nyc_taxis-hashes-load 30910 +- 3.0% (20 datapoints) 31650 2.4% No Change
search-aggregate-post-filter-simple.yml 16946 +- 0.9% (20 datapoints) 17009 0.4% No Change
search-filtering-tag-numeric-filter-pipeline 10550 +- 1.4% (20 datapoints) 10465 -0.8% No Change
search-ftsb-10K-enwiki_abstract-hashes-fulltext-aggregate-sortby-limit-0-100 776 +- 2.2% (20 datapoints) 778 0.2% No Change
search-ftsb-10K-enwiki_abstract-hashes-fulltext-search-sortby-limit-0-100 N/A N/A 0.0%
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 14071 +- 1.1% (20 datapoints) 14149 0.6% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie N/A N/A 0.0%
search-ftsb-1700K-docs-union-iterators-q3 8.2 +- 2.3% (20 datapoints) 8.40 1.8% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable N/A N/A 0.0%
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-one-indexed-field N/A N/A 0.0%
search-ftsb-370K-docs-union-iterators-q4 8.5 +- 1.8% (20 datapoints) 8.50 0.6% No Change
search-ftsb-5200K-docs-union-iterators-q1 0.87 +- 1.6% (20 datapoints) 0.89 2.3% No Change
search-ftsb-5500K-docs-union-iterators-q2 1.2 +- 2.5% (20 datapoints) 1.30 2.4% No Change
search-geo N/A N/A 0.0%
search-high-cardinality-negation-term-baseline 34 +- 1.5% (20 datapoints) 34 -0.4% No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms 14 +- 1.5% (20 datapoints) 14 0.0% No Change
search-numeric N/A N/A 0.0%
search-numeric-optimize 7963 +- 0.7% (20 datapoints) 7944 -0.2% No Change
search-numeric-sortby N/A N/A 0.0%
search-numeric-sortby-optimize 29 +- 6.6% (20 datapoints) 30 3.7% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 604 +- 2.8% (20 datapoints) 605 0.3% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 153 +- 7.8% (20 datapoints) 161 5.0% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15474 +- 1.4% (20 datapoints) 15570 0.6% No Change

That's the version currently required to build with LTO.

Also: move llvm version to its own file so we can have the version in a single place.
memark
memark previously approved these changes Mar 16, 2026
@memark
Copy link
Copy Markdown
Collaborator

memark commented Mar 16, 2026

This PR only enable LTO on Ubuntu Noble atm.

Maybe we could rename the PR to reflect this before merging?

@gdesmott gdesmott changed the title [MOD-13606] Enable LTO in all CI workflows [MOD-13606] CI: Enable LTO on Ubuntu Noble Mar 16, 2026
@gdesmott gdesmott marked this pull request as ready for review March 16, 2026 13:12
@gdesmott gdesmott requested a review from LukeMathWalker March 16, 2026 13:20
LOG: 1
CLEAR_LOGS: 0
ENABLE_ASSERT: 1
LTO: ${{ needs.get-config.outputs.enable_lto }}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want LTO on coverage/sanitize tasks.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

But I don't remember why not?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For coverage, the fewer optimisations you have, the better the report (as a rule of thumb).
For sanitiser, it's complex/slow enough as is, I'd prefer to avoid adding yet another source of build complexity there. But I could be convinced otherwise.

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

That's the only platform with a new enough toolchain for now.
@sonarqubecloud
Copy link
Copy Markdown

@gdesmott gdesmott enabled auto-merge March 16, 2026 14:26
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 16, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.35%. Comparing base (4ce1cf5) to head (793bf88).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8092      +/-   ##
==========================================
- Coverage   82.39%   82.35%   -0.05%     
==========================================
  Files         449      449              
  Lines       62470    62501      +31     
  Branches    19805    19805              
==========================================
  Hits        51471    51471              
- Misses      10805    10836      +31     
  Partials      194      194              
Flag Coverage Δ
flow 83.73% <ø> (-0.06%) ⬇️
unit 51.15% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gdesmott gdesmott added this pull request to the merge queue Mar 16, 2026
Merged via the queue into master with commit 0668252 Mar 16, 2026
51 checks passed
@gdesmott gdesmott deleted the gd_lto_ci branch March 16, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants