Skip to content

[Native] Gradle-related tweaks to improve handling of the simdvec native library#144539

Merged
ldematte merged 3 commits intoelastic:mainfrom
ldematte:native/gradle-tweaks
Mar 26, 2026
Merged

[Native] Gradle-related tweaks to improve handling of the simdvec native library#144539
ldematte merged 3 commits intoelastic:mainfrom
ldematte:native/gradle-tweaks

Conversation

@ldematte
Copy link
Copy Markdown
Contributor

@ldematte ldematte commented Mar 19, 2026

This PR introduces a couple of changes related to native libraries build;

  1. In the script used to build and publish the native vec library, force gradle to rebuild native intermediate files to ensure a clean and up-to-date binary. Under some conditions, we ended with object files that were not at the latest version. Probably these were extreme conditions (locally built libraries, experimenting different solutions), but better be safe. The library is tiny and takes seconds to build anyway.

  2. Update gradle build files of benchmarks and KnnIndexTester to explicitly depend on native-libraries. Most of the code in benchmarks and KnnIndexTester depends on the native simdvec library, and it happened in the past that we unknowingly run with older, not updated versions of the library. In particular, we want the :libs:native:native-libraries:extractLibs to appear in the task graph for both :benchmarks:run and :qa:vector:checkVec.

@ldematte ldematte added WIP test-arm Pull Requests that should be tested against arm agents labels Mar 19, 2026
@ldematte ldematte changed the title Force gradle to rebuild native intermediate files [Native] Gradle-related tweaks to improve handling of the simdvec native library Mar 19, 2026
@ldematte ldematte added :Delivery/Build Build or test infrastructure >test Issues or PRs that are addressing/adding tests and removed WIP labels Mar 19, 2026
@ldematte ldematte marked this pull request as ready for review March 19, 2026 15:41
@elasticsearchmachine elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Mar 19, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 19, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: d9affe14-5a7e-4b18-b7b4-dbf17fea551a

📥 Commits

Reviewing files that changed from the base of the PR and between cc5db9d and 573a6d0.

📒 Files selected for processing (5)
  • benchmarks/build.gradle
  • libs/simdvec/native/Dockerfile.aarch64
  • libs/simdvec/native/Dockerfile.amd64
  • libs/simdvec/native/publish_vec_binaries.sh
  • qa/vector/build.gradle

📝 Walkthrough

Walkthrough

This change refactors how native library dependencies are consumed and passed to build tasks. The native library dependency declarations are updated to use a new nativeLib configuration, and the path to native libraries is now supplied through lazy Gradle providers rather than hardcoded filesystem paths. Additionally, --no-build-cache flags are added to all native library build invocations across multiple Dockerfiles and scripts to ensure fresh builds.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • 🛠️ Update Documentation: Commit on current branch
  • 🛠️ Update Documentation: Create PR
📝 Coding Plan
  • Generate coding plan for human review comments

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

CodeRabbit can generate a title for your PR based on the changes.

Add @coderabbitai placeholder anywhere in the title of your PR and CodeRabbit will replace it with a title based on the changes in the PR. You can change the placeholder by changing the reviews.auto_title_placeholder setting.

@ldematte ldematte requested a review from ChrisHegarty March 26, 2026 10:34
Copy link
Copy Markdown
Contributor

@ChrisHegarty ChrisHegarty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @ldematte

@ldematte ldematte merged commit 568254d into elastic:main Mar 26, 2026
42 checks passed
thecoop added a commit that referenced this pull request Mar 26, 2026
szybia added a commit to szybia/elasticsearch that referenced this pull request Mar 26, 2026
* upstream/main: (146 commits)
  Revert "[Native] Gradle-related tweaks to improve handling of the simdvec native library (elastic#144539)"
  Fix ArrayIndexOutOfBoundsException in fetch phase with partial results (elastic#144385)
  ESQL: Correctly manage NULL data type for SUM (elastic#144942)
  [ESQL] Fixes GroupedTopNBenchmark not executing (elastic#144944)
  Fix reader context leak when query response serialization fails (elastic#144708)
  Validate individual offset values in BULK_OFFSETS bounds checks (elastic#144643)
  Merge main21 source set into main in simdvec (elastic#144921)
  [TEST] Unmute TsidExtractingIdFieldMapperTests (elastic#144848)
  [Native] Gradle-related tweaks to improve handling of the simdvec native library (elastic#144539)
  Fix `ThreadedActionListenerTests#testRejectionHandling` (elastic#144795)
  Add new DLM Frozen Tier Transition execution plugin and service (elastic#144595)
  Prometheus: execute query_range via parsed EsqlStatement plan (elastic#144416)
  Investigate `testBulkIndexingRequestSplitting` failure (elastic#144766)
  Add test utility for wrapping directories in FilterDirectory layer (elastic#143563)
  Fix ES|QL decay tests with negative scale (elastic#144657)
  Fix circuit breaker leak in percolator query construction (elastic#144827)
  Use XPerFieldDocValuesFormat in AbstractTSDBSyntheticIdCodec (elastic#144744)
  [DOCS] Document how reindex work in CPS (elastic#144016)
  Fix Int4 vector library tests failing on Java 21 (elastic#144830)
  [DiskBBQ] Fix index sorting on flush (elastic#144938)
  ...
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 26, 2026
…ive library (elastic#144539)

This PR introduces a couple of changes related to native libraries build;

    In the script used to build and publish the native vec library, force gradle to rebuild native intermediate files to ensure a clean and up-to-date binary. Under some conditions, we ended with object files that were not at the latest version. Probably these were extreme conditions (locally built libraries, experimenting different solutions), but better be safe. The library is tiny and takes seconds to build anyway.

    Update gradle build files of benchmarks and KnnIndexTester to explicitly depend on native-libraries. Most of the code in benchmarks and KnnIndexTester depends on the native simdvec library, and it happened in the past that we unknowingly run with older, not updated versions of the library. In particular, we want the :libs:native:native-libraries:extractLibs to appear in the task graph for both :benchmarks:run and :qa:vector:checkVec.
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 26, 2026
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 27, 2026
…ive library (elastic#144539)

This PR introduces a couple of changes related to native libraries build;

    In the script used to build and publish the native vec library, force gradle to rebuild native intermediate files to ensure a clean and up-to-date binary. Under some conditions, we ended with object files that were not at the latest version. Probably these were extreme conditions (locally built libraries, experimenting different solutions), but better be safe. The library is tiny and takes seconds to build anyway.

    Update gradle build files of benchmarks and KnnIndexTester to explicitly depend on native-libraries. Most of the code in benchmarks and KnnIndexTester depends on the native simdvec library, and it happened in the past that we unknowingly run with older, not updated versions of the library. In particular, we want the :libs:native:native-libraries:extractLibs to appear in the task graph for both :benchmarks:run and :qa:vector:checkVec.
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 27, 2026
mamazzol pushed a commit to mamazzol/elasticsearch that referenced this pull request Mar 30, 2026
…ive library (elastic#144539)

This PR introduces a couple of changes related to native libraries build;

    In the script used to build and publish the native vec library, force gradle to rebuild native intermediate files to ensure a clean and up-to-date binary. Under some conditions, we ended with object files that were not at the latest version. Probably these were extreme conditions (locally built libraries, experimenting different solutions), but better be safe. The library is tiny and takes seconds to build anyway.

    Update gradle build files of benchmarks and KnnIndexTester to explicitly depend on native-libraries. Most of the code in benchmarks and KnnIndexTester depends on the native simdvec library, and it happened in the past that we unknowingly run with older, not updated versions of the library. In particular, we want the :libs:native:native-libraries:extractLibs to appear in the task graph for both :benchmarks:run and :qa:vector:checkVec.
mamazzol pushed a commit to mamazzol/elasticsearch that referenced this pull request Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team >test Issues or PRs that are addressing/adding tests test-arm Pull Requests that should be tested against arm agents v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants