Skip to content

Add Linux-based cross-compilation pipeline for libvec native binaries#144845

Merged
ldematte merged 26 commits intoelastic:mainfrom
ldematte:native/docker-cross-compile
Apr 2, 2026
Merged

Add Linux-based cross-compilation pipeline for libvec native binaries#144845
ldematte merged 26 commits intoelastic:mainfrom
ldematte:native/docker-cross-compile

Conversation

@ldematte
Copy link
Copy Markdown
Contributor

@ldematte ldematte commented Mar 24, 2026

This PR adds a Docker-based cross-compilation pipeline for building libvec for all three platforms from a single Linux/amd64 container, replacing the previous Mac-dependent workflow. It builds on top of #145066 (merged), which introduced the tinystd headers and builtin replacements.

The three output binaries are built using:

Platform File Compiler
darwin-aarch64 libvec.dylib clang++-18 → Mach-O via lld
linux-aarch64 libvec.so aarch64-linux-gnu-g++-14
linux-x64 libvec.so g++-14 (native)

Docker image

debian:trixie-slim
       │
       ▼
es-native-cross-toolchain:N     Dockerfile.cross-toolchain
  clang-18 + lld-18             (built from any machine, pushed when
  gcc-14 + g++-14                toolchain versions change)
  g++-14-aarch64-linux-gnu
  make
       │
       ▼ (docker run --platform linux/amd64 -v $(pwd):/workspace)
  make all → libvec.dylib + libvec.so × 2

The Docker image is a single public layer — no macOS SDK is included. This avoids any licensing concerns around redistributing Apple's proprietary SDK in a Docker image.

How does Darwin cross-compilation work without the SDK?

The Makefile uses -nostdinc to block host (glibc) headers from leaking into the Darwin build, then adds the tinystd headers (from #145066) and clang's freestanding builtins (arm_neon.h, stdint.h, etc.) via explicit -isystem paths:

CLANG = clang++-18 --target=arm64-apple-macos14 -nostdinc \
          -isystem src/vec/headers/darwin/tinystd \
          -isystem $(CLANG_RESOURCE)/include

At link time, -nostdlib -Wl,-undefined,dynamic_lookup avoids needing SDK stub libraries. The only remaining runtime symbols (_bzero, dyld_stub_binder) are resolved by macOS's dyld at load time.

Linux builds are unaffected — they use the real libstdc++ from GCC as before.

Additional cleanup in this PR

  • Added -DNDEBUG and -Winline to CXXFLAGS
  • Removed redundant #pragma target push/pop from _2.cpp files (the Makefile compiles them with the correct -march)
  • Removed standalone Gradle build (gradlew/build.gradle) — replaced by make local/make install
  • Removed obsolete Dockerfiles (Dockerfile.aarch64, Dockerfile.amd64)

ISA tiers

Source files are split by naming convention. *_2.cpp files implement
tier-2 ISA paths and are compiled with a higher -march; all other files
use the baseline.

Target Tier 1 (*_1.cpp, caps.cpp, …) Tier 2 (*_2.cpp)
darwin-aarch64 armv8.2-a+dotprod armv8.2-a+sve
linux-aarch64 armv8.2-a+dotprod armv8.2-a+sve
linux-x64 core-avx2 icelake-client

The Makefile compiles each group to .o files separately, then links them into the shared library.


Workflows

Publish a new release

Requires ARTIFACTORY_API_KEY to be set. Bump VERSION in
publish_vec_binaries.sh first.

./publish_vec_binaries.sh

Local test build (no upload)

Produces vec-<VERSION>-local.zip in the working directory.

./publish_vec_binaries.sh --local

Local build with upload

Build using the local Docker image, but upload to Artifactory.
Useful when you don't have access to the registry image but want to publish.

./publish_vec_binaries.sh --local --force-upload

Update compiler versions

Bump VERSION in build_cross_toolchain_image.sh, update package
versions in Dockerfile.cross-toolchain, then:

./build_cross_toolchain_image.sh          # build and push
./build_cross_toolchain_image.sh --local  # build locally only

After pushing, update TOOLCHAIN_IMAGE in publish_vec_binaries.sh.


Verification

Tested on all three platforms:

  • darwin-aarch64 (Mac): Docker + tinystd — all JDKVectorLibrary tests pass
  • linux-aarch64 (ARM EC2 c8gd.xlarge): native make + g++-14 — all tests pass, binary byte-identical to previous artifactory release
  • linux-x64 (Intel EC2 c8i.2xlarge): native make + g++-14 — all tests pass

Local development

The standalone Gradle build (gradlew/build.gradle) has been removed. Use the Makefile instead:

# Build the library for the current platform (macOS, Linux x64, or Linux ARM)
make local

# Build and copy to the Gradle test directory
make install

# Run tests with the locally built library
LOCAL_VEC_BINARY_OS=darwin ./gradlew :libs:native:test --tests "..."   # macOS
LOCAL_VEC_BINARY_OS=linux  ./gradlew :libs:native:test --tests "..."   # Linux

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ChrisHegarty
Copy link
Copy Markdown
Contributor

I assume that the actual built artefacts (the 3 native libraries) are identical when built with this new mechanism? We should confirm this.

@ldematte
Copy link
Copy Markdown
Contributor Author

I assume that the actual built artefacts (the 3 native libraries) are identical when built with this new mechanism? We should confirm this.

Confirmed with objdump

@brianseeders
Copy link
Copy Markdown
Contributor

This looks cool, but I don't think we can bundle the MacOS SDK into Docker images and redistribute it like this.

@ldematte
Copy link
Copy Markdown
Contributor Author

I was worried about licensing issues too.. what if we make this internal? (can we?)
In parallel, I'm investigating not needing the macOS SDK at all. I wanted to make this a follow up, but it makes sense to do it immediately.

@brianseeders
Copy link
Copy Markdown
Contributor

I think that just using the SDK outside of real Mac hardware is a violation of the license, so making it internal doesn't really matter

@brianseeders
Copy link
Copy Markdown
Contributor

(I could be wrong though)

@ldematte
Copy link
Copy Markdown
Contributor Author

I think that just using the SDK outside of real Mac hardware is a violation of the license, so making it internal doesn't really matter

I think you might be right, so I changed approach. No MacOS SDK.

@ldematte ldematte requested review from brianseeders and removed request for mark-vieira March 26, 2026 14:49
@ldematte
Copy link
Copy Markdown
Contributor Author

@brianseeders, can you check build_cross_toolchain_image.sh and tell me which names/accounts/endpoints I should be using? Of course the push of docker.elastic.co/es-dev/es-native-cross-toolchain:1 to docker.elastic.co/es-dev/es-native-cross-toolchain fails for me with an authentication error, at the moment.

@ldematte ldematte marked this pull request as ready for review March 26, 2026 15:09
@ldematte ldematte requested a review from a team as a code owner March 26, 2026 15:09
@elasticsearchmachine elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Mar 26, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 26, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change replaces the Gradle-based native build with a Makefile-driven cross-compilation workflow. Per-architecture Dockerfiles for amd64/aarch64 were removed and a new cross-toolchain Dockerfile and image build script were added. A Makefile now builds libvec for darwin-aarch64, linux-aarch64, and linux-x64 with tiered object sets. publish_vec_binaries.sh was rewritten to use the cross-toolchain Docker build and gained --local and --force-upload flags. Compiler pragmas forcing AVX-512/SVE per-translation-unit were removed from several sources. Gradle native build files and the Gradle wrapper for libs/simdvec/native were deleted; vec artifact version was updated in libs/native/libraries/build.gradle.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • 🛠️ Update Documentation: Commit on current branch
  • 🛠️ Update Documentation: Create PR

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@libs/simdvec/native/build_cross_toolchain_image.sh`:
- Around line 24-26: The script currently only recognizes "--local" by checking
"$1" and leaves typos to proceed to the docker push logic; change argument
parsing so any non-empty, non-"--local" argument causes an immediate error and
exit (fail-closed). Specifically, validate "$1" at the top where LOCAL is set:
if "$1" is empty allow normal behavior, if "$1" == "--local" set LOCAL=true,
else print a clear usage/error and exit 1; also apply the same strict check to
the later argument-handling region that controls the docker push (the block
referencing LOCAL and the docker push commands) so unknown args cannot fall
through to the push. Ensure the error message references the accepted flag
("--local") and that the exit code is non-zero.

In `@libs/simdvec/native/Dockerfile.cross-toolchain`:
- Around line 21-44: The Dockerfile currently leaves the container running as
root after package installation; to fix, after the package-installing RUN that
ends with "rm -rf /var/lib/apt/lists/*" add steps to create a non-root user and
group (e.g., "simdvec" or "builder"), create a home directory, set appropriate
ownership of relevant workspace/build directories, set HOME and any required
environment variables, and switch to that user with the USER instruction before
the image is finalized; locate these changes around the package installation RUN
block in Dockerfile.cross-toolchain and ensure subsequent build steps run as the
non-root user.

In `@libs/simdvec/native/publish_vec_binaries.sh`:
- Line 72: The curl upload command that pipes the zip archive into curl (the
line containing: (cd "$TEMP" && zip -rq - .) | curl -sS -X PUT -H
"X-JFrog-Art-Api: ${ARTIFACTORY_API_KEY}" --data-binary `@-` --location
"${ARTIFACTORY_REPOSITORY}/org/elasticsearch/vec/${VERSION}/vec-${VERSION}.zip")
should include the --fail flag so HTTP errors cause a non‑zero exit; update that
curl invocation to add --fail (e.g., curl -sS --fail -X PUT ...) so failures are
detected and the script exits appropriately.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 0013c400-d07b-447b-b4cf-16fb837bcf3c

📥 Commits

Reviewing files that changed from the base of the PR and between e260014 and 6b0a849.

📒 Files selected for processing (24)
  • libs/native/libraries/build.gradle
  • libs/simdvec/native/.gitignore
  • libs/simdvec/native/Dockerfile.aarch64
  • libs/simdvec/native/Dockerfile.amd64
  • libs/simdvec/native/Dockerfile.cross-toolchain
  • libs/simdvec/native/Makefile
  • libs/simdvec/native/build_cross_toolchain_image.sh
  • libs/simdvec/native/publish_vec_binaries.sh
  • libs/simdvec/native/src/vec/c/aarch64/caps.cpp
  • libs/simdvec/native/src/vec/c/aarch64/score_1.cpp
  • libs/simdvec/native/src/vec/c/aarch64/vec_1.cpp
  • libs/simdvec/native/src/vec/c/aarch64/vec_2.cpp
  • libs/simdvec/native/src/vec/c/amd64/caps.cpp
  • libs/simdvec/native/src/vec/c/amd64/score_1.cpp
  • libs/simdvec/native/src/vec/c/amd64/score_2.cpp
  • libs/simdvec/native/src/vec/c/amd64/vec_1.cpp
  • libs/simdvec/native/src/vec/c/amd64/vec_2.cpp
  • libs/simdvec/native/src/vec/c/amd64/vec_i4_2.cpp
  • libs/simdvec/native/src/vec/headers/darwin/tinystd/algorithm
  • libs/simdvec/native/src/vec/headers/darwin/tinystd/limits
  • libs/simdvec/native/src/vec/headers/darwin/tinystd/type_traits
  • libs/simdvec/native/src/vec/headers/darwin/tinystd/utility
  • libs/simdvec/native/src/vec/headers/score_common.h
  • libs/simdvec/native/src/vec/headers/vec_common.h
💤 Files with no reviewable changes (7)
  • libs/simdvec/native/src/vec/c/amd64/caps.cpp
  • libs/simdvec/native/src/vec/headers/vec_common.h
  • libs/simdvec/native/Dockerfile.amd64
  • libs/simdvec/native/src/vec/c/amd64/vec_2.cpp
  • libs/simdvec/native/Dockerfile.aarch64
  • libs/simdvec/native/src/vec/c/amd64/vec_i4_2.cpp
  • libs/simdvec/native/src/vec/c/aarch64/vec_2.cpp

ldematte and others added 2 commits March 26, 2026 16:26
… local/install

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ldematte ldematte requested a review from a team as a code owner March 26, 2026 15:36
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Member

@thecoop thecoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIPIT

@thecoop
Copy link
Copy Markdown
Member

thecoop commented Mar 27, 2026

Can we add the -local.zip file to gitignore?

@ldematte
Copy link
Copy Markdown
Contributor Author

Can we add the -local.zip file to gitignore?

Already done. Is it not working for you?

@thecoop
Copy link
Copy Markdown
Member

thecoop commented Mar 27, 2026

Argh, sorry, messing around with branches. Nevermind.

@ldematte ldematte added the test-arm Pull Requests that should be tested against arm agents label Apr 2, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@libs/simdvec/native/publish_vec_binaries.sh`:
- Around line 42-45: The publish script sets TOOLCHAIN_IMAGE to
docker.elastic.co/es-dev/es-native-cross-toolchain:1 which mismatches the image
published by build_cross_toolchain_image.sh
(docker.elastic.co/elasticsearch-infra/es-native-cross-toolchain:1) and can
break pulls; update the TOOLCHAIN_IMAGE assignment in publish_vec_binaries.sh to
use the exact published repository name
docker.elastic.co/elasticsearch-infra/es-native-cross-toolchain:1 (keeping the
existing LOCAL override behavior), or centralize the image name into a shared
constant so both publish_vec_binaries.sh (TOOLCHAIN_IMAGE) and
build_cross_toolchain_image.sh reference the same canonical image name.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: b3a08cd1-7ce8-45a3-8918-00e0f735e648

📥 Commits

Reviewing files that changed from the base of the PR and between 16db669 and 7df5171.

📒 Files selected for processing (4)
  • libs/native/libraries/build.gradle
  • libs/simdvec/native/build_cross_toolchain_image.sh
  • libs/simdvec/native/publish_vec_binaries.sh
  • libs/simdvec/native/src/vec/c/amd64/vec_2.cpp
💤 Files with no reviewable changes (1)
  • libs/simdvec/native/src/vec/c/amd64/vec_2.cpp
✅ Files skipped from review due to trivial changes (1)
  • libs/native/libraries/build.gradle

@ldematte ldematte merged commit 3b6ed52 into elastic:main Apr 2, 2026
42 checks passed
mromaios pushed a commit to mromaios/elasticsearch that referenced this pull request Apr 9, 2026
…elastic#144845)

This PR adds a Docker-based cross-compilation pipeline for building libvec for all three platforms from a single Linux/amd64 container, replacing the previous Mac-dependent workflow. It builds on top of elastic#145066 (merged), which introduced the tinystd headers and builtin replacements.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Delivery/Build Build or test infrastructure >refactoring Team:Delivery Meta label for Delivery team test-arm Pull Requests that should be tested against arm agents v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants