Skip to content

feat: configure cross to support x86_64 and aarch64 for linux/musl#674

Merged
h3nill merged 2 commits intobytebeamio:mainfrom
carlocorradini:cross
Aug 20, 2023
Merged

feat: configure cross to support x86_64 and aarch64 for linux/musl#674
h3nill merged 2 commits intobytebeamio:mainfrom
carlocorradini:cross

Conversation

@carlocorradini
Copy link
Contributor

@carlocorradini carlocorradini commented Aug 7, 2023

Related to PR #671 and PR #673

What has been changed:

  • Configure cross for enhanced cross compilation supporting x86_64-unknown-linux-musl and aarch64-unknown-linux-musl and more...

h3nill
h3nill suggested changes Aug 8, 2023
View reviewed changes
Copy link

@h3nill h3nill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is failing to build for me with error:

Error:
   0: failed to parse file `"/home/henil/Workspace/bytebeam/rumqtt/Cross.toml"` as TOML
   1: invalid type: map, expected a string for key `target.x86_64-unknown-linux-musl.image` at line 8 column 14

$ cross version
cross 0.2.5

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 8, 2023

@henil You must follow the official installation procedure from Cross docs:

cargo install cross --git https://github.com/cross-rs/cross

I don't know why they want to install using git and not directly from cargo but they decided to do this :/

The CI respect their decision :)

@h3nill
Copy link

h3nill commented Aug 8, 2023

gotcha, works when installed via git and cargo.

btw why was gnu build removed, We would like to have both gnu and musl build if possible

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 8, 2023 •
edited
Loading

@henil The top level [build] is responsbile for a default implementation (gnu, etc...)
Since musl is particular the two targets only override the default one ([build]).
gnu is and always must be supported :)

@h3nill
Copy link

h3nill commented Aug 8, 2023 •
edited
Loading

oh my bad missed that, been a while since i used cross :)

86_64-unknown-linux-gnu and x86_64-unknown-linux-musl are building properly 🎉

but aarch64-unknown-linux-musl fails to build for me (also tried disabling buildkit), i am running using:

cross build --target aarch64-unknown-linux-musl

with error (click to expand) 

[+] Building 1.2s (5/5) FINISHED                                                                                                                 docker:default
 => [internal] load build definition from Dockerfile.aarch64-unknown-linux-musl-custom                                                                     0.0s
 => => transferring dockerfile: 212B                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/alpine:edge                                                                                             0.8s
 => CACHED [1/2] FROM docker.io/library/alpine:edge@sha256:2d01a16bab53a8405876cec4c27235d47455a7b72b75334c614f2fb0968b3f90                                0.0s
 => ERROR [2/2] RUN eval "apk add --no-cache build-base openssl-dev"                                                                                       0.3s
------
 > [2/2] RUN eval "apk add --no-cache build-base openssl-dev":
0.329 exec /bin/sh: exec format error
------
Dockerfile.aarch64-unknown-linux-musl-custom:5
--------------------
   3 |                     ARG CROSS_DEB_ARCH=
   4 |                     ARG CROSS_CMD
   5 | >>>                 RUN eval "${CROSS_CMD}"
--------------------
ERROR: failed to solve: process "/bin/sh -c eval \"${CROSS_CMD}\"" did not complete successfully: exit code: 1
Error:
   0: could not run container
   1: when building custom image
   2: when pre-building
   3: `docker buildx build --platform linux/arm64 --progress auto --label 'org.cross-rs.for-cross-target=aarch64-unknown-linux-musl' --label 'org.cross-rs.runs-with=aarch64-unknown-linux-musl' --label 'org.cross-rs.workspace_root=/home/henil/Workspace/bytebeam/rumqtt' --tag localhost/cross-rs/cross-custom-rumqtt:aarch64-unknown-linux-musl-d90ee-pre-build --build-arg 'CROSS_CMD=apk add --no-cache build-base openssl-dev' --build-arg 'CROSS_DEB_ARCH=musl-linux-arm64' --file /home/henil/Workspace/bytebeam/rumqtt/target/aarch64-unknown-linux-musl/Dockerfile.aarch64-unknown-linux-musl-custom --output 'type=docker' /home/henil/Workspace/bytebeam/rumqtt` failed with exit status: 1

Location:
   /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/convert/mod.rs:717

Warning: call to docker failed
Suggestion: is `buildx` available for the container engine?
Note: disable the `buildkit` dependency optionally with `CROSS_CONTAINER_ENGINE_NO_BUILDKIT=1`
Note: CROSS_CMD=apk add --no-cache build-base openssl-dev

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

How are you running it for aarch target?

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 8, 2023

@henil Try starting buildx:

docker buildx create --name mybuilder --use --bootstrap

See this for more information

Moreover, try update Docker (I don't have found anything about buildx in Cross docs)

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

@henil Any update? 🤗🥳

@h3nill
Copy link

h3nill commented Aug 17, 2023

i was working on this only :)

turns out to build for aarch target you need to install qemu and one other related package, see: https://stackoverflow.com/questions/71531771/docker-buildx-armv7-platform-missing. can you mention this somewhere? (also due to emulation compilation is very slow)

PS: sorry for the delay it was kinda hard to notice that i was missing arm platform support in docker so took a while to debug.

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

I don't have qemu in my machine and it works 😅

@h3nill
Copy link

h3nill commented Aug 17, 2023

what machine are you on?

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

@henil See https://www.docker.com/blog/how-to-rapidly-build-multi-architecture-images-with-buildx

I have already configured Docker buildx. I think this is the source of errors. Start/Configure Docker buildx with:

docker buildx create --name mybuilder --use --bootstrap

Restart Docker daemon and rebuild with cross

cross use qemu only for test (https://github.com/cross-rs/cross#supported-targets)

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

what machine are you on?

WSL2 on Windows -> If it works on WLS2 it should/must work on a "true" Linux machine :)

@h3nill
Copy link

h3nill commented Aug 17, 2023

i have buildx image running already, before installing qemu it didn't show arm platforms here.

image

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

Well install qemu and setup Docker buildx also in CI

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

I've created the script for building the binaries (see PR #673)
I can add a check for qemu?

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 17, 2023

Nevertheless this is strange because from cross docs qemu is used only for tests

@h3nill
Copy link

h3nill commented Aug 17, 2023

can anyone else with linux system try this?

cc: @de-sh @swanandx

@h3nill
Copy link

h3nill commented Aug 20, 2023

@carlocorradini I confirmed with someone else as well, aarch binaries doesn't build unless you have qemu related dependencies installed (i think its probably a docker thing not a cross), can you mention this two dependencies qemu qemu-user-static inside Cross.tom in comments.

ref: https://stackoverflow.com/questions/71531771/docker-buildx-armv7-platform-missing

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 20, 2023

Good to know!

@de-sh
Copy link
Contributor

de-sh commented Aug 20, 2023

i think its probably a docker thing not a cross

It's a cross thing, as in qemu is used to emulate hardware for running/testing against expected backend.

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 20, 2023

@henil Done

@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 20, 2023 •
edited
Loading

In the script available in PR #673 I'll add a check before running cross to assert the presence of qemu and qemu-user-static

@h3nill
Copy link

h3nill commented Aug 20, 2023 •
edited
Loading

as in qemu is used to emulate hardware for running/testing against expected backend

@de-sh but running the following command shouldn't run binary or tests

cross build --target aarch64-unknown-linux-musl

@h3nill h3nill merged commit 3c40444 into bytebeamio:main Aug 20, 2023
@h3nill
Copy link

h3nill commented Aug 20, 2023

Thank you @carlocorradini! 🎉

@carlocorradini carlocorradini deleted the cross branch August 20, 2023 16:05
@carlocorradini
Copy link
Contributor Author

carlocorradini commented Aug 21, 2023 •
edited
Loading

@henil build.sh script in PR #673 has been updated to check if qemu and qemu-user-static are installed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@h3nill h3nill h3nill requested changes

@de-sh de-sh de-sh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@carlocorradini @h3nill @de-sh