Skip to content

cross-rust.mk: Fix regular build post PR #5435#5466

Merged
th0ma7 merged 2 commits intoSynoCommunity:masterfrom
th0ma7:rust-fix
Oct 28, 2022
Merged

cross-rust.mk: Fix regular build post PR #5435#5466
th0ma7 merged 2 commits intoSynoCommunity:masterfrom
th0ma7:rust-fix

Conversation

@th0ma7
Copy link
Copy Markdown
Contributor

@th0ma7 th0ma7 commented Oct 24, 2022

Description

Fix regular build post PR #5435

Fixes #5463

Checklist

  • Build rule all-supported completed successfully
  • New installation of package completed successfully
  • Package upgrade completed successfully (Manually install the package again)
  • Package functionality was tested
  • Any needed documentation is updated/created

Type of change

  • Bug fix
  • New Package
  • Package update
  • Includes small framework changes
  • This change requires a documentation update (e.g. Wiki)

@th0ma7 th0ma7 requested a review from hgy59 October 24, 2022 22:54
@th0ma7 th0ma7 self-assigned this Oct 24, 2022
@th0ma7
Copy link
Copy Markdown
Contributor Author

th0ma7 commented Oct 24, 2022

@hgy59 I'm getting a weird bug when calling cargo build that does not occur when calling cargo install which also does the build. I actually don't get why build cannot handle the same args as install but I must have misread something somehow in the cargo handbook.

Anyhow, I've kept comments for potential follow-ups. Let me know if this solves your issue.

@hgy59
Copy link
Copy Markdown
Contributor

hgy59 commented Oct 27, 2022

Anyhow, I've kept comments for potential follow-ups. Let me know if this solves your issue.

Yes it builds now, but only when I use the build container with the root user (and sudo cargo ... does not work, as it does not find cargo).

I do not really understand, why rust was added to the build environment (i.e. to the Dockerfile)

  • The docker image was increased from < 1GB to > 3GB
  • cargo install works only with root user, as regular user dependencies are not found and crates is not updated
  • how can a package use a different version of rust? (i.e. nightly vs stable)

With the previous rust integration none of the problems above occurred.

- remove unused code
Copy link
Copy Markdown
Contributor

@hgy59 hgy59 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyhow, I've kept comments for potential follow-ups. Let me know if this solves your issue.

We can use the git history for any follow-ups, but I don't think we need those...

It solved the build of synocli-file package (including most tools built with rust)
❤️

@hgy59 hgy59 added framework build/rust Requires rust build system labels Oct 28, 2022
@th0ma7
Copy link
Copy Markdown
Contributor Author

th0ma7 commented Oct 28, 2022

Anyhow, I've kept comments for potential follow-ups. Let me know if this solves your issue.

Yes it builds now, but only when I use the build container with the root user (and sudo cargo ... does not work, as it does not find cargo).

Indeed this is what I noticed. cargo cannot really be installed in "shared" mode whereas multiple users simply make use of and as needed enhance it from their user-profile (or I didn't found the right method yet). For my lxc install I end-up installing it as spksrc user to alleviate this (whereas docker is in priviledge mode thus as root). In the best of world I'd really much like to always install default cargo & rust for all users, then allowing it to be per user defined (e.g. downloading per arch toolchains and other bits such as switching to nightly vs stable).

I do not really understand, why rust was added to the build environment (i.e. to the Dockerfile)

I thought of, well, we also do it for meson for instance, why not for rust. Also I needed something "fixed" to finally get a hold on why it kept failing always. Now that this is solved it can certainly be revisited. And on the other hand, per-arch toolchains aren't in the Dockerfile neither.

  • The docker image was increased from < 1GB to > 3GB
  • cargo install works only with root user, as regular user dependencies are not found and crates is not updated
  • how can a package use a different version of rust? (i.e. nightly vs stable)

With the previous rust integration none of the problems above occurred.

Now that we have a working environment for both regular cross-rustc & python rust wheel I'll be more than happy to revisit this and re-use (and perhaps enhanced) previous integration method.

@hgy59
Copy link
Copy Markdown
Contributor

hgy59 commented Oct 28, 2022

I propose to go with this for now.
Can you merge please?

@th0ma7
Copy link
Copy Markdown
Contributor Author

th0ma7 commented Oct 28, 2022

Yeah, that's what I was thinking too, lets merge this to complete the fixing part then later in another PR find ways to optimize this furtner. merging :)

@th0ma7 th0ma7 merged commit aa46ebc into SynoCommunity:master Oct 28, 2022
@th0ma7 th0ma7 deleted the rust-fix branch October 28, 2022 20:15
@hgy59
Copy link
Copy Markdown
Contributor

hgy59 commented Nov 16, 2022

@th0ma7, I finally found that crates are downloaded to the /opt/cargo/registry folder.
Initially this folder does not exist, and only root can create it.

As a first atempt I created the /opt/cargo/registry folder with world access (777).
This fixed the build of rust targets as regular user.

But a better solution would be to use a sub folder of /spksrc/distrib/ as we do for downloaded sources, go packages, nuget packages, pip wheels, etc.

Unfortunately I couldn't find out how to configure the rust registry folder (aka crates-io)

Error message before:

  ==> Cargo install rust package bat
  Installing bat v0.22.1 (/spksrc/spk/synocli-file/work-x64-6.1/bat-0.22.1)
error: failed to compile `bat v0.22.1 (/spksrc/spk/synocli-file/work-x64-6.1/bat-0.22.1)`, intermediate artifacts can be found at `/spksrc/spk/synocli-file/work-x64-6.1/bat-0.22.1/target`

Caused by:
  failed to get `ansi_colours` as a dependency of package `bat v0.22.1 (/spksrc/spk/synocli-file/work-x64-6.1/bat-0.22.1)`

Caused by:
  failed to load source for dependency `ansi_colours`

Caused by:
  Unable to update registry `crates-io`

Caused by:
  failed to create directory `/opt/cargo/registry/index/github.com-1ecc6299db9ec823`

Caused by:
  Permission denied (os error 13)
make[3]: *** [../../mk/spksrc.cross-rust.mk:59: rust_install_target] Error 101

BTW: for all my containers (except the one to create packages to publish, where I use the root user), I mount a docker volume to /spksrc/distrib to share all the downloaded files between my (~2-20 running containers, switching between >200 clones)

@hgy59
Copy link
Copy Markdown
Contributor

hgy59 commented Nov 16, 2022

@th0ma7 after reading this (rust-lang/cargo#10252) I think there is no way to configure a different folder for the registry.

I propose to use CARGO_HOME = /spksrc/distrib/cargo (as we did before) and create $(CARGO_HOME)/bin as link to /opt/cargo/bin on demand somewhere in spksrc.cross-rust.mk.

@th0ma7
Copy link
Copy Markdown
Contributor Author

th0ma7 commented Nov 16, 2022

after reading this (rust-lang/cargo#10252) I think there is no way to configure a different folder for the registry.

@hgy59 This is exactly the roadblock I had hit which made me go the other way around so we at least get a first implementation fully working until we find something better. On that note, I thought of a slightly different approach (e.g. making mileage on your proposal). Untested so unsure how things would pan-out:

  • I propose to leave only the rustup setup in the Dockerfile (thus leaving RUSTUP_HOME as is in /opt)
  • At toolchain time invoke the rustup target add <toolchain> - this in turn will run as the "user" being used. Therefore we could then indeed move CARGO_HOME to /spksrc/distrib/cargo folder. There may be a need to add a spksrc.tc-rustc.mk file to be included by default in spksrc.tc.mk or something similar.

I believe the changes are relatively minimal and would allow:

  1. freing-up tons of space on the Docker image
  2. allow both priviledge and unpriviledge containers to function properly

As I side note, I migrated entirely to LXC/LXD a while back. This allows me to "share" a spksrc user between my container and my favorite linux distribution. As such I'm always using it in unpriviledged mode but I can always switch back to root and easily dive into it natively from the ~spksrc/. path.

hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 28, 2023
hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 30, 2023
hgy59 added a commit to hgy59/spksrc that referenced this pull request Feb 24, 2023
hgy59 added a commit to hgy59/spksrc that referenced this pull request Apr 6, 2023
hgy59 added a commit to hgy59/spksrc that referenced this pull request May 20, 2023
hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 20, 2024
AlexPresso pushed a commit to AlexPresso/spksrc that referenced this pull request Jan 30, 2025
…unity#5466)

* cross-rust.mk: Fix regular build post PR SynoCommunity#5435

* finalize rust fix

- remove unused code

Co-authored-by: hgy59 <hpgy59@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build/rust Requires rust build system framework

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build with rust fails

2 participants