What exactly is the reMarkable architecture? It is Linux and Arm, does it have a custom libc or something? I would rather GCC from source for this than bundle their pre-built stuff. I know we do just that for Android, but I wish we didn't there, too (#38411).

See https://github.com/toltec-dev/toolchain , which builds the toolchain from source instead of using the pre-built toolchain.

This is how the toolchain is built on Debian; https://github.com/toltec-dev/toolchain/blob/master/base/Dockerfile

@GrahamcOfBorg build pkgs.pkgsCross.remarkable1.hello

@GrahamcOfBorg build pkgsCross.remarkable1.hello

Author of the overlay these were derived from here!

I did look into using an upstream GCC, and I remember running into some issues, but I don't recall precisely what they were, unfortunately. I needed ABI compatible with the software on the device, and using the upstream toolchain was the simplest way to get that. I think their libc is glibc-based---the device is pretty much a GNU userland running on an i.MX6 SoloLite, as far as I'm aware---but I didn't have success trying to build it from scratch.

I'm also happy to maintain this as an overlay (I'll also try to figure out how to add Flakes support) if you'd rather not see the upstream binary toolchain and related hackery added to nixpkgs directly---dealing with the ugliness of that is the main reason that I didn't try to upstream the configuration in the first place.

Ben, if using a built-from-source nixpkgs version of the standard gcc and glibc is working for you, I don't think you need the rmToolchain packages at all---the various wrappers around oecore that I'm setting up there are only really useful for satisfying the constraints that Nixpkgs expects out of a compiler/libc. If you want to remove them here, I'm happy to continue to maintain the upstream-binaries packages out of tree (hopefully with flakes to make using them easy, once I figure out flakes).

Hi @pl-semiotics! The toltec-dev toolchain seems to be able to replace the reMarkable toolchain with one using Crosstool-NG. And importantly the closed-source libqsgepaper Qt library as well. Do you think we could follow their example?

@siraben, unfortunately, I have not used their toolchain, so I can't say for sure---I wasn't aware of it (or, possibly, it didn't exist---I'm not sure) when I first set up the nix-remarkable overlay. I could try to take a look at it ~this weekend to audit it and see if it works for my current use cases (unfortunately, I don't believe I still have copies of the experiments that necessitated the switch to the binary toolchain in the first place). It does look promising, though!

If the open source toolchain here works well enough (hopefully!), we should remove the rmToolchain packages in their entirety, since they're just the equivalent of cc-wrapper for the binary toolchain.

I've been recently testing the from-source toolchain and it produces dynamic executables that don't run on the tablet. the toolchain from the company also produces dynamic executables but links and runs properly. @Ericson2314 What do you suggest?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/remarkable1-cross-compile-toolchain-for-nix/9961/1

@siraben OK so first a few things:

1. If the PR isn't ready pleas mark it WIP, so I know what I'm reviewing.

2. "produces dynamic executables that don't run on the tablet." Thank you for taking the time to do the build, but note that this isn't enough information for anyone to actually debug the issue.

I would like to get to the bottom of this, as having from-source builds is much more maintainable. Perhaps you can paste some errors you get? Or, if there error is very vague, try comparing the readelf output of the final binary or something?

If the PR isn't ready pleas mark it WIP, so I know what I'm reviewing.

Ok, when would the PR be considered ready? It currently cross-compiles many packages.

I would like to get to the bottom of this, as having from-source builds is much more maintainable. Perhaps you can paste some errors you get? Or, if there error is very vague, try comparing the readelf output of the final binary or something?

I have some executables I cross-compiled from various setups; hello was cross-compiled using the cross toolchain based on the company's GCC and linker, hello_new was compiled using the from-source version of the cross toolchain, and hello_new_static was cross-compiled statically via nix-build . -A pkgs.pkgsCross.remarkable1.pkgsStatic.hello. The only binaries that run correctly on the tablet are hello and hello_new_static

root@reMarkable:~# file hello_new
hello_new: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/3s2x09bbl1xss1lcx7a6dbw69zmip3gx-glibc-2.32-armv7l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped
root@reMarkable:~# ./hello_new
-sh: ./hello_new: No such file or directory
root@reMarkable:~# file hello
.hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c213556b777ae8e1fb8f644a343a65cc0a83b2e7, not stripped
root@reMarkable:~# file hello_new_static
hello_new_static: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

root@reMarkable:~# file hello_new
hello_new: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/3s2x09bbl1xss1lcx7a6dbw69zmip3gx-glibc-2.32-armv7l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped

That would be your first problem. You must override the dynamic linker used by the CC wrapper to match the one present on the device. For comparison, this is how I do it with the proprietary toolchain: https://github.com/pl-semiotics/nix-remarkable/blob/master/rM/toolchain-pkgs.nix#L38 . I think since this is intended for inclusion in Nixpkgs the most reasonable thing to do would likely be to extend this if tree: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/bintools-wrapper/default.nix#L52 .

Ok, when would the PR be considered ready? It currently cross-compiles many packages.

Sorry! I mistakenly thought you had begun pushing the source version.

@pl-semiotics Thanks! I think that will work just fine.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-wanted-remarkable1-cross-compile-toolchain-for-nix/9961/7

Per discussion on Discourse, there seems to be an issue with setting the appropriate dynamic linker. @Ericson2314 is there some way we could do the following two commands to set the dynamic linker?

# copied from the default wrapper script since there's no way to
# set this :-(
echo "/lib/ld-linux-armhf.so.3" >$out/nix-support/dynamic-linker
echo "-dynamic-linker /lib/ld-linux-armhf.so.3" > $out/nix-support/libc-ldflags-before

@siraben I'm pretty sure all you need is the zero-gravitas definition and the remarkable1 example that uses it.

@Ericson2314 Ok. I've followed your suggestions regarding the custom toolchain/depending on on-device dynamic linker.

Build from scratch, statically linked, should be able to drop in
Build from scratch, dynamically linked, can use patchelf to replace libraries with pre-installed versions if desired.

Could you clarify how patchelf could be used to achieve this?

Could you clarify how patchelf could be used to achieve this?

patchelf has a --set-interpreter flag. --remove-rpath or --set-rpath is also useful the packages are being mashed together.

Using patchelf worked. I might need to note this somewhere in the manual or wiki so users would know to do this.

I got Nix installed on my tablet and did various cross-builds and used nix-copy-closure to send them over, and they all work, these packages include: hello, tmux, chicken, openssh, gcc.

Ofborg error was unrelated, guess something bad got merged at some point.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-wanted-remarkable1-cross-compile-toolchain-for-nix/9961/16