Skip to content

Add SerenityOS as target#186484

Open
emilytrau wants to merge 19 commits intoNixOS:masterfrom
emilytrau:serenity
Open

Add SerenityOS as target#186484
emilytrau wants to merge 19 commits intoNixOS:masterfrom
emilytrau:serenity

Conversation

@emilytrau
Copy link
Copy Markdown
Member

@emilytrau emilytrau commented Aug 13, 2022

Description of changes

A graphical Unix-like operating system for desktop computers! SerenityOS is a love letter to '90s user interfaces with a custom Unix-like core. It flatters with sincerity by stealing beautiful ideas from various other systems.

image

I've spent about a week working on porting nixpkgs to SerenityOS. I wanted to play around with both lower-level nixpkgs infra and SerenityOS for the first time and this was a super fun project to do both! There's still plenty of work to be done but I wanted to get it out for play/testing/feedback from both the nix and serenity side.

Bootstrapping stdenv got very complicated but I'm proud that this builds entirely from source with no bootstrapping binaries required.

This is able to build against the SerenityOS main branch without any patches. I am planning a PR to make the nix profile work out-of-the-box, and start an overlay for Serenity specific package fixes.

:^)

How to test
# Cross-compile a package
nix-build . -A pkgsCross.x86_64-serenity.hello

# Install the closure into your SerenityOS system
nix copy --to $SERENITY_DIR/Build/x86_64/Root/ ./result --no-check-sigs

# Optional: Install a profile for easier access (WIP)
mkdir -p $SERENITY_DIR/Build/x86_64/Root/nix/var/nix/profiles/per-user
nix profile install --profile $SERENITY_DIR/Build/x86_64/Root/nix/var/nix/profiles/per-user/default ./result
nix copy --to $SERENITY_DIR/Build/x86_64/Root/ $SERENITY_DIR/Build/x86_64/Root/nix/var/nix/profiles/per-user/default --no-check-sigs

# Start SerenityOS!
cd $SERENITY_DIR
Meta/serenity.sh run

Serenity patch to load the nix profile automatically https://gist.github.com/emilytrau/2c4f8346ca71405527521fbbe0113989

Todo
  • There is a pre-libc bootstrapping stage where we "manually" compile a GCC cross-compiler toolchain from source. I think this is a bit hacky and ideally it could leverage the existing nixpkgs tooling
  • Package SerenityOS's libraries. The toolchain is able to compile simple programs right now, but packages with more complex features (eg. SDL2) will need the libraries to integrate with the OS
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@NixOS/exotic-platform-maintainers

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Aug 13, 2022
Copy link
Copy Markdown
Contributor

@ADKaster ADKaster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty cool! There's already a large collection of patches in the Ports/ directory of serenity, with xexxa's cool https://ports.serenityos.net/ that organizes the AvailablePorts by category.

Your question about SDL2 in the description makes me wonder how Nix would want to pick up or reproduce our existing patches.

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Aug 13, 2022
@emilytrau
Copy link
Copy Markdown
Member Author

Packaged serenity libraries, lagom, and added support for x86_64-serenity. Will squash and organiser properly before merging :)

@emilytrau
Copy link
Copy Markdown
Member Author

Not sure why ofborg eval is failing? 😕

@emilytrau
Copy link
Copy Markdown
Member Author

@ofborg eval

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 31, 2022
@emilytrau emilytrau force-pushed the serenity branch 3 times, most recently from eb35b21 to 5f698c1 Compare January 16, 2023 07:58
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 16, 2023
Copy link
Copy Markdown
Member

@alyssais alyssais left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only reviewed the first 1½ commits so far, because it didn't build for me when I tried, either on the second commit, or the head of the PR:

checking for references to /build/ in /nix/store/rh54vhdmfpablzzfjmgnhypbfcs8jq50-serenity-bootstrap-toolchain-unstable-2022-08-07...
RPATH of binary /nix/store/rh54vhdmfpablzzfjmgnhypbfcs8jq50-serenity-bootstrap-toolchain-unstable-2022-08-07/x86_64-pc-linux-gnu/i686-pc-serenity/lib/libopcodes-2.38.so contains a forbidden reference to /build/
find: 'standard output': Broken pipe
find: write error

@alyssais alyssais added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jan 29, 2023
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 7, 2023
@emilytrau
Copy link
Copy Markdown
Member Author

Hello friends! I revisited SerenityOS support again and rewrote it from scratch, with changes from all the review comments (thank you!) and upstream support improvements. In my opinion the new patch set is smaller, simpler, and better integrated with nixpkgs compiler tooling. Please let me know your thoughts! @NixOS/exotic-platform-maintainers

@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Apr 7, 2023
@emilytrau
Copy link
Copy Markdown
Member Author

Glad to hear this project has helped motivate the upstreaming effort! In the mean time I will work on getting the LLVM toolchain to work as well. To focus on the compiler toolchain I've dropped the patches for OpenSSL.

@ghost
Copy link
Copy Markdown

ghost commented Jun 28, 2023

We should consider keeping enough patches for Serenity in-tree to make stdenv work but not patches and conditionals outside of that scope. I think there is less work overall if Serenity testers don't have to build their own toolchains and if we don't have to work around their conditionals in the leaf packages here.

Accumulating patches for toolchains isn't such a terrible thing. It would be a resource for people looking to port toolchains elsewhere and if there are problems at least its not something that we deal with on the master branch.

@alyssais
Copy link
Copy Markdown
Member

Accumulating patches for toolchains isn't such a terrible thing.

Agreed, which is why I'm happy to accept them if they're rejected upstream. But it's definitely not better than the patches being upstreamed, which is why we should only do it if they can't be upstreamed.

@ghost
Copy link
Copy Markdown

ghost commented Jun 28, 2023

Yes, upstreaming is ideal, but complicated. It's easy enough to patch an LLVM release but patching their master branch and waiting for the next release can take a while. Maybe the platform only wants to support building with GCC and not LLVM. These are the sorts of thing that blocked Genode support.

@alyssais
Copy link
Copy Markdown
Member

Maybe the platform only wants to support building with GCC and not LLVM.

This is not the case.

@alyssais
Copy link
Copy Markdown
Member

patching their master branch and waiting for the next release can take a while.

There's a major LLVM release every six months. That's not a long time in the context of adding a whole new platform. And it would be fine to backport patches to get support available early — backports are very different to patches we might never get rid of.

Maybe the platform only wants to support building with GCC and not LLVM.

In that case, it would be fine not to support LLVM for that platform.

@ghost
Copy link
Copy Markdown

ghost commented Jun 28, 2023

A release every six months means waiting three months on average, plus the time to get patches merged. This PR has been open for ten months already and I would not be surprised if it was open in ten months from now.

@ghost
Copy link
Copy Markdown

ghost commented Jun 28, 2023

It is immensely easier to say "no more patches" than it is to upstream work or write patches in the first place. Can we please have an example where we've been burned in the past by this?

@alyssais
Copy link
Copy Markdown
Member

A release every six months means waiting three months on average, plus the time to get patches merged. This PR has been open for ten months already and I would not be surprised if it was open in ten months from now.

Exactly, so if it takes three months for patches to get in, that's unlikely to make a difference at all to the length of time it takes for the new platform to be reviewed (which can start before the patches are upstream).

@alyssais
Copy link
Copy Markdown
Member

Can we please have an example where we've been burned in the past by this?

Example one: I maintain QEMU. For a long time updating QEMU was a total nightmare, because of Darwin patches that were applied in Nixpkgs, and those were just the bare minimum to deal with QEMU depending on a newer SDK than we are able to provide. Updates would be delayed weeks because of the need to update patches, and because they were for Darwin, I couldn't even test them. Later there was a proposal to add eleven more patches to implement 9p support on Darwin. I put my foot down, because at that point there's no way I'd be able to handle updates without them getting blocked on Darwin. As a result, the patches were submitted and accepted upstream, and now every QEMU user on Darwin, not just Nixpkgs users, benefits from that support, and I don't have to try to rebase a dozen patches for a platform I don't use every time I update QEMU.

Example two: we patch systemd to work on musl, because upstream is hostlie to musl. As a result of a compromise with the systemd maintainers, these patches are only applied during musl builds, so they don't have to wait for updated musl patches to update systemd. (This is the same setup originally proposed for the Serenity toolchain.) This works, but it means that as musl enthusiasts, we have to be very on the ball so that new systemd releases don't completely break the ability to run NixOS on musl. (Basically, the patches have to be updated before systemd gets out of staging.) A lot of work would have been saved if upstream had been willing to accept those patches once, so they didn't have to be redone every time. Sadly, in this case that didn't happen, so we're stuck with extra boring, repetitive work forever keeping those patches up to date. It was absolutely worth the attempt to upstream the patches to try to save all that future work, especially because, if upstream is entirely unwilling to support a platform, it's probably not going to take very long to find that out.

@alyssais
Copy link
Copy Markdown
Member

alyssais commented Jun 28, 2023

It is immensely easier to say "no more patches"

Nobody is saying "no more patches". I'm saying "no patches where there hasn't even been the bare minimum of an attempt to upstream them, where they would make sense to upstream, and where upstream is not known to be unreasonable (e.g. hostile to new platforms, bad CLA requirements, etc.)".

@wegank wegank added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 19, 2024
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024
@siraben
Copy link
Copy Markdown
Member

siraben commented Apr 2, 2024

@emilytrau are you still planning on merging this? This would be a cool addition to the cross infra!

@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: exotic Exotic hardware or software platform 6.topic: stdenv Standard environment 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants