ghostty: limit the amount of build cores by pluiedev · Pull Request #446056 · NixOS/nixpkgs

pluiedev · 2025-09-25T14:53:26Z

For some bizarre reason Zig contains a bug where the compiler would crash when a large enough file is embedded AND when the builder has too many cores. (It doesn't matter how many cores Zig has been specified to use via the -j flag — the only way to fix this bug is to use CPU scheduling tools to prevent Zig from using more than 32 cores.) This has been causing Hydra builds to repeatedly fail, while Ofborg and manual testing by maintainers were never able to reproduce the failure. However, given this is preventing Ghostty 1.2 from landing in the binary cache, we need to work around it somehow.

Links to upstream discussions:

Fixes #445326

Things done

Add a 👍 reaction to pull requests you find important.

pluiedev · 2025-09-25T14:56:10Z

Cc the folks who reported build failures in the original PR:

@eyJhb @ulrikstrid @Mic92 @Enzime

Could y'all give this a go and see if it works?

pluiedev · 2025-09-25T14:58:04Z

Also I'm not sure who to contact here, but the only one I know who's actively working on Zig stuff is @RossComputerGuy — it seems that the Zig build hook currently doesn't specify the -j flag at all so NIX_BUILD_CORES is being completely ignored in Zig builds right now AFAICT. Should we add that to the hook in a separate PR?

eyJhb · 2025-09-25T15:04:25Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 446056
Commit: 5dfa5c8c81d7115a65ecd34990d1011350ee2f77

`x86_64-linux`

❌ 5 packages failed to build:

ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim

Error logs: `x86_64-linux`

ghostty

NO_COLOR=1 pkg-config gtk4-wayland --cflags --libs 
NO_COLOR=1 pkg-config graphene-gobject-1.0 --cflags --libs 
NO_COLOR=1 pkg-config cairo-gobject --cflags --libs 
NO_COLOR=1 pkg-config pangocairo --cflags --libs 
NO_COLOR=1 pkg-config pango --cflags --libs 
NO_COLOR=1 pkg-config harfbuzz-gobject --cflags --libs 
NO_COLOR=1 pkg-config gmodule-2.0 --cflags --libs 
NO_COLOR=1 pkg-config gdk-pixbuf-2.0 --cflags --libs 
/nix/store/7sh77qdzh69644yw7i7s38iihci042q1-zig-0.14.1/bin/zig build-exe @/build/source/.zig-cache/args/c63a57624e650c2fd1a5ba23c5e008585cf336ef2636b410284eb2004824c008 
install
+- install ghostty
   +- zig build-exe ghostty ReleaseFast native-native failure
error: the following command terminated unexpectedly:
/nix/store/7sh77qdzh69644yw7i7s38iihci042q1-zig-0.14.1/bin/zig build-exe @/build/source/.zig-cache/args/c63a57624e650c2fd1a5ba23c5e008585cf336ef2636b410284eb2004824c008 
Build Summary: 154/157 steps succeeded; 1 failed
install transitive failure
+- install ghostty transitive failure
   +- zig build-exe ghostty ReleaseFast native-native failure
error: the following build command failed with exit code 1:
/build/source/.zig-cache/o/c9a5e930a5cb7ccee2a097127e8ca132/build /nix/store/7sh77qdzh69644yw7i7s38iihci042q1-zig-0.14.1/bin/zig /nix/store/7sh77qdzh69644yw7i7s38iihci042q1-zig-0.14.1/lib/zig /build/source /build/source/.zig-cache /build/tmp.m5BuaY6vBm --seed 0x7f4844da -Ze720b2ef844882b5 -Dcpu=baseline -Doptimize=ReleaseFast --color off --system -Dversion-string=1.2.0 -fsys=glslang --search-prefix /nix/store/cyjg0acqk1nr186a5p6sqzfgbsyzp9zg-glslang-15.4.0 -j24 --verbose

pluiedev · 2025-09-25T15:05:03Z

Still no? Hmm... What happens if you turn off parallel building altogether?

eyJhb · 2025-09-25T15:06:08Z

Still no? Hmm... What happens if you turn off parallel building altogether?

What would I need to do, to try that? Also, my comment was wrong in the other PR regarding how many cores I have. I'm building on a remote builder which has 64 cores :)

pluiedev · 2025-09-25T15:08:38Z

I think it's --option cores 1 for nix build or just --cores 1 for nix-build - ~~or you could .overrideAttrs { enableParallelBuilding = false; }~~ (scratch that, doesn't seem to work?)

getchoo

Otherwise LGTM. Thanks :)

getchoo · 2025-09-25T15:17:28Z

scratch that, doesn't seem to work?)

This is again because Zig's hook does not correctly (or even really try to...) handle regular controls of build parallelism like other hooks 🫠

ulrikstrid · 2025-09-25T15:20:35Z

I don't have my computer available right now but will try when I can. i have however tried just adding -j1 to the build flags without success

pluiedev · 2025-09-25T15:20:37Z

I'm honestly surprised that it even worked for other Zig-based software like TigerBeetle and Bun then... Maybe we were all just collectively really lucky that we never ran into any parallelism-based bugs yet

emilazy · 2025-09-25T15:20:46Z

I think non‐big-parallel Hydra runners only build with 2 to 4 cores? So I am not sure this would have any effect there.

Edit: Ah, #446056 (comment) would explain it.

pluiedev · 2025-09-25T15:22:52Z

✨ It never worked in the first place ✨

RossComputerGuy · 2025-09-25T15:25:52Z

This is again because Zig's hook does not correctly (or even really try to...) handle regular controls of build parallelism like other hooks 🫠

I'm actually working on a PR to add that in

I'm honestly surprised that it even worked for other Zig-based software like TigerBeetle and Bun then... Maybe we were all just collectively really lucky that we never ran into any parallelism-based bugs yet

I've tried building Bun and oddly enough, I don't think I've seen this problem on my Ampere hardware which is Arm. Last time I touched Bun, I had 64 cores. I have 128 cores and I remember updating zig and running a nixpkgs-review on it to build everything and Ghostty passing. It's possible this is only a problem on x86. I'm running some builds right now to verify this.

pluiedev · 2025-09-25T15:28:06Z

It's entirely plausible that Bun isn't (ab)using @embedFile nearly as heavily as we are — we use it for animation frames, Unicode lookup tables, etc., all of which are quite large. It's probably the same reason why upstream seemingly only took it seriously after Ghostty 1.2 came out, since initially it was stuck in upcoming jail for quite a long time during Zig 0.14's development

RossComputerGuy · 2025-09-25T15:30:02Z

Hmm, and it's always possible because I've got Arm cores that the difference in performance compared to x86 could have an impact. I've seen this, my individual cores are slower than most x86 hardware but together they're strong.

justDeeevin · 2025-09-25T15:31:18Z

I said in #445326 but i'll say here as well: this pr allows for successful build on my machine

RossComputerGuy · 2025-09-25T15:34:30Z

I just opened #446063 to add support for using --cores. However I can't test build ghostty because the zig version it uses crashes due to me running with a 64k page size. If it was using 0.14, I could test it. I was at least able to verify it is using the flag now.

pluiedev · 2025-09-25T15:35:14Z

Ghostty 1.2 is on Zig 0.14, though. 1.1.3 wouldn't work for sure

RossComputerGuy · 2025-09-25T15:40:57Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 446056
Commit: 429c48e2a62c02d5e92ad7bb1e0087d75cf50840

`aarch64-linux`

❌ 5 packages failed to build:

ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim

✅ 1 package built:

nixpkgs-manual

Error logs: `aarch64-linux`

ghostty

NO_COLOR=1 pkg-config gtk4-wayland --cflags --libs 
NO_COLOR=1 pkg-config graphene-gobject-1.0 --cflags --libs 
NO_COLOR=1 pkg-config cairo-gobject --cflags --libs 
NO_COLOR=1 pkg-config pangocairo --cflags --libs 
NO_COLOR=1 pkg-config pango --cflags --libs 
NO_COLOR=1 pkg-config harfbuzz-gobject --cflags --libs 
NO_COLOR=1 pkg-config gmodule-2.0 --cflags --libs 
NO_COLOR=1 pkg-config gdk-pixbuf-2.0 --cflags --libs 
/nix/store/ilpzihsg7avag4a0z4arwgkzar3h5jzq-zig-0.14.1/bin/zig build-exe @/build/source/.zig-cache/args/ea5af3d6daad594c61beb5917ee46d616c4d20b1f4f44822b59e8e7523e9477f 
install
+- install ghostty
   +- zig build-exe ghostty ReleaseFast native-native failure
error: the following command terminated unexpectedly:
/nix/store/ilpzihsg7avag4a0z4arwgkzar3h5jzq-zig-0.14.1/bin/zig build-exe @/build/source/.zig-cache/args/ea5af3d6daad594c61beb5917ee46d616c4d20b1f4f44822b59e8e7523e9477f 
Build Summary: 154/157 steps succeeded; 1 failed
install transitive failure
+- install ghostty transitive failure
   +- zig build-exe ghostty ReleaseFast native-native failure
error: the following build command failed with exit code 1:
/build/source/.zig-cache/o/eec229b1f8bb9f00e165fd2d494bab17/build /nix/store/ilpzihsg7avag4a0z4arwgkzar3h5jzq-zig-0.14.1/bin/zig /nix/store/ilpzihsg7avag4a0z4arwgkzar3h5jzq-zig-0.14.1/lib/zig /build/source /build/source/.zig-cache /build/tmp.6AA9tMHdTv --seed 0xcfc3f440 -Z34691887ebfc6d50 -Dcpu=baseline -Doptimize=ReleaseFast --color off --system -Dversion-string=1.2.0 -fsys=glslang --search-prefix /nix/store/7fxg13l7zr7v4wpzm9r5vhkd9fgpb153-glslang-15.4.0 -j24 --verbose

justDeeevin · 2025-09-25T15:44:19Z

Disregard my success report—it was run on my laptop which has 16 cores so the bug doesn't apply

pluiedev · 2025-09-25T22:38:33Z

Oh that is disgusting. But it should work. At this point since I'm not nearly qualified enough to try to fix the bug at its source, I'm tempted to go with this anyways so we can at least get Ghostty in Hydra first.

pluiedev · 2025-09-25T23:01:01Z

@Mic92 I modified your approach and used a shell script wrapper instead of manually replicating what the hook currently does. Could you give it a whirl?

For some bizarre reason Zig contains a bug where the compiler would crash when a large enough file is embedded AND when too many cores are used during compilation. This has been causing Hydra builds to repeatedly fail, while Ofborg and manual testing by maintainers were never able to reproduce the failure. However, given this is preventing Ghostty 1.2 from landing in the binary cache, we need to work around it somehow. 24 cores should be *more* than enough for now, but if it still doesn't work we can lower the threshold even further. Links to upstream discussions: * ziglang/zig#25297 * ziglang/zig#22867 * ghostty-org/ghostty#8676 Fixes NixOS#445326

qubitnano · 2025-09-26T01:10:10Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 446056
Commit: fb8d932c8622103c17640fd65234e52e5b1e3e78

`x86_64-linux`

✅ 5 packages built:

ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim

RossComputerGuy · 2025-09-26T01:20:37Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 446056
Commit: fb8d932c8622103c17640fd65234e52e5b1e3e78

`aarch64-linux`

✅ 5 packages built:

ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim

pluiedev · 2025-09-26T01:35:58Z

Let's freaking go

qubitnano

thanks @pluiedev @Mic92

ulrikstrid · 2025-09-26T12:08:54Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 446056
Commit: fb8d932c8622103c17640fd65234e52e5b1e3e78

`x86_64-linux`

✅ 5 packages built:

ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim

ulrikstrid · 2025-09-26T12:09:29Z

Builds on my Threadripper now, 64 physical cores

SuperSandro2000 · 2025-09-26T12:27:50Z

Shouldn't we limit this in zig itself as this could effect any package big enough?

pluiedev · 2025-09-26T12:41:53Z

Not really. The only way this triggers is if you embed a lot of strings directly into the binary, which apparently is not the case for TigerBeetle, Bun, River, etc. It also heavily depends on just how many strings are embedded - in Ghostty 1.2 we added new Unicode tables that I suspected brought our total usage over the "limit". In reality there isn't a single limit, but rather the limit decreases as core count goes up, so the limit we want to set is completely dependent on the needs of any given project.

Given all of this and the fact that upstream Zig has marked the root issue as something they'd like to iron out for 0.15.2, I don't think it makes sense to add this wrapper to the compiler itself. If that fix does land in 0.15.2 I might even consider backporting the 1.2.x releases to use Zig 0.15 instead of 0.14 — my original plan is to switch to Zig 0.15 for 1.3.x, but plans might have to change now.

Mic92 · 2025-09-26T14:39:08Z

Not really. The only way this triggers is if you embed a lot of strings directly into the binary, which apparently is not the case for TigerBeetle, Bun, River, etc. It also heavily depends on just how many strings are embedded - in Ghostty 1.2 we added new Unicode tables that I suspected brought our total usage over the "limit". In reality there isn't a single limit, but rather the limit decreases as core count goes up, so the limit we want to set is completely dependent on the needs of any given project.

Given all of this and the fact that upstream Zig has marked the root issue as something they'd like to iron out for 0.15.2, I don't think it makes sense to add this wrapper to the compiler itself. If that fix does land in 0.15.2 I might even consider backporting the 1.2.x releases to use Zig 0.15 instead of 0.14 — my original plan is to switch to Zig 0.15 for 1.3.x, but plans might have to change now.

However what we should introduce in our zig builder is -j$NIX_BUILD_CORES... but this is orthogonal to this PR.

pluiedev · 2025-09-26T14:39:22Z

I'd be happy to see this merged given that multiple people have confirmed that this works for them :) (I don't have the perms myself. Yet.)

pluiedev · 2025-09-26T14:40:05Z

However what we should introduce in our zig builder is -j$NIX_BUILD_CORES... but this is orthogonal to this PR.

That would be #446063 I believe

pluiedev mentioned this pull request Sep 25, 2025

Build failure: ghostty #445326

Closed

3 tasks

nix-owners bot requested review from getchoo and jcollie September 25, 2025 14:58

nixpkgs-ci bot added the 9.needs: reviewer This PR currently has no reviewers requested and needs attention. label Sep 25, 2025

getchoo reviewed Sep 25, 2025

View reviewed changes

Comment thread pkgs/by-name/gh/ghostty/package.nix Outdated

nixpkgs-ci bot removed the 9.needs: reviewer This PR currently has no reviewers requested and needs attention. label Sep 25, 2025

pluiedev force-pushed the pluie/jj-wvossqnpmvpy branch from 5dfa5c8 to 429c48e Compare September 25, 2025 15:16

pluiedev force-pushed the pluie/jj-wvossqnpmvpy branch from b6766fe to b58cdc0 Compare September 25, 2025 22:59

pluiedev force-pushed the pluie/jj-wvossqnpmvpy branch from b58cdc0 to 43c085b Compare September 25, 2025 23:02

pluiedev force-pushed the pluie/jj-wvossqnpmvpy branch from 43c085b to fb8d932 Compare September 25, 2025 23:03

pluiedev mentioned this pull request Sep 25, 2025

nixos/community-builder: add pluiedev nix-community/infra#1985

Merged

RossComputerGuy approved these changes Sep 26, 2025

View reviewed changes

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Sep 26, 2025

qubitnano approved these changes Sep 26, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Sep 26, 2025

ulrikstrid approved these changes Sep 26, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 3+ This PR was reviewed and approved by three or more persons. and removed 12.approvals: 2 This PR was reviewed and approved by two persons. labels Sep 26, 2025

pluiedev mentioned this pull request Sep 26, 2025

[Upstream] Compiler segfaults when built on systems with >~32 cores ghostty-org/ghostty#8924

Closed

SuperSandro2000 approved these changes Sep 26, 2025

View reviewed changes

Mic92 added this pull request to the merge queue Sep 26, 2025

Merged via the queue into NixOS:master with commit b765170 Sep 26, 2025
34 of 37 checks passed

Uh oh!

Conversation

pluiedev commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

pluiedev commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

eyJhb commented Sep 25, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

eyJhb commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

getchoo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

getchoo commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ulrikstrid commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

emilazy commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

RossComputerGuy commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

RossComputerGuy commented Sep 25, 2025

Uh oh!

justDeeevin commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RossComputerGuy commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

RossComputerGuy commented Sep 25, 2025

nixpkgs-review result

aarch64-linux

Uh oh!

justDeeevin commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

pluiedev commented Sep 25, 2025

Uh oh!

qubitnano commented Sep 26, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

RossComputerGuy commented Sep 26, 2025

nixpkgs-review result

aarch64-linux

Uh oh!

pluiedev commented Sep 26, 2025

Uh oh!

qubitnano left a comment

Choose a reason for hiding this comment

Uh oh!

ulrikstrid commented Sep 26, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

ulrikstrid commented Sep 26, 2025

Uh oh!

pluiedev commented Sep 25, 2025 •

edited

Loading

pluiedev commented Sep 25, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

pluiedev commented Sep 25, 2025 •

edited

Loading

getchoo commented Sep 25, 2025 •

edited

Loading

emilazy commented Sep 25, 2025 •

edited

Loading

justDeeevin commented Sep 25, 2025 •

edited

Loading

`nixpkgs-review` result

`aarch64-linux`

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`aarch64-linux`

`nixpkgs-review` result

`x86_64-linux`