The biggest problem I see with this is, that we must make it maintainable. There need to be good comments explaining why things were done like they were done and helping others fix future problems.

I made an attempt at refactoring the meta attribute checking/augmentation so it can be reused outside mkDerivation with 21aaabc. I feel the change should be /ok/ but would love to hear if anyone has better ideas for implementing it.

EDIT: scratch that ofborg eval fails :(

Hi. I happen to have been working on this same project last week.

I've hastily put together a repo so you folks can view my work. See https://github.com/roconnor/nixpkgs-bootstrap. So far I've gone as far as building bash-2.05b musl-1.1.24.

I wasn't really ready to showcase this yet, so it is still unpolished and has no documentation. My packages for tinycc and lower are not as refined as yours. In particular, I wasn't really aware of your builtins.placeholder trick for bootstrapping.

Another difference is that my project is deliberately designed to be independent of nixpkgs, though with the idea that it could be integrated into nixpkgs without difficulty.

Fixes #123095 ?

Split out stdenv: factor out meta attr augmentation for reusability #228759 (eval failure fixed there)

Is there a 1-pager that explains what the end state of bootstrap should look like for nixpkgs (a few crucial package dependency arrows for stdenv)? If not, would it be reasonable to ask to outline it in PR? Ideally compared to the current bootstrap we have today.

I have a few idle questions that might steer writing a short overview:

1. How many steps do we need to get gcc-12 built?
2. Is the result practical for tweaking and porting to other architectures and OSes?
3. Is it expected to be an experiment to tinker from time to time? Or it's expected to be the primary way to build nixpkgs "soon" (say, in a year).
4. Do we expect (many) other upstream linux distributions to use it or will it always be a burden to maintain in 2-3 downstream distributions? Toolchains are not easy to maintain by a small group of people.

https://man.sr.ht/~oriansj/bootstrappable/live-bootstrap.md illustrates a 100 step process to get to gcc 4.7.4. I've been working through it, and it doesn't seem that bad. Certainly the live-bootstrap folks have already done all the hard work. The live-bootstrap project does actually all the way to gcc 12, but Nixpkg's bootstrap-tools currently starts at gcc 8.3, so arguably that is as far as we need to go (along with gernerating other tooling in bootstrap-tools).

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-supply-chain-security-project/34345/5

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-supply-chain-security-project/34345/8

Hi. Just curious. What are the remaining steps before this can replace bootstrap-files?

Also, how easy or hard is it to extend this to x86_64? From what I understand, guix is already able to do that?

Btw @emilytrau if there's anything we can help you with getting full-source bootstrap over the finish line, let us know!

Yeah it would be really cool to have this for 24.05!

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/aux-foundational-packages/46707/4

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-nixos-and-reproducible-builds-could-have-detected-the-xz-backdoor-for-the-benefit-of-all/61956/9

Also, how easy or hard is it to extend this to x86_64? From what I understand, guix is already able to do that?

The two main hurdles are mes and tinycc, which both have the x86 support hardcoded in nixpkgs. Mes was trivial and just entailed replacing every instance of x86 with a variable can switch to x86_64. It also seems to support 32-bit arm and 64-bit riscv, but I haven't tested them.

Tinycc seems to be more of an issue, which could also be related to issues with mes. Attempting to build the very first stage of tcc with mes, after un-hardcoding the target, gave the attached log:
tinycc-boot-mes.log