Skip to content

ci: add keep-sorted workflow; format and sort top-level/python-packages.nix#391087

Merged
infinisil merged 5 commits intoNixOS:masterfrom
katexochen:treewide/sort
Mar 24, 2025
Merged

ci: add keep-sorted workflow; format and sort top-level/python-packages.nix#391087
infinisil merged 5 commits intoNixOS:masterfrom
katexochen:treewide/sort

Conversation

@katexochen
Copy link
Copy Markdown
Contributor

@katexochen katexochen commented Mar 18, 2025

keep-sorted is a language-agnostic formatter that sorts lines between two markers in a larger file.

This is a first step to use keep-sorted to enforce sorting in nixpkgs, adding a CI job and sorting python-packages.nix (as it had a quite uniform structure already). The nixfmt format comes in handy here, as the way blocks are formatted make it really easy for keep-sorted.

As the workflow has pull_request_target trigger, it won't run on this PR, I've tested it on my fork: https://github.com/katexochen/nixpkgs/actions/runs/13934128359/job/38998162445?pr=1

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 6.topic: policy discussion Discuss policies to work in and around Nixpkgs backport release-24.11 labels Mar 18, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Mar 18, 2025
@katexochen katexochen marked this pull request as ready for review March 18, 2025 21:52
Copy link
Copy Markdown
Contributor

@aforemny aforemny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 19, 2025
@katexochen katexochen force-pushed the treewide/sort branch 2 times, most recently from ce86a99 to 9735ecf Compare March 19, 2025 09:53
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 19, 2025
@wegank wegank added 12.approvals: 2 This PR was reviewed and approved by two persons. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Mar 23, 2025
Copy link
Copy Markdown
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future I hope that we can migrate all lists of packages to something like pkgs/by-name, but for now this looks like a good alternative.

LGTM, I'll merge!

@infinisil
Copy link
Copy Markdown
Member

Oh, conflict to resolve. It would also be good to add the formatting commit to .git-blame-ignore-revs btw

Signed-off-by: Paul Meyer <katexochen0@gmail.com>
@katexochen
Copy link
Copy Markdown
Contributor Author

Oh, conflict to resolve.

Yes, getting this in without a conflict isn't easy. 😅

Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
@katexochen
Copy link
Copy Markdown
Contributor Author

In the future I hope that we can migrate all lists of packages to something like pkgs/by-name, but for now this looks like a good alternative.

I hope we can do so, but keep-sorted can do much more! It can sort markdown tables, large lists of buildInputs, ...

@infinisil infinisil merged commit 848d816 into NixOS:master Mar 24, 2025
20 of 24 checks passed
@nixpkgs-ci
Copy link
Copy Markdown
Contributor

nixpkgs-ci bot commented Mar 24, 2025

Backport failed for release-24.11, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.11
git worktree add -d .worktree/backport-391087-to-release-24.11 origin/release-24.11
cd .worktree/backport-391087-to-release-24.11
git switch --create backport-391087-to-release-24.11
git cherry-pick -x 5f6f5e13ae0b6960cbf1be8aeb3d0048285a08d1 cb755af53647e28dd54d576c074ac8de7ad6fc73 60b30dd3164e656eb5ca049064781d14999fc77c 3b0f56da6197c20bbd56132654a9a1a424ddf743 fd14c067813572afc03ddbf7cdedc3eab5a59954

@wolfgangwalther
Copy link
Copy Markdown
Contributor

Backport failed for release-24.11, because it was unable to cherry-pick the commit(s).

We should do a manual backport as well to avoid merge conflicts when backporting python packages.

@FliegendeWurst
Copy link
Copy Markdown
Member

I'm wondering, is there any point to including the "fixed" ordering in the CI log:

image

I think it would be enough to report the underlined path, and maybe the affected region. The gigantic "new_content" field below that is not really helping, at least for python-packages.nix. Maybe it helps on smaller files?

@FliegendeWurst
Copy link
Copy Markdown
Member

Also, now that we have three workflows that install stuff via pinned nixpkgs, we often get 429 rate limited on the tarball for that nixpkgs revision.

https://github.com/NixOS/nixpkgs/actions/runs/14143005549/job/39626911331?pr=341767

@katexochen
Copy link
Copy Markdown
Contributor Author

Also, now that we have three workflows that install stuff via pinned nixpkgs, we often get 429 rate limited on the tarball for that nixpkgs revision.

https://github.com/NixOS/nixpkgs/actions/runs/14143005549/job/39626911331?pr=341767

@FliegendeWurst any idea how to fix this?

@FliegendeWurst
Copy link
Copy Markdown
Member

I see three options:

  • combine all uses of nix-env into one workflow
  • install packages without fetching the nixpkgs tarball
  • cache nix-env result using the GH Actions cache

Option 1 could still lead to failures, but is less likely.
Option 2 is tricky to implement, I think.
Option 3 is probably the easiest fix.

@wolfgangwalther
Copy link
Copy Markdown
Contributor

I think we should run more workflows in one job in general. We seem to have quite a few workflows, where the time it takes to run the actual check is far less than the time it takes to set up the whole job, check out the repo, etc. - quite a waste of resources, imho.

Imho, all the "checks" could go into one workflow, because they are only breaking very rarely anyway. We might need to be smart about it, so that all the checks are still run, even if one of the prior steps fails

This will also make the huge job list for each PR less cluttered.

@katexochen
Copy link
Copy Markdown
Contributor Author

install packages without fetching the nixpkgs tarball

We could use oras to install binaries in a content addressed way, storing them in ghcr.io (which has far less rate limiting).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 6.topic: policy discussion Discuss policies to work in and around Nixpkgs 6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 12.approvals: 2 This PR was reviewed and approved by two persons. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants