Skip to content

ci/eval: rewrite script to use nix-eval-jobs --select#1658

Merged
brianmcgillion merged 2 commits intomainfrom
nix-eval-jobs-ci
Dec 23, 2025
Merged

ci/eval: rewrite script to use nix-eval-jobs --select#1658
brianmcgillion merged 2 commits intomainfrom
nix-eval-jobs-ci

Conversation

@Mic92
Copy link
Copy Markdown
Contributor

@Mic92 Mic92 commented Dec 20, 2025 •
edited
Loading

Description of Changes

This way we do no longer need impure evaluation (side-effect of --expr)

Also improved the error output so it looks like this:

[  24.3s] ✓ packages.aarch64-linux.hardware-scan
[  32.9s] ✓ packages.aarch64-linux.nvidia-jetson-orin-agx-release-nodemoapps
[  36.2s] ✓ packages.aarch64-linux.nxp-imx8mp-evk-release
[  37.7s] ✓ packages.x86_64-linux.audit-rules
[  37.9s] ✗ packages.x86_64-linux.demo-tower-mk1-debug-installer
[  38.0s] ✓ packages.x86_64-linux.ghaf-vms
[  38.2s] ✗ packages.x86_64-linux.lenovo-t14-amd-gen5-release-installer
[  38.4s] ✗ packages.x86_64-linux.lenovo-x1-carbon-gen11-release-installer
[  38.6s] ✗ packages.x86_64-linux.lenovo-x1-extras-debug-installer
[  40.6s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-debug-from-x86_64-flash-qspi
[  48.8s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-industrial-debug-nodemoapps-from-x86_64-flash-script
[  58.5s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-release-nodemoapps-from-x86_64
[  60.5s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx64-release-from-x86_64-flash-qspi
[  68.7s] ✓ packages.x86_64-linux.nvidia-jetson-orin-nx-debug-nodemoapps-from-x86_64-flash-script
[  69.7s] ✓ packages.x86_64-linux.rtl8126
[  79.1s] ✓ packages.x86_64-linux.vm-debug
============================================================
Evaluated 16 attributes in 79.2s
  ✓ 12 succeeded
  ✗ 4 failed
============================================================
Errors:
  packages.x86_64-linux.demo-tower-mk1-debug-installer:
    error:
           … from call site
             at «github:tiiuae/nixpkgs/f5588cc02080b7fcc2c47b5d7daf44fb3c5dd476?narHash=sha256-oOmNB5vpk3y%2B7DbGzYPwFwuDORqtJq5NW2BBBZi0Vcg%3D»/lib/attrsets.nix:1344:61:
             1343|   */
             1344|   genAttrs = names: f: genAttrs' names (n: nameValuePair n (f n));
                 |                                                             ^
             1345|
...

This way we make no longer need impure evaluation.

Type of Change

  • New Feature
  • Bug Fix
  • Improvement / Refactor

Related Issues / Tickets

Checklist

  • Clear summary in PR description
  • Detailed and meaningful commit message(s)
  • Commits are logically organized and squashed if appropriate
  • Contribution guidelines followed
  • Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
  • Author has run make-checks and it passes
  • All automatic GitHub Action checks pass - see actions
  • Author has added reviewers and removed PR draft status

Testing Instructions

Applicable Targets

  • Orin AGX aarch64
  • Orin NX aarch64
  • Lenovo X1 x86_64
  • Dell Latitude x86_64
  • System 76 x86_64

Installation Method

  • Requires full re-installation
  • Can be updated with nixos-rebuild ... switch
  • Other:

Test Steps To Verify:

  1. ...

github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 20, 2025
View reviewed changes
@Mic92 Mic92 force-pushed the nix-eval-jobs-ci branch 2 times, most recently from 5205a64 to 01447eb Compare December 20, 2025 08:22
@Mic92
Copy link
Copy Markdown
Contributor Author

Mic92 commented Dec 20, 2025

Accidentally, the sharding is now also better? It now takes consistently ~4min instead of 10m:

https://github.com/tiiuae/ghaf/actions/runs/20164147792/job/57883359067?pr=1634

@Mic92 Mic92 marked this pull request as ready for review December 20, 2025 10:39
@Mic92 Mic92 changed the title rewrite script to use nix-eval-jobs rewrite script to use nix-eval-jobs --select Dec 20, 2025
@Mic92 Mic92 changed the title rewrite script to use nix-eval-jobs --select ci/eval: rewrite script to use nix-eval-jobs --select Dec 20, 2025
Mic92 added 2 commits December 20, 2025 10:45
@Mic92
ci/eval: rewrite script to use nix-eval-jobs --select
045bc42 
This way we make no longer need impure evaluation.

Also improved the error output so it looks like this:

[  24.3s] ✓ packages.aarch64-linux.hardware-scan
[  32.9s] ✓ packages.aarch64-linux.nvidia-jetson-orin-agx-release-nodemoapps
[  36.2s] ✓ packages.aarch64-linux.nxp-imx8mp-evk-release
[  37.7s] ✓ packages.x86_64-linux.audit-rules
[  37.9s] ✗ packages.x86_64-linux.demo-tower-mk1-debug-installer
[  38.0s] ✓ packages.x86_64-linux.ghaf-vms
[  38.2s] ✗ packages.x86_64-linux.lenovo-t14-amd-gen5-release-installer
[  38.4s] ✗ packages.x86_64-linux.lenovo-x1-carbon-gen11-release-installer
[  38.6s] ✗ packages.x86_64-linux.lenovo-x1-extras-debug-installer
[  40.6s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-debug-from-x86_64-flash-qspi
[  48.8s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-industrial-debug-nodemoapps-from-x86_64-flash-script
[  58.5s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx-release-nodemoapps-from-x86_64
[  60.5s] ✓ packages.x86_64-linux.nvidia-jetson-orin-agx64-release-from-x86_64-flash-qspi
[  68.7s] ✓ packages.x86_64-linux.nvidia-jetson-orin-nx-debug-nodemoapps-from-x86_64-flash-script
[  69.7s] ✓ packages.x86_64-linux.rtl8126
[  79.1s] ✓ packages.x86_64-linux.vm-debug
============================================================
Evaluated 16 attributes in 79.2s
  ✓ 12 succeeded
  ✗ 4 failed
============================================================
Errors:
  packages.x86_64-linux.demo-tower-mk1-debug-installer:
    error:
           … from call site
             at «github:tiiuae/nixpkgs/f5588cc02080b7fcc2c47b5d7daf44fb3c5dd476?narHash=sha256-oOmNB5vpk3y%2B7DbGzYPwFwuDORqtJq5NW2BBBZi0Vcg%3D»/lib/attrsets.nix:1344:61:
             1343|   */
             1344|   genAttrs = names: f: genAttrs' names (n: nameValuePair n (f n));
                 |                                                             ^
             1345|

The script is now overall a bit simpler what we had before

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
@Mic92
ci/eval: run on ubuntu-24.04-arm
96f3c13 
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
brianmcgillion
brianmcgillion reviewed Dec 21, 2025
View reviewed changes
henrirosten
henrirosten reviewed Dec 22, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@henrirosten henrirosten left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Mic92, this looks really good!
Just one question/comment:

@henrirosten
Copy link
Copy Markdown
Collaborator

henrirosten commented Dec 22, 2025 •
edited
Loading

Accidentally, the sharding is now also better? It now takes consistently ~4min instead of 10m:

https://github.com/tiiuae/ghaf/actions/runs/20164147792/job/57883359067?pr=1634

I also believe that explains most of the execution time improvement. Before this change, some eval workers got assigned many of the ghaf outputs that take longer time to evaluate. Therefore, some eval workers ran in ~3 minutes, while others took more than 10 minutes.

Also, you are now running with --no-instantiate so you avoid writing .drv files, which I believe also noticeably improves eval time (it was completely unnecessary to write them earlier too).

As you wrote above, perhaps the arm runners are also somewhat faster for the eval execution.

@henrirosten henrirosten mentioned this pull request Dec 22, 2025
Closed
@brianmcgillion brianmcgillion merged commit 9d23a68 into main Dec 23, 2025
32 checks passed
@brianmcgillion brianmcgillion deleted the nix-eval-jobs-ci branch December 23, 2025 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brianmcgillion brianmcgillion brianmcgillion approved these changes

@henrirosten henrirosten henrirosten approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Mic92 @henrirosten @brianmcgillion @github-advanced-security