jq: 1.6 -> 2022-05-26 by edef1c · Pull Request #217345 · NixOS/nixpkgs

edef1c · 2023-02-20T18:49:12Z

Description of changes

jq has not seen a release in four years, despite a bug that silently mangles large numbers being fixed several years ago. Let's just use git HEAD instead.

Things done

lf- · 2023-02-20T18:57:25Z

merge conflict fyi

jq has not seen a release in four years, despite a bug that silently mangles large numbers being fixed several years ago. Let's just use git HEAD instead.

lf-

Nice. fetchTarball is a good solution to avoiding depending on GitHub archive hashes

cole-h · 2023-02-21T15:55:39Z

pkgs/development/tools/jq/default.nix

-  src = fetchurl {
-    url = "https://github.com/stedolan/jq/releases/download/jq-${version}/jq-${version}.tar.gz";
-    sha256 = "sha256-XejI4pqqP7nMa0e7JymfJxNU67clFOOsytx9OLW7qnI=";
+  src = builtins.fetchTarball {


We can't use builtins.fetchTarball because fetching then happens at eval time. This should work fine if it switches back to fetchurl.

The trouble is that the hash is then of the archive itself which is not guaranteed stable. that said, bazel seems to have Hyrum's law'd it onto github's priority list to keep them stable even though it's not guaranteed.

Ah, sorry -- I meant fetchzip, which will unpack the resulting tarball and gives me the same hash:

$ nix repl '<nixpkgs>' # [...snip...] nix-repl> :b pkgs.fetchzip { url = "https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz"; sha256 = ""; } warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' error: hash mismatch in fixed-output derivation '/nix/store/hhilyllp37xpf5shzmnjf1bdkr3fis3d-source.drv': specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= got: sha256-gsngye7EAmdSqH7ZH1g8/sQFj1nJWEGHpt48T8qXjZ0= nix-repl> builtins.fetchTarball { url = "https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz"; sha256 = ""; } warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' error: … while calling the 'fetchTarball' builtin at «string»:1:1: 1| builtins.fetchTarball { url = "https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz"; sha256 = ""; } | ^ error: hash mismatch in file downloaded from 'https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz': specified: sha256:0000000000000000000000000000000000000000000000000000 got: sha256:17cdjz54yg6yls3l2n69b67hbi7y7ic1znbym196f0n4xv4y1jc2 nix-repl> $ nix hash to-sri 'sha256:17cdjz54yg6yls3l2n69b67hbi7y7ic1znbym196f0n4xv4y1jc2' sha256-gsngye7EAmdSqH7ZH1g8/sQFj1nJWEGHpt48T8qXjZ0=

builtins.fetchTarball is not allowed in Nixpkgs (and is why ofborg fails): it moves the source fetching from build time to eval time, which can easily balloon said eval time.

Right. fetchFromGitHub is a thin wrapper around fetchzip which i imagine isn't going to be acceptable per the comment above, however i may be wrong.

Oh, indeed, you are right. I missed that comment.

How about manually doing what fetchzip does? It avoids depending on unzip but should avoid the issue um unstable archive hashes.

Suggested change

src = builtins.fetchTarball {

src = fetchurl {

url = "https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz";

sha256 = "sha256-gsngye7EAmdSqH7ZH1g8/sQFj1nJWEGHpt48T8qXjZ0=";

recursiveHash = true;

downloadToTemp = true;

postFetch = ''

unpackDir="$TMPDIR/unpack"

mkdir "$unpackDir"

cd "$unpackDir"

renamed="$TMPDIR/jq.tar.gz"

mv "$downloadedFile" "$renamed"

tar -xf "$renamed"

chmod -R +w "$unpackDir"

fn=$(cd "$unpackDir" && ls -A)

mv "$unpackDir/$fn" "$out"

chmod 755 "$out"

'';

};

I'm not quite following, what's the issue with just using fetchFromGitHub?

If you change it to fetchFromGitHub (same with fetchzip), you get this when trying to access jq from stdenv.__bootPackages:

$ nix-instantiate -A stdenv.__bootPackages.jq error: … while calling the 'derivationStrict' builtin at <nix/derivation-internal.nix>:9:12: 8| 9| strict = derivationStrict drvAttrs; | ^ 10| … while evaluating derivation 'jq-1.6' whose name attribute is located at /home/vin/workspace/vcs/nixpkgs/master/pkgs/stdenv/generic/make-derivation.nix:293:7 … while evaluating attribute 'src' of derivation 'jq-1.6' at /home/vin/workspace/vcs/nixpkgs/master/pkgs/development/tools/jq/default.nix:15:3: 14| # Note: do not use fetchpatch or fetchFromGitHub to keep this package available in __bootPackages 15| src = fetchFromGitHub { | ^ 16| # url = "https://github.com/stedolan/jq/releases/download/jq-${version}/jq-${version}.tar.gz"; (stack trace truncated; use '--show-trace' to show the full trace) error: function 'anonymous lambda' called with unexpected argument 'nativeBuildInputs' at /home/vin/workspace/vcs/nixpkgs/master/pkgs/build-support/fetchurl/boot.nix:5:1: 4| 5| { url ? builtins.head urls | ^ 6| , urls ? []

Ah, so jq is used during bootstrapping? That's the piece of information I was missing.

cole-h · 2023-02-21T15:56:44Z

pkgs/development/tools/jq/default.nix

 stdenv.mkDerivation rec {
  pname = "jq";
-  version = "1.6";
+  version = "2022-05-26";


Suggested change

version = "2022-05-26";

version = "unstable-2022-05-26";

Since this isn't a release, we should prefix with unstable- to prevent version sorting from thinking the date is an actual version.

The configure script for jq 1.6 does not define certain symbols required on Darwin to make `drem`, `significand`, and `lgamma_r` available. This works fine on clang 11 but fails with clang 16. This is fixed upstream. It can be removed once NixOS#217345 is merged or jq is released and updated to 1.7.

lheckemann · 2023-07-13T10:06:36Z

Looking at jqlang/jq#2305, https://github.com/jqlang/jq might be sensible to use as the "new upstream". It seems to be run by the (most recently) active maintainers of stedolan/jq.

Artturin · 2023-08-03T23:32:07Z

#246515

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Feb 20, 2023

lf- approved these changes Feb 20, 2023

View reviewed changes

edef1c added 2 commits February 20, 2023 19:11

jq: use builtins.fetchTarball

c96bf15

jq: 1.6 -> 2022-05-26

9b63cf4

jq has not seen a release in four years, despite a bug that silently mangles large numbers being fixed several years ago. Let's just use git HEAD instead.

edef1c force-pushed the jq branch from 09d82d9 to 9b63cf4 Compare February 20, 2023 19:14

edef1c removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Feb 20, 2023

lf- approved these changes Feb 20, 2023

View reviewed changes

cole-h requested changes Feb 21, 2023

View reviewed changes

reckenrode mentioned this pull request Jun 30, 2023

jq: fix build with clang 16 on Darwin #240640

Merged

12 tasks

alyssais closed this Sep 22, 2023

edef1c deleted the jq branch June 27, 2024 02:31

-  src = builtins.fetchTarball {
+  src = fetchurl {
+    url = "https://github.com/stedolan/jq/archive/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8.tar.gz";
+    sha256 = "sha256-gsngye7EAmdSqH7ZH1g8/sQFj1nJWEGHpt48T8qXjZ0=";
+    recursiveHash = true;
+    downloadToTemp = true;
+    postFetch = ''
+      unpackDir="$TMPDIR/unpack"
+      mkdir "$unpackDir"
+      cd "$unpackDir"
+      renamed="$TMPDIR/jq.tar.gz"
+      mv "$downloadedFile" "$renamed"
+      tar -xf "$renamed"
+      chmod -R +w "$unpackDir"
+      fn=$(cd "$unpackDir" && ls -A)
+      mv "$unpackDir/$fn" "$out"
+      chmod 755 "$out"
+    '';
+  };

Uh oh!

Conversation

edef1c commented Feb 20, 2023

Description of changes

Things done

Uh oh!

lf- commented Feb 20, 2023

Uh oh!

lf- left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lheckemann commented Jul 13, 2023

Uh oh!

Artturin commented Aug 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants