jq: 1.6 -> 1.7 by ncfavier · Pull Request #246515 · NixOS/nixpkgs

ncfavier · 2023-08-01T08:48:08Z

After five years, jq is getting a new release! https://github.com/jqlang/jq/releases/tag/jq-1.7

Not sure if this deserves a release note? The breaking changes seem relatively limited, but jq is kind of a core package.

Things done

ncfavier · 2023-08-27T20:03:57Z

jq 1.7rc2 introduces a cycle between outputs and I'm not sure how to resolve it. dev depends on bin because of the default propagatedBuildOutputs value, and bin now depends on dev because of jqlang/jq@583e4a2. We could:

Break the first link by not including bin in the propagated build outputs (I think this could break builds?)
Break the second link using removeReferencesTo
Merge bin and dev (how?)

Artturin · 2023-08-27T22:57:07Z

jq 1.7rc2 introduces a cycle between outputs and I'm not sure how to resolve it. dev depends on bin because of the default propagatedBuildOutputs value, and bin now depends on dev because of jqlang/jq@583e4a2. We could:
* Break the first link by not including `bin` in the propagated build outputs (I think this could break builds?)

Would break many builds because dev is the default output chosen.

* Break the second link using `removeReferencesTo`

This would be good.

* Merge `bin` and `dev` (how?)

By removing dev from outputs.

ncfavier · 2023-08-28T07:58:39Z

This would be good.

Done that.

miallo · 2023-09-08T10:19:38Z

Nice that you are already onto this! 🎉

Unfortunately when I apply this patch I am running into a test failure. For anyone else also running into this, this seems to be tracked here: jqlang/jq#2885

ncfavier · 2023-09-08T10:24:09Z

Tests are passing for me (and for ofborg) on x86_64-linux, aarch64-linux and aarch64-darwin. Are you on x86_64-darwin?

What revision did you apply this patch to?

ncfavier · 2023-09-08T10:33:22Z

As far as I can tell this is a locale problem, so Nix builds should not be affected. Are you building with the sandbox enabled?

miallo · 2023-09-08T19:13:50Z

Tests are passing for me (and for ofborg) on x86_64-linux, aarch64-linux and aarch64-darwin. Are you on x86_64-darwin?

Interesting... No x86_64-linux (i.e. NixOS)

What revision did you apply this patch to?

Current nixos-unstable (0bffda1). Only other two commits in my fork are unrelated (they add two options to nixos-rebuild).

As far as I can tell this is a locale problem, so Nix builds should not be affected. Are you building with the sandbox enabled?

Nope, just a normal nixos-rebuild switch... 🤔

ncfavier · 2023-09-08T19:21:22Z

Nope, just a normal nixos-rebuild switch... 🤔

Well then the sandbox should be enabled by default. Please ensure that your /etc/nix/nix.conf contains sandbox = true.

ncfavier · 2023-09-08T19:22:36Z

Can you paste the .drv file that fails to build somewhere? (name and contents)

miallo · 2023-09-08T19:28:48Z

Please ensure that your /etc/nix/nix.conf contains sandbox = true.

It does

Can you paste the .drv file that fails to build somewhere? (name and contents)

Oh... Damn... I only saw the error with jq, but in fact it is from python3.10-yq:

/nix/store/y0nkkmhxhc3fkhc3q0wwqjm0ks8lqh08-python3.10-yq-3.2.2.drv

Derive([("dist","/nix/store/4g45xi9866myvha3vxq8yahm29gxpsga-python3.10-yq-3.2.2-dist","",""),("out","/nix/store/64lyjyh5q39akgwcd71yj53yaw4i15k4-python3.10-yq-3.2.2","","")],[("/nix/store/02kq0w1zn1v9qzvanz1gvsds6vf1kkh6-python3.10-argcomplete-3.1.1.drv",["out"]),("/nix/store/0sawbbskl41aq4ii9kl5w77r4m92das8-bash-5.2-p15.drv",["out"]),("/nix/store/1ybxfm6v8ijnr72wm1fx4d3ry59256sr-pypa-install-hook.drv",["out"]),("/nix/store/4lky73gnpd3k90hlhlybik83hv5b12a4-python-remove-tests-dir-hook.drv",["out"]),("/nix/store/599xrhrkyx743jz8lsqjmhhjynswr87f-python3.10-xmltodict-0.13.0.drv",["out"]),("/nix/store/7miy4j2xy6p97f34savhl0jsgy5syf0b-python-catch-conflicts-hook.drv",["out"]),("/nix/store/9663sml6h17kv61wg8am9dmpvfccwrq3-setuptools-check-hook.drv",["out"]),("/nix/store/9fwlz0myh3lwwmj3rcic4ssv7ygrig6x-stdenv-linux.drv",["out"]),("/nix/store/9r57db02g9pvj2fl914m5y9xjma26zr5-pytest-check-hook.drv",["out"]),("/nix/store/9zjcxviqkl4mr6wsvs9554w5ywnr3fas-python3.10-setuptools-scm-7.1.0.drv",["out"]),("/nix/store/bcchpqhqph735wvgkvnf3bc6mzfwjj8g-python3.10-tomlkit-0.12.1.drv",["out"]),("/nix/store/hw5ygglyq53qzx75gdv01wh8qnmd66hq-wrap-python-hook.drv",["out"]),("/nix/store/l0kin7jf2qm401vzqrlrd2cb7zz1x8ah-jq-path.patch.drv",["out"]),("/nix/store/mpaa4mcp8zvhxc5mg10cwmnmhqkzvlak-python-imports-check-hook.sh.drv",["out"]),("/nix/store/mv79m4vxlqrjsasnwf2j9gaa256vyi6h-python3-3.10.12.drv",["out"]),("/nix/store/mx50p8hbrwg9xiimaanmyg7yyq6szdpf-python3.10-pyyaml-6.0.1.drv",["out"]),("/nix/store/n66c5lb52caw80dcyacp2c61bz88vk46-python-remove-bin-bytecode-hook.drv",["out"]),("/nix/store/pqmn7mhg2fz0dz58a7h1sxbd4vvgznv1-ensure-newer-sources-hook.drv",["out"]),("/nix/store/s894x9568fhi765589235am0yjnw2m00-yq-3.2.2.tar.gz.drv",["out"]),("/nix/store/vbhxxiq5ylm82qdpds92ybjpv7hbmc9c-python-namespaces-hook.sh.drv",["out"]),("/nix/store/yav55g1lzz5smcm6lfvjkiczaacx80gz-setuptools-setup-hook.drv",["out"]),("/nix/store/ysl7znxpxa7ibnxpiflgn8mbvls8zx0d-python-output-dist-hook.drv",["out"])],["/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"],"x86_64-linux","/nix/store/m36d29gn5gm9bk0g7fcln1v8171hvn95-bash-5.2-p15/bin/bash",["-e","/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"],[("LANG","C.UTF-8"),("__structuredAttrs",""),("buildInputs",""),("builder","/nix/store/m36d29gn5gm9bk0g7fcln1v8171hvn95-bash-5.2-p15/bin/bash"),("cmakeFlags",""),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("disallowedReferences",""),("dist","/nix/store/4g45xi9866myvha3vxq8yahm29gxpsga-python3.10-yq-3.2.2-dist"),("doCheck",""),("doInstallCheck","1"),("mesonFlags",""),("name","python3.10-yq-3.2.2"),("nativeBuildInputs","/nix/store/vagb0sjv83ybi435i6iiv10hjrdghph9-python3-3.10.12 /nix/store/cwcdrvv94c4rys2cgxhs2byn6a4rczgr-wrap-python-hook /nix/store/2ay7q7ligvy0zkhb22z9hb6ksk68rgcf-ensure-newer-sources-hook /nix/store/4bm7si6b8hjfr3vncl1b3phckawdbhah-python-remove-tests-dir-hook /nix/store/3l5vkplnmwxdmccajy7knsx6hrzj9jf7-python-catch-conflicts-hook /nix/store/ihc2jyzhx3crnlws34qhhv4d73zp6zy6-python-remove-bin-bytecode-hook /nix/store/b9284xij30zf4zfmqvvamqag3gzfqrmh-setuptools-setup-hook /nix/store/414d6mi9bw0d3p48sy5z3468n445kzz8-pypa-install-hook /nix/store/l2lrx1kx0phn947rmylddrday6l6z1ig-python-imports-check-hook.sh /nix/store/4x04hyzyz8pfh2dqdvx7j5svp315j66k-python-namespaces-hook.sh /nix/store/9gcs29995dq04c8r8sx46nxczz083vw1-python-output-dist-hook /nix/store/6bx9kdapg02a19mkrhdyhdl4s0rnajyv-python3.10-setuptools-scm-7.1.0 /nix/store/47y9smr6b6g7542p1iahj1qzv4ncdr7a-setuptools-check-hook /nix/store/rkapa2ajfn7bphiy98bjyrgnfgvb2isa-pytest-check-hook"),("out","/nix/store/64lyjyh5q39akgwcd71yj53yaw4i15k4-python3.10-yq-3.2.2"),("outputs","out dist"),("patches","/nix/store/cvwrvbb09jqqz5mv2f4jl5rzzj58ycfv-jq-path.patch"),("pname","yq"),("postFixup","wrapPythonPrograms\n"),("propagatedBuildInputs","/nix/store/pngxdvmmc32m3hsy8psnxbk2rfrkdqyx-python3.10-argcomplete-3.1.1 /nix/store/31zpzgn2hr04mmg7dqp8gkf5n99i29ab-python3.10-pyyaml-6.0.1 /nix/store/r6ii1v9v4wrr6m29dksl5raqjn29q95n-python3.10-tomlkit-0.12.1 /nix/store/mdbiflqwg46s7yn46cww05zcqp52709z-python3.10-xmltodict-0.13.0 /nix/store/vagb0sjv83ybi435i6iiv10hjrdghph9-python3-3.10.12"),("propagatedNativeBuildInputs",""),("pytestFlagsArray","test/test.py"),("pythonImportsCheck","yq"),("src","/nix/store/7a84byjhj33rrc5yjd4mkz1yqiwkc6pn-yq-3.2.2.tar.gz"),("stdenv","/nix/store/7cni7ndy2pm18ysl5znq6znb30sxp156-stdenv-linux"),("strictDeps","1"),("system","x86_64-linux"),("version","3.2.2")])

ncfavier · 2023-09-08T19:32:28Z

Is jq failing to build or not? If so, please paste that .drv file. This PR is not about yq.

miallo · 2023-09-08T20:18:30Z

Sorry for the confusion - the jq–build is indeed working, but pkgs.yq depends on jq and with this PR yq fails the tests.

AFAICT the issue is indeed because of a change of behaviour of jq of something in the yq test and I raised an issue there kislyuk/yq#172

miallo · 2023-09-08T21:09:02Z

The reason why I directly jumped to the conclusion that it had to do with the mentioned jq issue is that that log contained the lines:

==57649== Command: /usr/ports/textproc/jq/work/jq-jq-1.7/jq -b -n --jsonargs null invalid
==57649== 
/usr/ports/textproc/jq/work/jq-jq-1.7/jq: invalid JSON text passed to --jsonargs
Use /usr/ports/textproc/jq/work/jq-jq-1.7/jq --help for help with command-line options,
or see the jq manpage, or online docs  at https://jqlang.github.io/jq

which was pretty much what I saw in the error message so I felt confirmed. I did not question my beliefs. Sorry for that!

What is the procedure for such a dependency failure? This PR in itself is great, but I don't know how to deal with it breaking other packages...

ncfavier · 2023-09-08T21:13:40Z

We can wait for known problems to be resolved upstream and bump the reverse dependencies simultaneously.

python3Packages.jq also seems to have failing tests.

miallo · 2023-09-09T07:13:34Z

python3Packages.jq also seems to have failing tests.

Good catch! I created an issue over there as well: mwilliamson/jq.py#93

ncfavier · 2023-09-10T10:15:01Z

Successfully built nix.

Artturin · 2023-09-11T18:22:40Z

Added a release note

nixos/doc/manual/release-notes/rl-2311.section.md

benley · 2023-09-16T17:12:35Z

Aside from the pending comment about the release notes, are there any remaining objections to merging this change?

ofborg bot requested review from 7c6f434c, Artturin and globin August 1, 2023 09:55

Artturin mentioned this pull request Aug 3, 2023

jq: 1.6 -> 2022-05-26 #217345

Closed

12 tasks

ncfavier force-pushed the jq branch from 2ce6ccc to 8379732 Compare August 28, 2023 07:57

ncfavier changed the title ~~jq: 1.6 -> 1.7rc1~~ jq: 1.6 -> 1.7rc2 Aug 30, 2023

ncfavier force-pushed the jq branch from 8379732 to d82b721 Compare September 7, 2023 07:48

ncfavier marked this pull request as ready for review September 7, 2023 07:48

ncfavier changed the title ~~jq: 1.6 -> 1.7rc2~~ jq: 1.6 -> 1.7 Sep 7, 2023

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Sep 10, 2023

python3Packages.jq: 1.4.1 -> 1.5.0, apply patch for jq 1.7

37f76d3

ncfavier force-pushed the jq branch from 6872b6b to 37f76d3 Compare September 10, 2023 10:14

ofborg bot requested review from SuperSandro2000 and benley September 10, 2023 11:22

miallo mentioned this pull request Sep 10, 2023

nixos-rebuild: Add list-generations #250501

Merged

12 tasks

miallo approved these changes Sep 10, 2023

View reviewed changes

delroth added the 12.approvals: 1 This PR was reviewed and approved by one person. label Sep 10, 2023

benley approved these changes Sep 11, 2023

View reviewed changes

delroth added 12.approvals: 2 This PR was reviewed and approved by two persons. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Sep 11, 2023

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation This PR adds or changes documentation 8.has: changelog This PR adds or changes release notes labels Sep 11, 2023

delroth removed the 12.approvals: 2 This PR was reviewed and approved by two persons. label Sep 11, 2023

ofborg bot requested a review from benley September 11, 2023 18:48

benley approved these changes Sep 11, 2023

View reviewed changes

delroth added the 12.approvals: 2 This PR was reviewed and approved by two persons. label Sep 11, 2023

7c6f434c approved these changes Sep 12, 2023

View reviewed changes

delroth added 12.approvals: 3+ This PR was reviewed and approved by three or more persons. and removed 12.approvals: 2 This PR was reviewed and approved by two persons. labels Sep 12, 2023

SuperSandro2000 reviewed Sep 12, 2023

View reviewed changes

nixos/doc/manual/release-notes/rl-2311.section.md Outdated Show resolved Hide resolved

jq: add a release note

09e3d7d

ncfavier force-pushed the jq branch from a54dc7d to 09e3d7d Compare September 16, 2023 17:14

ncfavier merged commit 57b9e41 into NixOS:staging Sep 16, 2023

ncfavier deleted the jq branch September 16, 2023 17:15

Artturin mentioned this pull request Sep 16, 2023

Update request: jq 1.6 → 1.7 #255527

Closed

1 task

Uh oh!

Conversation

ncfavier commented Aug 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

ncfavier commented Aug 27, 2023

Uh oh!

Artturin commented Aug 27, 2023

Uh oh!

ncfavier commented Aug 28, 2023

Uh oh!

miallo commented Sep 8, 2023

Uh oh!

ncfavier commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncfavier commented Sep 8, 2023

Uh oh!

miallo commented Sep 8, 2023

Uh oh!

ncfavier commented Sep 8, 2023

Uh oh!

ncfavier commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miallo commented Sep 8, 2023

Uh oh!

ncfavier commented Sep 8, 2023

Uh oh!

miallo commented Sep 8, 2023

Uh oh!

miallo commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncfavier commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miallo commented Sep 9, 2023

Uh oh!

ncfavier commented Sep 10, 2023

Uh oh!

Artturin commented Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

benley commented Sep 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ncfavier commented Aug 1, 2023 •

edited

Loading

ncfavier commented Sep 8, 2023 •

edited

Loading

ncfavier commented Sep 8, 2023 •

edited

Loading

miallo commented Sep 8, 2023 •

edited

Loading

ncfavier commented Sep 8, 2023 •

edited

Loading

Artturin commented Sep 11, 2023 •

edited

Loading