Skip to content

bash: default to 5#131681

Closed
happysalada wants to merge 614 commits intoNixOS:stagingfrom
happysalada:bash_default_upgrade
Closed

bash: default to 5#131681
happysalada wants to merge 614 commits intoNixOS:stagingfrom
happysalada:bash_default_upgrade

Conversation

@happysalada
Copy link
Copy Markdown
Contributor

@happysalada happysalada commented Jul 27, 2021

Motivation for this change

This is just a test to see if defaulting to bash 5 and removing old bash 4.4 works properly.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • 21.11 Release Notes (or backporting 21.05 Relase notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@happysalada happysalada marked this pull request as draft July 27, 2021 12:41
@happysalada happysalada mentioned this pull request Jul 27, 2021
Merged
11 tasks
@happysalada happysalada marked this pull request as ready for review July 28, 2021 03:35
@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Jul 29, 2021

@GrahamcOfBorg eval

@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Jul 31, 2021 •
edited
Loading

I've checked the error on my local, here is what I found out

local input=/nix/store/c9w69kl4caarcs2j84hck0icrdj9jqr6-fix-darwin-dylib-names.sh
++ local output=/nix/store/wiqpq8fwh190c9msc8iq68ai6cszca88-hook/nix-support/setup-hook
++ shift 2
++ '[' '!' -f /nix/store/c9w69kl4caarcs2j84hck0icrdj9jqr6-fix-darwin-dylib-names.sh ']'
++ local content
++ consumeEntire content
++ IFS=
++ read -r -N 0 content
++ echo 'consumeEntire(): ERROR: Input null bytes, won'\''t process'
consumeEntire(): ERROR: Input null bytes, won't process
++ return 1

the file

❯ cat /nix/store/c9w69kl4caarcs2j84hck0icrdj9jqr6-fix-darwin-dylib-names.sh
# On macOS, binaries refer to dynamic library dependencies using
# either relative paths (e.g. "libicudata.dylib", searched relative to
# $DYLD_LIBRARY_PATH) or absolute paths
# (e.g. "/nix/store/.../lib/libicudata.dylib").  In Nix, the latter is
# preferred since it allows programs to just work.  When linking
# against a library (e.g. "-licudata"), the linker uses the install
# name embedded in the dylib (which can be shown using "otool -D").
# Most packages create dylibs with absolute install names, but some do
# not.  This setup hook fixes dylibs by setting their install names to
# their absolute path (using "install_name_tool -id").  It also
# rewrites references in other dylibs to absolute paths.

fixupOutputHooks+=('fixDarwinDylibNamesIn $prefix')

fixDarwinDylibNames() {
    local flags=()
    local old_id

    for fn in "$@"; do
        flags+=(-change "$(basename "$fn")" "$fn")
    done

    for fn in "$@"; do
        if [ -L "$fn" ]; then continue; fi
        echo "$fn: fixing dylib"
        int_out=$(@targetPrefix@install_name_tool -id "$fn" "${flags[@]}" "$fn" 2>&1)
        result=$?
        if [ "$result" -ne 0 ] &&
            ! grep "shared library stub file and can't be changed" <<< "$out"
        then
            echo "$int_out" >&2
            exit "$result"
        fi
    done
}

fixDarwinDylibNamesIn() {
    local dir="$1"
    fixDarwinDylibNames $(find "$dir" -name "*.dylib")
}

I'm trying to add a debug statement to see where the null byte is encountered.
I couldn't find anything that would say that the behavior of read had changed between bash versions.
I also have no idea why there is a null byte check and an exit. From my knowledge, there can be no null byte in strings in bash except if you deliberately put one with printf.
I've tried to consumeEntire the file on local and did not encounter any errors in both bash 4.4 and 5.1

@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Aug 2, 2021

I've found 2 previous attempt and one person suggested a solution
#108101

second attempt is older and just like mine so far
#53687

I'm trying the fix documented in the first link. In bash 5 the behavior of read -N 0 has changed and that was not documented anywhere. I'm testing this on local, I will push the commit after.

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Aug 2, 2021
@happysalada happysalada mentioned this pull request Aug 2, 2021
Closed
06kellyjac
06kellyjac reviewed Aug 2, 2021
View reviewed changes
pkgs/stdenv/generic/setup.sh Outdated
Copy link
Copy Markdown
Member

@06kellyjac 06kellyjac Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this covered by line 13?

Copy link
Copy Markdown
Contributor Author

@happysalada happysalada Aug 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is a leftover from my debugging. Sorry for leaving that!
For some reason I can't get it to work with NIX_DEBUG, so I just forcefully added it.
My bad. Removing it from the commit

I'm under fish shell and just doing
env NIX_DEBUG=10 nix-build ...
doesn't work. When I try to echo NIX_DEBUG in setup.sh it exits with an unbound variable error. Maybe the way to enable NIX_DEBUG is with the -vvv flag. I have to say I don't know here.

@ofborg ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. 8.has: clean-up This PR removes packages or removes other cruft labels Aug 2, 2021
@ofborg ofborg bot requested review from andir, edolstra, lovek323 and np August 2, 2021 11:58
@ofborg ofborg bot added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Aug 2, 2021
@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Aug 3, 2021 •
edited
Loading

The next two errors are test errors while building nix.

running test tests/user-envs.sh... [FAIL]
    + '[' -z '' ']'
    + clearStore
    + echo 'clearing store...'
    clearing store...
    + chmod -R +w /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + nix-store --init
    + clearProfiles
    + profiles=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + clearProfiles
    + profiles=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    ++ nix-env -p /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles/test -q '*'
    ++ wc -l
    + test 0 -eq 0
    + mkdir -p /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/test-home
    + nix-env --switch-profile /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles/test
    ++ nix-env -f ./user-envs.nix -qa '*'
    ++ wc -l
    + test 6 -eq 6
    ++ nix-env -f ./user-envs.nix -qa --out-path --no-name '*'
    ++ grep foo-1.0
    + outPath10=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/4rchg0azqmx526arwhv8a7b3817q1c3x-foo-1.0
    ++ nix-env -f ./user-envs.nix -qa --drv-path --no-name '*'
    ++ grep foo-1.0
    + drvPath10=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/sq8ikm091c6jldai40sdp4ax420mab5f-foo-1.0.drv
    + '[' -n /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/4rchg0azqmx526arwhv8a7b3817q1c3x-foo-1.0 -a -n /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/sq8ikm091c6jldai40sdp4ax420mab5f-foo-1.0.drv ']'
    + nix-env -f ./user-envs.nix -qa '*' --description
    + grep -q silly
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/test-home/.nix-defexpr
    ++ pwd
    + ln -s /private/tmp/nix-build-nix-2.3.14.drv-0/nix-2.3.14/tests/user-envs.nix /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/test-home/.nix-defexpr
    + nix-env -qa '*' --description
    + grep -q silly
    + nix-env -qa '*' --system
    + grep -q x86_64-darwin
    + nix-env -i foo-1.0
    installing 'foo-1.0'
    these derivations will be built:
      /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/sq8ikm091c6jldai40sdp4ax420mab5f-foo-1.0.drv
    building '/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/sq8ikm091c6jldai40sdp4ax420mab5f-foo-1.0.drv'...
    error: unexpected EOF reading a line

running test tests/fetchurl.sh... [FAIL]
    + clearStore
    + echo 'clearing store...'
    clearing store...
    + chmod -R +w /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + nix-store --init
    + clearProfiles
    + profiles=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    ++ nix-hash --flat --type sha256 ./fetchurl.sh
    + hash=9b1d81c5333e8c25296214cbf642e479131b8437f69d6e50cde80029a9e55e84
    +++ pwd
    ++ nix-build '<nix/fetchurl.nix>' --argstr url file:///private/tmp/nix-build-nix-2.3.14.drv-0/nix-2.3.14/tests/fetchurl.sh --argstr sha256 9b1d81c5333e8c25296214cbf642e479131b8437f69d6e50cde80029a9e55e84 --no-out-link --hashed-mirrors ''
    these derivations will be built:
      /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/jcmp6jplgqc3vdwj1pp3nxs5fj7gvqdk-fetchurl.sh.drv
    building '/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/jcmp6jplgqc3vdwj1pp3nxs5fj7gvqdk-fetchurl.sh.drv'...
    + outPath=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/zr0g75mi67dazv4hby9lgj8jk14avh2v-fetchurl.sh
    + cmp /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/zr0g75mi67dazv4hby9lgj8jk14avh2v-fetchurl.sh fetchurl.sh
    + clearStore
    + echo 'clearing store...'
    clearing store...
    + chmod -R +w /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + nix-store --init
    + clearProfiles
    + profiles=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    ++ nix hash-file --type sha512 --base64 ./fetchurl.sh
    + hash=aKjXWoXYIDfLkn4MV1P6FR/rkiXOqps6A4J3cWCYc1G3mgtTbIK5/YY7wXuExg/ltSf8JHlD25wCyovEU6ZI9Q==
    +++ pwd
    ++ nix-build '<nix/fetchurl.nix>' --argstr url file:///private/tmp/nix-build-nix-2.3.14.drv-0/nix-2.3.14/tests/fetchurl.sh --argstr sha512 aKjXWoXYIDfLkn4MV1P6FR/rkiXOqps6A4J3cWCYc1G3mgtTbIK5/YY7wXuExg/ltSf8JHlD25wCyovEU6ZI9Q== --no-out-link --hashed-mirrors ''
    these derivations will be built:
      /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/v4wknf7v874g819pr9vc20rw5inywb31-fetchurl.sh.drv
    building '/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/v4wknf7v874g819pr9vc20rw5inywb31-fetchurl.sh.drv'...
    + outPath=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/qjwwfb7nq62w30pzmiijnl0289jfhnf7-fetchurl.sh
    + cmp /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/qjwwfb7nq62w30pzmiijnl0289jfhnf7-fetchurl.sh fetchurl.sh
    + clearStore
    + echo 'clearing store...'
    clearing store...
    + chmod -R +w /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + mkdir /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix
    + nix-store --init
    + clearProfiles
    + profiles=/private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    + rm -rf /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/var/nix/profiles
    ++ nix hash-file ./fetchurl.sh
    + hash=sha256-mx2BxTM+jCUpYhTL9kLkeRMbhDf2nW5QzegAKanlXoQ=
    + [[ sha256-mx2BxTM+jCUpYhTL9kLkeRMbhDf2nW5QzegAKanlXoQ= =~ ^sha256- ]]
    +++ pwd
    ++ nix-build '<nix/fetchurl.nix>' --argstr url file:///private/tmp/nix-build-nix-2.3.14.drv-0/nix-2.3.14/tests/fetchurl.sh --argstr hash sha256-mx2BxTM+jCUpYhTL9kLkeRMbhDf2nW5QzegAKanlXoQ= --no-out-link --hashed-mirrors ''
    these derivations will be built:
      /private/tmp/nix-build-nix-2.3.14.drv-0/nix-test/store/iwjcbsicdy9zbzr3mli6109c1lzf5zsv-fetchurl.sh.drv
    error: unexpected EOF reading a line
    + outPath=

it only took 19h on my machine to build.

Here are the problematic files
https://github.com/NixOS/nix/blob/master/tests/fetchurl.sh
https://github.com/NixOS/nix/blob/master/tests/user-envs.sh

I can't see anything problematic in those files. My initial guess is to try to build with -vvv flag. I'm not sure it will be more verbose for nix tests though. I don't know how I would go about changing the nix tests flags to be more verbose to get more output.

@happysalada happysalada force-pushed the bash_default_upgrade branch from e0a26bd to 96974fe Compare August 3, 2021 02:08
fabaff and others added 6 commits August 10, 2021 12:25
@fabaff
Merge pull request NixOS#129550 from fabaff/bump-b2sdk
e221bca 
python3Packages.b2sdk: 1.9.0 -> 1.11.0
@olebedev
nodePackages.node-gyp: local nodejs header files
98b9a7f 
From within the sandbox, it couldn't be possible to do HTTP requests so we need
to make `node-gyp` aware of local `nodejs` and use header files out of it
instead of attempt to download it and fail.
@SuperSandro2000
Merge pull request NixOS#133247 from fabaff/bump-coapthon3
7e060a5
@SuperSandro2000
Merge pull request NixOS#133120 from kidonng/iasl
b17e970
@ostylk
conda: invoke bash login shell to source /etc/profile
3747daf 
The `profile` parameter in `buildFHSUserEnv` is used to create an /etc/profile file
which should be sourced by bash. However, the default runScript is a non-login bash
which does not source /etc/profile. Therefore the conda package breaks, as a script
which is required for `conda activate`, is not sourced. Also the installationPath
parameter is ignored as it is setup in the /etc/profile file.
@fabaff
python3Packages.runway-python: fix build
110c632
@github-actions github-actions bot added 6.topic: emacs Text editor 6.topic: haskell General-purpose, statically typed, purely functional programming language 6.topic: kernel The Linux kernel 6.topic: lua Lua is a powerful, efficient, lightweight, embeddable scripting language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: ruby A dynamic, open source programming language with a focus on simplicity and productivity. 6.topic: vim Advanced text editor 6.topic: xfce The Xfce Desktop Environment 8.has: changelog This PR adds or changes release notes 8.has: documentation This PR adds or changes documentation 8.has: module (update) This PR changes an existing module in `nixos/` labels Aug 11, 2021
@06kellyjac
Copy link
Copy Markdown
Member

06kellyjac commented Aug 11, 2021

I think you just rebased off master instead of staging 🙂

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 11, 2021
@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Aug 11, 2021

Yes :-)
I'm creating a new branch and then It shows i should not stay in front of the computer anymore.

@06kellyjac 06kellyjac mentioned this pull request Aug 11, 2021
Closed
11 tasks
@happysalada
Copy link
Copy Markdown
Contributor Author

happysalada commented Aug 11, 2021

Recreated the PR leaving bash_4 around
#133495

Sorry about messing up the branch and notifying half of the world.

@06kellyjac 06kellyjac mentioned this pull request Aug 11, 2021
Merged
11 tasks
@happysalada happysalada deleted the bash_default_upgrade branch April 28, 2023 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SuperSandro2000 SuperSandro2000 SuperSandro2000 left review comments

@06kellyjac 06kellyjac 06kellyjac left review comments

@Ericson2314 Ericson2314 Awaiting requested review from Ericson2314

@matthewbauer matthewbauer Awaiting requested review from matthewbauer

@edolstra edolstra Awaiting requested review from edolstra

@np np Awaiting requested review from np

@andir andir Awaiting requested review from andir

@lovek323 lovek323 Awaiting requested review from lovek323

@adisbladis adisbladis Awaiting requested review from adisbladis

@cdepillabout cdepillabout Awaiting requested review from cdepillabout

@expipiplus1 expipiplus1 Awaiting requested review from expipiplus1

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@maralorn maralorn Awaiting requested review from maralorn

@marsam marsam Awaiting requested review from marsam

@mmahut mmahut Awaiting requested review from mmahut

@RaghavSood RaghavSood Awaiting requested review from RaghavSood

@roberth roberth Awaiting requested review from roberth

@sternenseemann sternenseemann Awaiting requested review from sternenseemann

@teto teto Awaiting requested review from teto

@WilliButz WilliButz Awaiting requested review from WilliButz

Assignees

No one assigned

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 6.topic: emacs Text editor 6.topic: haskell General-purpose, statically typed, purely functional programming language 6.topic: kernel The Linux kernel 6.topic: lua Lua is a powerful, efficient, lightweight, embeddable scripting language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: ruby A dynamic, open source programming language with a focus on simplicity and productivity. 6.topic: stdenv Standard environment 6.topic: vim Advanced text editor 6.topic: xfce The Xfce Desktop Environment 8.has: changelog This PR adds or changes release notes 8.has: clean-up This PR removes packages or removes other cruft 8.has: documentation This PR adds or changes documentation 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.