Skip to content

Add temporary workaround for missing subuid/subgid in Rawhide#1688

Merged
debarshiray merged 1 commit intocontainers:mainfrom
calancha:rawhide-subgid-subuid
Jul 25, 2025
Merged

Add temporary workaround for missing subuid/subgid in Rawhide#1688
debarshiray merged 1 commit intocontainers:mainfrom
calancha:rawhide-subgid-subuid

Conversation

@calancha
Copy link
Copy Markdown
Contributor

Some CI jobs on Fedora Rawhide fail with Podman rootless builds due to missing subuid/subgid ranges for the zuul-worker user:

Error: cannot find mappings for user zuul-worker in /etc/subuid

Fedora 42 and other distros already ship default ranges (e.g., zuul-worker:524288:65536), but Rawhide seems to lack them.

Add pre-tasks to setup-env.yaml to:

  • Dump current /etc/subuid and /etc/subgid contents for debug
  • Ensure the required ranges exist without overwriting existing configs

Once the Rawhide base image is fixed, this can be reverted.

@calancha calancha requested a review from debarshiray as a code owner July 22, 2025 10:32
calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
Some CI jobs on Fedora Rawhide fail with Podman rootless builds due to
missing subuid/subgid ranges for the zuul-worker user:

    Error: cannot find mappings for user zuul-worker in /etc/subuid

Fedora 42 and other distros already ship default ranges
(e.g., zuul-worker:524288:65536), but Rawhide seems to lack them.

Add pre-tasks to setup-env.yaml to:

- Dump current /etc/subuid and /etc/subgid contents for debug
- Ensure the required ranges exist without overwriting existing configs

Once the Rawhide base image is fixed, this can be reverted.

containers#1688
@calancha calancha force-pushed the rawhide-subgid-subuid branch from 530348a to bece3fb Compare July 22, 2025 10:34
@softwarefactory-project-zuul
Copy link
Copy Markdown

calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
Some CI jobs on Fedora Rawhide still failed after adding the missing
subuid/subgid ranges for zuul-worker, showing:

potentially insufficient UIDs or GIDs available in user namespace
(requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid
if configured locally and run "podman system migrate"

Add a podman system migrate step right after updating the mappings to
ensure Podman refreshes its user namespace configuration.

This can be reverted once the Rawhide base image ships proper defaults.

containers#1688
calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
The fix was first added to setup-env.yaml but ran too late in the CI job.
Since Podman is invoked early in dependencies-fedora.yaml, the workaround
now runs right after installing shadow-utils-subid-devel.

containers#1688
calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
Some CI jobs on Fedora Rawhide fail with Podman rootless builds due to
missing subuid/subgid ranges for the zuul-worker user:

    Error: cannot find mappings for user zuul-worker in /etc/subuid

Fedora 42 and other distros already ship default ranges
(e.g., zuul-worker:524288:65536), but Rawhide seems to lack them.

Add pre-tasks to setup-env.yaml to:

- Dump current /etc/subuid and /etc/subgid contents for debug
- Ensure the required ranges exist without overwriting existing configs

Once the Rawhide base image is fixed, this can be reverted.

containers#1688
calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
Some CI jobs on Fedora Rawhide still failed after adding the missing
subuid/subgid ranges for zuul-worker, showing:

potentially insufficient UIDs or GIDs available in user namespace
(requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid
if configured locally and run "podman system migrate"

Add a podman system migrate step right after updating the mappings to
ensure Podman refreshes its user namespace configuration.

This can be reverted once the Rawhide base image ships proper defaults.

containers#1688
calancha added a commit to calancha/toolbox that referenced this pull request Jul 22, 2025
The fix was first added to setup-env.yaml but ran too late in the CI job.
Since Podman is invoked early in dependencies-fedora.yaml, the workaround
now runs right after installing shadow-utils-subid-devel.

containers#1688
@calancha calancha force-pushed the rawhide-subgid-subuid branch from 589ef5f to 8f5b06a Compare July 22, 2025 12:49
@softwarefactory-project-zuul
Copy link
Copy Markdown

Copy link
Copy Markdown
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick workaround, @calancha ! I would have had to spend some time expanding my Ansible skills to do this myself. :)

This error seems unrelated, and is probably due to some change in Fedora Rawhide:

fedora-rawhide | not ok 186 run: Try /etc as a command
fedora-rawhide | # tags: commands-options
fedora-rawhide | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
fedora-rawhide | #  in test file test/system/104-run.bats, line 814)
fedora-rawhide | #   `assert_line --index 1 "bash: line 1: exec: /etc: cannot execute: Is a directory"' failed
fedora-rawhide | # ~ /home/zuul-worker/src/github.com/containers/toolbox
fedora-rawhide | #
fedora-rawhide | # -- line differs --
fedora-rawhide | # index    : 1
fedora-rawhide | # expected : bash: line 1: exec: /etc: cannot execute: Is a directory
fedora-rawhide | # actual   : Error: failed to invoke command /etc in container fedora-toolbox-43
fedora-rawhide | # --

calancha added a commit to calancha/toolbox that referenced this pull request Jul 23, 2025
Some CI jobs on Fedora Rawhide fail with Podman rootless builds due to
missing subuid/subgid ranges for the zuul-worker user:

    Error: cannot find mappings for user zuul-worker in /etc/subuid

Fedora 42 and other distros already ship default ranges
(e.g., zuul-worker:524288:65536), but Rawhide seems to lack them.

Add pre-tasks to dependencies-fedora.yaml to ensure the required ranges
exist without overwriting existing configs, and run `podman system migrate`
to refresh user namespace mappings.

Once the Rawhide base image is fixed, this can be reverted.

containers#1688
@calancha calancha force-pushed the rawhide-subgid-subuid branch from 8f5b06a to 047296a Compare July 23, 2025 06:47
@softwarefactory-project-zuul
Copy link
Copy Markdown

@Luap99
Copy link
Copy Markdown
Member

Luap99 commented Jul 23, 2025

FYI this may not be temporary, seems like useradd no longer assign a subuid/gid range by default: https://bugzilla.redhat.com/show_bug.cgi?id=2382662

@debarshiray
Copy link
Copy Markdown
Member

FYI this may not be temporary, seems like useradd no longer assign a subuid/gid range by default: https://bugzilla.redhat.com/show_bug.cgi?id=2382662

Thanks for the reference, @Luap99 ! It saved me a trip down the rabbit hole. :)

debarshiray pushed a commit to calancha/toolbox that referenced this pull request Jul 24, 2025
On Fedora 42 onwards, useradd(8) stopped automatically assigning
subordinate user and group ID ranges [1,2] to address a security concern
marked as CVE-2024-56433 [3].  This breaks rootless Podman and Skopeo,
and therefore Toolbx [4].

Restore the subordinate user and group ID ranges until a different
solution emerges.

[1] Fedora shadow-utils commit e1cfa31731cd68aa
    https://src.fedoraproject.org/rpms/shadow-utils/c/e1cfa31731cd68aa
    https://bugzilla.redhat.com/show_bug.cgi?id=2334168

[2] Fedora shadow-utils commit 4929903292e027ca
    https://src.fedoraproject.org/rpms/shadow-utils/c/4929903292e027ca
    https://bugzilla.redhat.com/show_bug.cgi?id=2334169

[3] shadow-maint/shadow#1157

[4] https://bugzilla.redhat.com/show_bug.cgi?id=2382662

containers#1688
@debarshiray debarshiray force-pushed the rawhide-subgid-subuid branch from 047296a to 1d65b85 Compare July 24, 2025 22:03
On Fedora 42 onwards, useradd(8) stopped automatically assigning
subordinate group and user ID ranges [1,2] to address a security concern
marked as CVE-2024-56433 [3].  This breaks rootless Podman and Skopeo,
and therefore Toolbx [4].

Restore the subordinate group and user ID ranges until a different
solution emerges.

[1] Fedora shadow-utils commit e1cfa31731cd68aa
    https://src.fedoraproject.org/rpms/shadow-utils/c/e1cfa31731cd68aa
    https://bugzilla.redhat.com/show_bug.cgi?id=2334168

[2] Fedora shadow-utils commit 4929903292e027ca
    https://src.fedoraproject.org/rpms/shadow-utils/c/4929903292e027ca
    https://bugzilla.redhat.com/show_bug.cgi?id=2334169

[3] shadow-maint/shadow#1157

[4] https://bugzilla.redhat.com/show_bug.cgi?id=2382662

containers#1688
@debarshiray debarshiray force-pushed the rawhide-subgid-subuid branch from 1d65b85 to a61b85c Compare July 24, 2025 22:09
@softwarefactory-project-zuul
Copy link
Copy Markdown

@debarshiray
Copy link
Copy Markdown
Member

I updated the commit message with references to the useradd(8) change and discussions, to clarify what's going on.

@debarshiray debarshiray merged commit a61b85c into containers:main Jul 25, 2025
2 of 3 checks passed
@debarshiray
Copy link
Copy Markdown
Member

Thank you for your contribution, @calancha !

DaliborKr added a commit to DaliborKr/toolbox that referenced this pull request Aug 7, 2025
In the new version of bash, the error output has been changed in
this situation:

GNU bash, version 5.2.37 (bash-5.2.37-1.fc42.x86_64):
  $ bash -c 'exec "$@"' bash /etc
  bash: line 1: /etc: Is a directory
  bash: line 1: exec: /etc: cannot execute: Is a directory

GNU bash, version 5.3.0 (bash-5.3.0-2.fc43.x86_64):
  $ bash -c 'exec "$@"' bash /etc
  bash: line 1: /etc: Is a directory

containers#1688 (review)
DaliborKr added a commit to DaliborKr/toolbox that referenced this pull request Aug 7, 2025
In the new version of bash, the error output has been changed in
this situation:

GNU bash, version 5.2.37 (bash-5.2.37-1.fc42.x86_64):
  $ bash -c 'exec "$@"' bash /etc
  bash: line 1: /etc: Is a directory
  bash: line 1: exec: /etc: cannot execute: Is a directory

GNU bash, version 5.3.0 (bash-5.3.0-2.fc43.x86_64):
  $ bash -c 'exec "$@"' bash /etc
  bash: line 1: /etc: Is a directory

containers#1688 (review)
debarshiray pushed a commit to DaliborKr/toolbox that referenced this pull request Aug 7, 2025
Bash 5.3.0 changed the error messages shown by its exec built-in [1].

With Bash 5.2.37:
  $ exec /etc
  bash: /etc: Is a directory
  bash: exec: /etc: cannot execute: Is a directory

With Bash 5.3.0:
  $ exec /etc
  bash: /etc: Is a directory

The 'assert' function cannot directly handle compound commands.  So,
those need to be wrapped in 'bash -c "..."' [2].

[1] Bash commit b8c60bc9ca365f82
    See how exec_builtin() handles EX_NOEXEC and EISDIR from
    shell_execve() to avoid printing a duplicate error message.
    https://cgit.git.savannah.gnu.org/cgit/bash.git/commit/?id=b8c60bc9ca365f82

[2] https://github.com/bats-core/bats-assert

containers#1688
containers#1699
debarshiray pushed a commit to DaliborKr/toolbox that referenced this pull request Aug 7, 2025
Bash 5.3.0 changed the error messages shown by its exec built-in [1].

With Bash 5.2.37:
  $ exec /etc
  bash: /etc: Is a directory
  bash: exec: /etc: cannot execute: Is a directory

With Bash 5.3.0:
  $ exec /etc
  bash: /etc: Is a directory

The 'assert' function cannot directly handle compound commands.  So,
those need to be wrapped in 'bash -c "..."' [2].

[1] Bash commit b8c60bc9ca365f82
    See how exec_builtin() handles EX_NOEXEC and EISDIR from
    shell_execve() to avoid printing a duplicate error message.
    https://cgit.git.savannah.gnu.org/cgit/bash.git/commit/?id=b8c60bc9ca365f82

[2] https://github.com/bats-core/bats-assert

containers#1688
containers#1699
debarshiray pushed a commit to debarshiray/toolbox that referenced this pull request Jan 26, 2026
Bash 5.3.0 changed the error messages shown by its exec built-in [1].

With Bash 5.2.37:
  $ exec /etc
  bash: /etc: Is a directory
  bash: exec: /etc: cannot execute: Is a directory

With Bash 5.3.0:
  $ exec /etc
  bash: /etc: Is a directory

The 'assert' function cannot directly handle compound commands.  So,
those need to be wrapped in 'bash -c "..."' [2].

[1] Bash commit b8c60bc9ca365f82
    See how exec_builtin() handles EX_NOEXEC and EISDIR from
    shell_execve() to avoid printing a duplicate error message.
    https://cgit.git.savannah.gnu.org/cgit/bash.git/commit/?id=b8c60bc9ca365f82

[2] https://github.com/bats-core/bats-assert

containers#1688
containers#1699
containers#1739
(backported from commit 6c98db6)
debarshiray pushed a commit to debarshiray/toolbox that referenced this pull request Jan 27, 2026
Bash 5.3.0 changed the error messages shown by its exec built-in [1].

With Bash 5.2.37:
  $ exec /etc
  bash: /etc: Is a directory
  bash: exec: /etc: cannot execute: Is a directory

With Bash 5.3.0:
  $ exec /etc
  bash: /etc: Is a directory

The 'assert' function cannot directly handle compound commands.  So,
those need to be wrapped in 'bash -c "..."' [2].

[1] Bash commit b8c60bc9ca365f82
    See how exec_builtin() handles EX_NOEXEC and EISDIR from
    shell_execve() to avoid printing a duplicate error message.
    https://cgit.git.savannah.gnu.org/cgit/bash.git/commit/?id=b8c60bc9ca365f82

[2] https://github.com/bats-core/bats-assert

containers#1688
containers#1699
containers#1739
containers#1743
(backported from commit 6c98db6)
(cherry picked from commit 0090893)
debarshiray pushed a commit to debarshiray/toolbox that referenced this pull request Jan 27, 2026
Bash 5.3.0 changed the error messages shown by its exec built-in [1].

With Bash 5.2.37:
  $ exec /etc
  bash: /etc: Is a directory
  bash: exec: /etc: cannot execute: Is a directory

With Bash 5.3.0:
  $ exec /etc
  bash: /etc: Is a directory

The 'assert' function cannot directly handle compound commands.  So,
those need to be wrapped in 'bash -c "..."' [2].

[1] Bash commit b8c60bc9ca365f82
    See how exec_builtin() handles EX_NOEXEC and EISDIR from
    shell_execve() to avoid printing a duplicate error message.
    https://cgit.git.savannah.gnu.org/cgit/bash.git/commit/?id=b8c60bc9ca365f82

[2] https://github.com/bats-core/bats-assert

containers#1688
containers#1699
containers#1739
containers#1744
(backported from commit 6c98db6)
(cherry picked from commit 0090893)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants