Description

Since the latest stable release update to 3602.2.0 for Microsoft Azure from our previous version 3510.2.8 for Microsoft Azure, the docker runtime has been updated to version 20.10.24 from version 20.10.23 prior.
With the current stable version, docker builds and runs seem to omit ownership information during filesystem extraction and all files are owned by root regardless of chown usage during building.

Impact

Building container images and running them on the current stable version of Flatcar leads most likely to permission denied errors on non-root containers attempting to write files as all files belong to root regardless of chown commands during building.

Environment and steps to reproduce

To reproduce the bug:

1. Set-up: Flatcar 3602.2.0 for Microsoft Azure with built-in docker runtime 20.10.24
2. Task: Run the following docker commands as a user in the docker group or as root
3. Action(s):
   a. docker run --name bug-test alpine ls -l /etc/shadow
   This command will show that file is owned by root:shadow (UID/GID 0/42)
   b. docker export bug-test | tar tv etc/shadow
   This command will show that it is owned by UID/GID 0/0
4. Error: The second command should show that the file is owned by UID/GID 0/42 but shows 0/0 (root/root)

Expected behavior

To show intended behaviour:

1. Set-up: Flatcar 3510.2.8 for Microsoft Azure with built-in docker runtime 20.10.23
2. Task: Run the following docker commands as a user in the docker group or as root
3. Action(s):
   a. docker run --name bug-test alpine ls -l /etc/shadow
   This command will show that file is owned by root:shadow (UID/GID 0/42)
   b. docker export bug-test | tar tv etc/shadow
   This command will show that it is owned by UID/GID 0/42
4. Result: Error is no longer here as file is correctly shown as being owned by UID/GID 0/42 (root/shadow)

Additional information

We have currently resolved this with a forced rollback to Flatcar 3510.2.8 for Microsoft Azure.

After more searching this seems to be the exact same bug that Ubuntu release had back in August as referenced here : https://bugs.launchpad.net/ubuntu/+source/docker.io-app/+bug/2029523/

Tar archive created by docker export command is missing ownership information (all files are owned by root). If this archive is then used to recreate filesystem for unprivileged processes (like via docker import or just unpacking it and chrooting), they can fail with permission denied error or in some other way.

This bug happens when package is built with Go older than 1.19: this version of Go introduced build tag unix that is used by upstream to determine if it should add UNIX-specific attributes into archive. Older Go versions silently ignore this source code, and the result is missing UIDs and GIDs in tar archives. As Go 1.20 was backported to affected releases, patches attached use this version to fix the bug.

The issue is caused by this change moby/moby@721358e#diff-12919f88ca9c04e478a6ffdf37e9a67ccdd2997afdc2e51adb1e67c53dcdbd8cL5-R5 and by packaging using Go 1.18. It can be fixed by appending this tag to DOCKER_BUILDTAGS, but building package with newer Go version is even better.