Toolbox command now requires single layer docker images #465
Description
Description
Moving from the rkt version of toolbox to the update here:
We've noticed that multi-layer images are not supported for custom toolboxes. I imagine this isn't intentional (they worked on the old version), so wanted to report it.
Impact
Minimal, we refactored our toolbox image to create a single layer final container. However, it does spit out a fairly cryptic error that took awhile to debug.
Environment and steps to reproduce
- Any Flatcar EC2 instance using 2905.2.0 Stable
- When a custom toolbox image is defined (see below), if that image contains more than a single layer, layers beyond the first will not be successfully unpacked.
TOOLBOX_DOCKER_IMAGE=<ecr>/<custom_image>
TOOLBOX_DOCKER_TAG=trunk
TOOLBOX_USER=<user>
- Running the
toolboxcommand in a shell - Error output below. The culprit appears to be the tar command used to unpack layers. Layers beyond the first have the full path stripped and the .tar files cannot be found:
sudo --preserve-env tar xvf /var/lib/toolbox/<image>/166f5ca608dd023369f6ed13256b1216823827d93aaea2e5c50caa4e4106f7e6/layer.tar 455fdf9537ae177715405ec8320ce0efb71cd8ee433733613805f725a4f163dc/layer.tar 241197fae1234fce4c43e482aaf71fcc7dbb7286fc8af1417dac8985fd7081f9/layer.tar 43ec1bba72d1a3f5ae33a28dbc7523d20804b484fc911d0aa5240881c2cd79cb/layer.tar 062a4e81e8a4ac5a41b6d1a343e526bb8cf1e10a9a750f46a1a5b5cf1b9e9d99/layer.tar 26b50f604577fedf9bce2d40cd7440568d001d3fa9ebf0bcb7f38e8a4ec7f535/layer.tar f4a65a53216db27f7ae31da74d0428c4e3ff6acb3569f8e9757a6d8b8c471272/layer.tar e6b50753df35b28315d0dbf92313216e57877c4d8e4fe74fb064231c28d6951c/layer.tar 80740d51e59e7c19edf8d6813dd21ee464996062cc2b4d281f2182d23184e23a/layer.tar a66a006f55aaff0befda99883c4d4f890e8ec60ccc8fef372e989b2a21a82d03/layer.tar e681f6f4c78e5ecf03cfadf22f52bbdeabe61dbf60caee0aa2e73644151669e5/layer.tar -C /var/lib/toolbox/<image>
tar: 455fdf9537ae177715405ec8320ce0efb71cd8ee433733613805f725a4f163dc/layer.tar: Not found in archive
tar: 241197fae1234fce4c43e482aaf71fcc7dbb7286fc8af1417dac8985fd7081f9/layer.tar: Not found in archive
tar: 43ec1bba72d1a3f5ae33a28dbc7523d20804b484fc911d0aa5240881c2cd79cb/layer.tar: Not found in archive
tar: 062a4e81e8a4ac5a41b6d1a343e526bb8cf1e10a9a750f46a1a5b5cf1b9e9d99/layer.tar: Not found in archive
tar: 26b50f604577fedf9bce2d40cd7440568d001d3fa9ebf0bcb7f38e8a4ec7f535/layer.tar: Not found in archive
tar: f4a65a53216db27f7ae31da74d0428c4e3ff6acb3569f8e9757a6d8b8c471272/layer.tar: Not found in archive
tar: e6b50753df35b28315d0dbf92313216e57877c4d8e4fe74fb064231c28d6951c/layer.tar: Not found in archive
tar: 80740d51e59e7c19edf8d6813dd21ee464996062cc2b4d281f2182d23184e23a/layer.tar: Not found in archive
tar: a66a006f55aaff0befda99883c4d4f890e8ec60ccc8fef372e989b2a21a82d03/layer.tar: Not found in archive
tar: e681f6f4c78e5ecf03cfadf22f52bbdeabe61dbf60caee0aa2e73644151669e5/layer.tar: Not found in archive
working image (Default Image):
$ ls /var/lib/toolbox/density-fedora-latest/
32ad7b7f7ecb4840ac2e7588e37462dadd877f428c0baa0ee3b72cb3d31606e9 dev lib64 mnt run tmp
bin etc lost+found opt sbin usr
boot home manifest.json proc srv var
dce66322d6472e195c624cd3bd0c6c13918c00d1022c653a95341491a1597f52.json lib media root sys
non-working image (Custom Image):
$ ls /var/lib/toolbox/<image>/
062a4e81e8a4ac5a41b6d1a343e526bb8cf1e10a9a750f46a1a5b5cf1b9e9d99 80740d51e59e7c19edf8d6813dd21ee464996062cc2b4d281f2182d23184e23a
166f5ca608dd023369f6ed13256b1216823827d93aaea2e5c50caa4e4106f7e6 a66a006f55aaff0befda99883c4d4f890e8ec60ccc8fef372e989b2a21a82d03
241197fae1234fce4c43e482aaf71fcc7dbb7286fc8af1417dac8985fd7081f9 e681f6f4c78e5ecf03cfadf22f52bbdeabe61dbf60caee0aa2e73644151669e5
26b50f604577fedf9bce2d40cd7440568d001d3fa9ebf0bcb7f38e8a4ec7f535 e6b50753df35b28315d0dbf92313216e57877c4d8e4fe74fb064231c28d6951c
2c61969bd70f07746081f62692f91cd04c1128889cf243bd2fb3fcdca70cbb25.json f4a65a53216db27f7ae31da74d0428c4e3ff6acb3569f8e9757a6d8b8c471272
43ec1bba72d1a3f5ae33a28dbc7523d20804b484fc911d0aa5240881c2cd79cb manifest.json
455fdf9537ae177715405ec8320ce0efb71cd8ee433733613805f725a4f163dc```
Expected behavior
Same image works on prior versions of Flatcar. Specifically, before this PR:
Additional information
I believe this line is the culprit:
https://github.com/kinvolk/toolbox/blob/fa17ec6310db9b1c2eb3fceab0334204b89c2c8c/toolbox#L58
It prepends $machinepath to the values pulled from the manifest.json, but when multiple layers are present, the $machinepath is only prepended to the first .tar file.