Skip to content

LCOW: Autoselect platform#36418

Closed
lowenna wants to merge 1 commit intomoby:masterfrom
microsoft:jjh/fromplatformbuilder-autoselect
Closed

LCOW: Autoselect platform#36418
lowenna wants to merge 1 commit intomoby:masterfrom
microsoft:jjh/fromplatformbuilder-autoselect

Conversation

@lowenna
Copy link
Member

@lowenna lowenna commented Feb 27, 2018

Signed-off-by: John Howard jhoward@microsoft.com

Fixes #36560

Addresses #35089 (comment) and #35089 (comment).

With this PR, an LCOW daemon will choose to pull the Linux image (both for multi-manifest and single tag images) if

  • The daemon is obviously in LCOW mode
  • No API flag has been added to indicate an explicit platform
  • In the case of the builder, no explicit platform indicator in the FROM statement
  • No matching Windows image exists if multi-manifest

So for example:

PS E:\docker\build\nyan> docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
57310166fe88: Pull complete
Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db
Status: Downloaded newer image for busybox:latest
PS E:\docker\build\nyan> docker run --rm busybox uname -r; docker rmi busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
57310166fe88: Pull complete
Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db
Status: Downloaded newer image for busybox:latest
4.14.29-linuxkit
Untagged: busybox:latest
Untagged: busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db
Deleted: sha256:5b0d59026729b68570d99bc4f3f7c31a2e4f2a5736435641565d93e7c25bd2c3
Deleted: sha256:4febd3792a1fb2153108b4fa50161c6ee5e3d16aa483a63215f936a113a88e9a
PS E:\docker\build\nyan>
PS E:\docker\build\nyan> docker create ubuntu sh
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
1be7f2b886e8: Pull complete
6fbc4a21b806: Pull complete
c71a6f8e1378: Pull complete
4be3072e5a37: Pull complete
06c6d2f59700: Pull complete
Digest: sha256:e27e9d7f7f28d67aa9e2d7540bdc2b33254b452ee8e60f388875e5b7d9b2b696
Status: Downloaded newer image for ubuntu:latest
285bc85171a6e97d82852fedc614a99b5913fe6df5d253722bd7f24b42d54d64
PS E:\docker\build\nyan> docker inspect -f "{{.Os}}" ubuntu
linux
PS E:\docker\build\nyan>
PS E:\docker\build\nyan> type dockerfile
FROM supertest2014/nyan
ADD Dockerfile /
PS E:\docker\build\nyan> docker build .
e472211e4879: Pull complete
bba4bc236bcb: Pull complete
9d52b79a9349: Pull complete
c1f720c0024d: Pull complete
Digest: sha256:757f8da8d28e0ac8b1ddda3e8082acf94216b5dc719f6df71d92fe87a782648f 557.1kB/67.49MB
Status: Downloaded newer image for supertest2014/nyan:latest
 ---> 3dd6111154fcload complete
Step 2/2 : ADD Dockerfile /lete
 ---> 9676ceb6201b
Successfully built 9676ceb6201b
PS E:\docker\build\nyan>

Note microsoft/dotnet has both Linux and multiple Windows images in it's manifest list, so it should pull the best matching Windows image, not the Linux one.

PS E:\docker\build\dotnet> docker pull microsoft/dotnet
Using default tag: latest
latest: Pulling from microsoft/dotnet
407ada6e90de: Pull complete
3cb15ab58a2c: Pull complete
7d9a060b42e0: Pull complete
ebf9532b71d2: Pull complete
b070f5627406: Pull complete
a8c0bb44ce47: Pull complete
255a996d7ecd: Pull complete
fb09697854d5: Pull complete
Digest: sha256:c5916d40aa6b210fa7e587bf4cb890615177e70ab4196d702e29addd3a29ffdc
Status: Downloaded newer image for microsoft/dotnet:latest
PS E:\docker\build\dotnet> docker inspect -f "{{.Os}}" microsoft/dotnet
windows
PS E:\docker\build\dotnet>

And the above through the builder

PS E:\docker\build\dotnet> type dockerfile
FROM microsoft/dotnet
ENV foo=bar
PS E:\docker\build\dotnet> docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM microsoft/dotnet
latest: Pulling from microsoft/dotnet
407ada6e90de: Pull complete
3cb15ab58a2c: Pull complete
7d9a060b42e0: Pull complete
ebf9532b71d2: Pull complete
b070f5627406: Pull complete
a8c0bb44ce47: Pull complete
255a996d7ecd: Pull complete
fb09697854d5: Pull complete
Digest: sha256:c5916d40aa6b210fa7e587bf4cb890615177e70ab4196d702e29addd3a29ffdc
Status: Downloaded newer image for microsoft/dotnet:latest
 ---> 0e25af8b4121
Step 2/2 : ENV foo=bar
 ---> Running in e54406702885
Removing intermediate container e54406702885
 ---> b3f1d2bcdc14
Successfully built b3f1d2bcdc14
PS E:\docker\build\dotnet> docker inspect -f "{{.Os}}" b3f1d
windows
PS E:\docker\build\dotnet>

And an explicit Linux request of microsoft/dotnet

PS E:\docker\build\dotnet> type dockerfile
FROM --platform=linux microsoft/dotnet
ENV foo=bar
PS E:\docker\build\dotnet> docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM --platform=linux microsoft/dotnet
latest: Pulling from microsoft/dotnet
3e731ddb7fc9: Pull complete
47cafa6a79d0: Pull complete
79fcf5a213c7: Pull complete
68e99216b7ad: Pull complete
0bb6c6c2d2ad: Pull complete
11ab17bd0b71: Pull complete
181624c7f5b3: Pull complete
Digest: sha256:c5916d40aa6b210fa7e587bf4cb890615177e70ab4196d702e29addd3a29ffdc
Status: Downloaded newer image for microsoft/dotnet:latest
 ---> 9969575612df
Step 2/2 : ENV foo=bar
 ---> Running in efbed1aa945f
Removing intermediate container efbed1aa945f
 ---> 17db0ce2d48d
Successfully built 17db0ce2d48d
PS E:\docker\build\dotnet> docker inspect -f "{{.Os}}" microsoft/dotnet
linux
PS E:\docker\build\dotnet>

And using scratch

PS E:\docker\build\scratch> type dockerfile
FROM scratch
ADD testfile.txt /
PS E:\docker\build\scratch> docker build .
Sending build context to Docker daemon    810kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD testfile.txt /
 ---> 6281659d3b21
Successfully built 6281659d3b21
PS E:\docker\build\scratch> docker inspect -f "{{.Os}}" 6281
linux
PS E:\docker\build\scratch>

@codecov
Copy link

codecov bot commented Mar 2, 2018

Codecov Report

❗ No coverage uploaded for pull request base (master@ed7b642). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master   #36418   +/-   ##
=========================================
  Coverage          ?   34.94%           
=========================================
  Files             ?      613           
  Lines             ?    45563           
  Branches          ?        0           
=========================================
  Hits              ?    15924           
  Misses            ?    27550           
  Partials          ?     2089

@lowenna lowenna changed the title *WORK-IN-PROGRESS - IGNORE* LCOW: Autoselect platform *WIP Pending 35089 and 36327* LCOW: Autoselect platform Mar 2, 2018
@lowenna lowenna self-assigned this Mar 8, 2018
@thaJeztah
Copy link
Member

ping @johnstep @salah-khan PTAL

@lowenna lowenna force-pushed the jjh/fromplatformbuilder-autoselect branch from c8ae8fb to 53bdc73 Compare March 8, 2018 22:49
@lowenna lowenna changed the title *WIP Pending 35089 and 36327* LCOW: Autoselect platform *WIP Pending 35089* LCOW: Autoselect platform Mar 8, 2018
@lowenna
Copy link
Member Author

lowenna commented Mar 8, 2018

Rebased.

Signed-off-by: John Howard <jhoward@microsoft.com>

Addresses moby#35089 (comment).
This change enables the daemon to automatically select an image under LCOW
that can be used if the API doesn't specify an explicit platform.

For example:

FROM supertest2014/nyan
ADD Dockerfile /

And docker build . will download the linux image (not a multi-manifest image)

And similarly docker pull ubuntu will match linux/amd64
@lowenna lowenna changed the title *WIP Pending 35089* LCOW: Autoselect platform LCOW: Autoselect platform Mar 27, 2018
@lowenna lowenna force-pushed the jjh/fromplatformbuilder-autoselect branch from 98fc141 to 6baf604 Compare March 27, 2018 17:56
@lowenna
Copy link
Member Author

lowenna commented Mar 27, 2018

@tonistiigi @dnephin @johnstep Thanks for reviewing #35089. Now that is merged, this is rebased and ready for review. It addresses your outstanding comments about platform auto-select if the daemon finds an image which it can use. See the description at the top of the PR for examples.

}
if os == "windows" {
if foundWindowsMatch {
sort.Stable(manifestsByVersion{osVersion, matches})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe note that this is how a Windows image is selected over Linux image, for an unspecified OS with LCOW enabled. This is probably fine since the Windows build number must match anyway.

func filterManifests(manifests []manifestlist.ManifestDescriptor, requestedOS string) []manifestlist.ManifestDescriptor {
version := system.GetOSVersion()
osVersion := fmt.Sprintf("%d.%d.%d", version.MajorVersion, version.MinorVersion, version.Build)
logrus.Debugf("will prefer Windows entries with version %s", osVersion)
Copy link
Member

@johnstep johnstep Mar 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the above 3 lines should only be called if requestedOS is not linux? Calling system.GetOSVersion() is not useful in that case, and it seems weird to output the message. However, I guess it will be output anyway, when the OS is not specified, and the manifest contains only Linux images. That could be addressed by only getting the version once the first Windows image is found.

Copy link
Member Author

@lowenna lowenna Mar 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or simply put "will prefer any found Windows entries with version ….". This is only a debug statement ;) To note, this is in a _windows.go file, so it won't be called when running on Linux.

@lowenna
Copy link
Member Author

lowenna commented Apr 25, 2018

Ping @tonistiigi again. PTAL. This was an issue you (and to some extent) @dnephin were both insisting was done post #35089, yet it's been sitting un-reviewed for a month now. I assume you still want this functionality? Thanks.

@lowenna
Copy link
Member Author

lowenna commented May 19, 2018

ping @tonistiigi yet again. PTAL. @johnstep - any further comments?

@thaJeztah thaJeztah requested a review from tianon May 24, 2018 18:17
@tianon
Copy link
Member

tianon commented May 24, 2018

I was asked to provide my thoughts on whether the overall approach here makes sense, and this:

With this PR, an LCOW daemon will choose to pull the Linux image (both for multi-manifest and single tag images) if

  • The daemon is obviously in LCOW mode
  • No API flag has been added to indicate an explicit platform
  • In the case of the builder, no explicit platform indicator in the FROM statement
  • No matching Windows image exists if multi-manifest

Seems like sane logic IMO -- if the user hasn't specified otherwise, and we're Windows and the image we want to run is Windows, run Windows, else if we can Linux, let's try Linux. 👍

@lowenna
Copy link
Member Author

lowenna commented May 31, 2018

Ping again....

@ghost
Copy link

ghost commented May 31, 2018

Currently, the user experience horks with this bizarre message:

C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: open \.\pipe\docker_engine_windows
: The system cannot find the file specified.
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.

#36773 (comment). It would make things much better. However, if we get the real helpful advice in output:

Please right-click docker icon in system tray and click 'Switch to Linux container...' in context menu

that would equally help. Just imaging the world where we will have BSD based dockers, so running Dragonfly on Windows when mode is Linux should show me the error, instead of auto selecting stuff.

@lowenna
Copy link
Member Author

lowenna commented Jun 26, 2018

Ping @thaJeztah @carlfischer1, on the 4th month anniversary of the PR not being merged, how can it be moved forward?

Copy link
Member

@johnstep johnstep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tonistiigi
Copy link
Member

Opened #37350 that builds on this and combines with #37346 and updated parser pkg.

@thaJeztah We could still merge this separately, but we shouldn't release without the full platform support.

@thaJeztah
Copy link
Member

@tonistiigi is this one still needed now that #37350 was merged?

@lowenna
Copy link
Member Author

lowenna commented Jul 16, 2018

Ah, no. Closing

@lowenna lowenna closed this Jul 16, 2018
@thaJeztah
Copy link
Member

Thanks! I wasn't sure if there was anything left in this one 🤗

@thaJeztah thaJeztah added the area/lcow Issues and PR's related to the experimental LCOW feature label Oct 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/lcow Issues and PR's related to the experimental LCOW feature status/2-code-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants