Skip to content

Builder: fix "COPY --from" to non-existing directory on Windows#39695

Merged
kolyshkin merged 2 commits intomoby:masterfrom
thaJeztah:fix_copy_on_windows
Aug 8, 2019
Merged

Builder: fix "COPY --from" to non-existing directory on Windows#39695
kolyshkin merged 2 commits intomoby:masterfrom
thaJeztah:fix_copy_on_windows

Conversation

@thaJeztah
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah commented Aug 8, 2019

fixes docker/for-win#4349
built on top of #39698

This fixes a regression introduced in 6d87f19 (#38599),
causing COPY --from to fail if the target directory does not exist:

FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world

Would produce an error:

Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.

The cause for this was that Go's os.MkdirAll() does not support/detect volume GUID paths
(\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}), and as a result
attempted to create the volume as a directory (\\?), causing it to fail.

This patch replaces os.MkdirAll() with our own system.MkdirAll() function, which
is capable of detecting GUID volumes.

- How to verify it

To verify (on Linux; don't know the equivalent on Windows, LOL):

make TEST_FILTER=TestBuildMultiStageCopy test-integration

...
INFO: Testing against a local daemon
=== RUN   TestBuildMultiStageCopy
=== RUN   TestBuildMultiStageCopy/copy_to_root
=== RUN   TestBuildMultiStageCopy/copy_to_newdir
=== RUN   TestBuildMultiStageCopy/copy_to_newdir_nested
=== RUN   TestBuildMultiStageCopy/copy_to_existingdir
=== RUN   TestBuildMultiStageCopy/copy_to_newsubdir
--- PASS: TestBuildMultiStageCopy (11.32s)
    --- PASS: TestBuildMultiStageCopy/copy_to_root (5.76s)
    --- PASS: TestBuildMultiStageCopy/copy_to_newdir (1.33s)
    --- PASS: TestBuildMultiStageCopy/copy_to_newdir_nested (1.41s)
    --- PASS: TestBuildMultiStageCopy/copy_to_existingdir (1.34s)
    --- PASS: TestBuildMultiStageCopy/copy_to_newsubdir (1.48s)
PASS
...

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

@thaJeztah
Copy link
Copy Markdown
Member Author

ping @ddebroy @StefanScherer PTAL

Copy link
Copy Markdown
Contributor

@StefanScherer StefanScherer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@thaJeztah
Copy link
Copy Markdown
Member Author

ah, dang; guess I need to flip a slash

11:12:59 === RUN   TestBuildMultiStageCopy
11:12:59 --- FAIL: TestBuildMultiStageCopy (0.00s)
11:12:59     build_test.go:148: assertion failed: error is not nil: open testdata/Dockerfile.TestBuildMultiStageCopy: The system cannot find the path specified.

@thaJeztah thaJeztah force-pushed the fix_copy_on_windows branch from 53b3e9c to 95a63be Compare August 8, 2019 09:23
@thaJeztah
Copy link
Copy Markdown
Member Author

Fixed (hopefully)

@StefanScherer
Copy link
Copy Markdown
Contributor

still looks good on Windows

@thaJeztah
Copy link
Copy Markdown
Member Author

Still doesn't work on Windows 😞

09:38:32 === RUN   TestBuildMultiStageCopy
09:38:32 --- FAIL: TestBuildMultiStageCopy (0.00s)
09:38:32     build_test.go:149: assertion failed: error is not nil: open testdata\Dockerfile.TestBuildMultiStageCopy: The system cannot find the path specified.

@thaJeztah thaJeztah force-pushed the fix_copy_on_windows branch from 95a63be to e7ae1b7 Compare August 8, 2019 10:04
@thaJeztah
Copy link
Copy Markdown
Member Author

Getting close

12:19:18 --- FAIL: TestBuildMultiStageCopy (0.00s)
12:19:18     build_test.go:151: assertion failed: error is not nil: open C:\gopath\src\github.com\docker\docker\testdata\Dockerfile.TestBuildMultiStageCopy: The system cannot find the path specified.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe inline it as in other tests?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didn't work; looks like on Windows the CWD is different, and it sees C:\gopath\src\github.com\docker\docker\ as current directory, not C:\gopath\src\github.com\docker\docker\integration\build\.

I can work around that, but looking why

CI is running the integration tests from here;

moby/hack/make.ps1

Lines 338 to 354 in 4fb5e9e

ForEach($dir in $integration_api_dirs) {
Set-Location $dir
Write-Host "Running $($PWD.Path)"
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "$($PWD.Path)\test.exe"
$pinfo.RedirectStandardError = $true
$pinfo.UseShellExecute = $false
$pinfo.Arguments = $env:INTEGRATION_TESTFLAGS
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start() | Out-Null
$p.WaitForExit()
$err = $p.StandardError.ReadToEnd()
if (($LASTEXITCODE -ne 0) -and ($err -notlike "*warning: no tests to run*")) {
Throw "Integration tests failed: $err"
}
}

Which looks quite similar to the Linux equivalent;

for dir in ${integration_api_dirs}; do
if ! (
cd "$dir"
echo "Running $PWD flags=${flags}"
test_env ./test.main ${flags}

I wonder if System.Diagnostics.ProcessStartInfo inherits the working directory, or if it needs to be set explicitly 🤔

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like that may be it: it has a ProcessStartInfo.WorkingDirectory property

From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8

When UseShellExecute is true, the fully qualified name of the directory that contains
the process to be started. When the UseShellExecute property is false, the working
directory for the process to be started. The default is an empty string ("").

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #39698 to fix that

I'll rebase this PR on top of that one to get the fix in this branch

This function changed to the correct working directory before starting the tests
(which is the same as on Linux), however the `ProcessStartInfo` process does
not inherit this working directory, which caused Windows tests to be running
with a different working directory as Linux (causing files used in tests to not
be found).

From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8

> When `UseShellExecute` is `true`, the fully qualified name of the directory that contains
> the process to be started. When the `UseShellExecute` property is `false`, the working
> directory for the process to be started. The default is an empty string (`""`).

This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working
directory before starting the process.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah force-pushed the fix_copy_on_windows branch from e7ae1b7 to b50f192 Compare August 8, 2019 11:22
@thaJeztah thaJeztah requested a review from tianon as a code owner August 8, 2019 11:22
@thaJeztah thaJeztah force-pushed the fix_copy_on_windows branch from b50f192 to 3ffe794 Compare August 8, 2019 11:24
@thaJeztah
Copy link
Copy Markdown
Member Author

hmf test is failing

13:42:07 === RUN   TestBuildMultiStageCopy
13:42:07 === RUN   TestBuildMultiStageCopy/copy_to_root
13:42:10 === RUN   TestBuildMultiStageCopy/copy_to_newdir
13:42:12 === RUN   TestBuildMultiStageCopy/copy_to_newdir_nested
13:42:14 === RUN   TestBuildMultiStageCopy/copy_to_existingdir
13:42:16 === RUN   TestBuildMultiStageCopy/copy_to_newsubdir
13:42:18 --- FAIL: TestBuildMultiStageCopy (11.00s)
13:42:18     --- FAIL: TestBuildMultiStageCopy/copy_to_root (2.86s)
13:42:18         build_test.go:183: assertion failed: error is not nil: Error: No such image: testbuildmultistagecopy/copy_to_root
13:42:18     --- FAIL: TestBuildMultiStageCopy/copy_to_newdir (2.16s)
13:42:18         build_test.go:183: assertion failed: error is not nil: Error: No such image: testbuildmultistagecopy/copy_to_newdir
13:42:18     --- FAIL: TestBuildMultiStageCopy/copy_to_newdir_nested (2.07s)
13:42:18         build_test.go:183: assertion failed: error is not nil: Error: No such image: testbuildmultistagecopy/copy_to_newdir_nested
13:42:18     --- FAIL: TestBuildMultiStageCopy/copy_to_existingdir (1.98s)
13:42:18         build_test.go:183: assertion failed: error is not nil: Error: No such image: testbuildmultistagecopy/copy_to_existingdir
13:42:18     --- FAIL: TestBuildMultiStageCopy/copy_to_newsubdir (1.92s)
13:42:18         build_test.go:183: assertion failed: error is not nil: Error: No such image: testbuildmultistagecopy/copy_to_newsubdir

@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented Aug 8, 2019

no output on those failures. perhaps it's also a race condition (build not yet completed?) never mind, I think I did something silly; trying again

@thaJeztah thaJeztah force-pushed the fix_copy_on_windows branch from 3298c55 to f38b340 Compare August 8, 2019 13:51
@thaJeztah
Copy link
Copy Markdown
Member Author

Whoop! Fixed the test; now works both on Windows and Linux

14:25:17 === RUN   TestBuildMultiStageCopy
14:25:17 === RUN   TestBuildMultiStageCopy/copy_to_root
14:25:28 === RUN   TestBuildMultiStageCopy/copy_to_newdir
14:25:29 === RUN   TestBuildMultiStageCopy/copy_to_newdir_nested
14:25:30 === RUN   TestBuildMultiStageCopy/copy_to_existingdir
14:25:33 === RUN   TestBuildMultiStageCopy/copy_to_newsubdir
14:25:35 --- PASS: TestBuildMultiStageCopy (18.27s)
14:25:35     --- PASS: TestBuildMultiStageCopy/copy_to_root (11.18s)
14:25:35     --- PASS: TestBuildMultiStageCopy/copy_to_newdir (0.95s)
14:25:35     --- PASS: TestBuildMultiStageCopy/copy_to_newdir_nested (1.71s)
14:25:35     --- PASS: TestBuildMultiStageCopy/copy_to_existingdir (2.04s)
14:25:35     --- PASS: TestBuildMultiStageCopy/copy_to_newsubdir (2.39s)

This fixes a regression introduced in 6d87f19,
causing `COPY --from` to fail if the target directory does not exist:

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```

Would produce an error:

```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```

The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.

This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Copy link
Copy Markdown
Contributor

@ddebroy ddebroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one clarification.

@thaJeztah
Copy link
Copy Markdown
Member Author

with one clarification.

what's the clarification? did yo forget to post that?

@thaJeztah
Copy link
Copy Markdown
Member Author

Looks like Docker Hub made a whoopsie

[2019-08-08T16:32:20.347Z] FAIL: docker_cli_pull_test.go:118: DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts
[2019-08-08T16:32:20.347Z] 
[2019-08-08T16:32:20.347Z] docker_cli_pull_test.go:145:
[2019-08-08T16:32:20.347Z]     pullFromV2("hello-world")
[2019-08-08T16:32:20.347Z] docker_hub_pull_suite_test.go:73:
[2019-08-08T16:32:20.347Z]     c.Assert(err, checker.IsNil, check.Commentf("%q failed with errors: %s, %v", strings.Join(arg, " "), out, err))
[2019-08-08T16:32:20.347Z] ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0009c8ce0), Stderr:[]uint8(nil)} ("exit status 1")
[2019-08-08T16:32:20.347Z] ... "hello-world" failed with errors: Using default tag: latest
[2019-08-08T16:32:20.347Z] latest: Pulling from library/hello-world
[2019-08-08T16:32:20.347Z] Get https://registry-1.docker.io/v2/library/hello-world/manifests/sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a: EOF
[2019-08-08T16:32:20.347Z] , exit status 1

@kolyshkin
Copy link
Copy Markdown
Contributor

#39698 is merged, can you please rebase?

Copy link
Copy Markdown
Contributor

@kolyshkin kolyshkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kolyshkin kolyshkin merged commit 1505305 into moby:master Aug 8, 2019
@thaJeztah thaJeztah deleted the fix_copy_on_windows branch August 8, 2019 17:27
@ddebroy
Copy link
Copy Markdown
Contributor

ddebroy commented Aug 8, 2019

oops about the note above. I meant to say for Windows MkdirAllAndChownNew and mkdirAs are just wrappers around mkdirall. So the if condition at https://github.com/docker/ee-engine/blob/master/builder/dockerfile/copy.go#L559 is probably not very relevant for Windows. But in the Linux paths, mkdirAs does a lot more work and therefore necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows multi-stage builds broken in 19.03.1

6 participants