Skip to content

Conversation

@vvoland
Copy link
Contributor

@vvoland vvoland commented Sep 7, 2023

The compressor is already closed a few lines below and there's no error
returns between so the defer is not needed.

Calling Close twice on a writerCloserWrapper is unsafe as it causes it
to put the same buffer to the pool multiple times.

- What I did
Fixed random stack overflow daemon panics after import is performed.

- How I did it
Removed double Close.

- How to verify it

$ make DOCKER_GRAPHDRIVER=overlayfs  TEST_INTEGRATION_USE_SNAPSHOTTER=1 TEST_FILTER='TestImport'    test-integration

Check the daemon doesn't panic with stack overflow.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

The compressor is already closed a few lines below and there's no error
returns between so the defer is not needed.

Calling Close twice on a writerCloserWrapper is unsafe as it causes it
to put the same buffer to the pool multiple times.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
@vvoland vvoland added status/2-code-review area/images Image Distribution kind/bugfix PR's that fix bugs containerd-integration Issues and PRs related to containerd integration process/cherry-pick/24.0 labels Sep 7, 2023
@vvoland vvoland added this to the 25.0.0 milestone Sep 7, 2023
if err != nil {
return "", "", errdefs.InvalidParameter(err)
}
defer compressor.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any situation where archive.CompressStream should be closed? (mostly thinking if it should be returning a io.Nopcloser in that case)

If it's specific to this case, perhaps we should replace this line with a comment to outline "why".

Also wondering if something like a io.TeeReader would be something to involve 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is closed later down

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! Should've expanded the diff.

So do they have to be closed in a specific order? Because I also see that pr and pw are potentially closed multiple times ("with" or "without" error);

	defer pr.Close()
	defer pw.Close()


	// ... do things

	go func() {
		// ...
		pr.CloseWithError(err)
		// ...
	}()

	// ... do things

 	compressor.Close()
	pw.CloseWithError(err)

Copy link
Contributor Author

@vvoland vvoland Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reader and writer returned from io.Pipe are explicitly stated by the godoc to be fine when closed multiple times.
Compressor must be closed first because it produces input for the other streams.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, would the reverse work? Keep this defer, and remove the other close further down? (would that be more readable?) The defers are executed in reverse order, so in this case,

defer pr.Close()
defer pw.Close()
defer compressor.Close()

Would be executed as;

pw.CloseWithError(err)
pr.CloseWithError(err)

# defers:
compressor.Close()
pw.Close()  // redundant, but ok
pr.Close()  // redundant, but ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the compressor must be closed before compressedDigest := <-writeChan and before pr and pw are closed.

Closing pr or pw before compressor closes the writer the compressor writes to, so it doesn't have the chance to write all data.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DOH! Looking at that channel twice, and still not considering it for the order of events in my comment. 🙈 yup, makes sense.

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/images Image Distribution containerd-integration Issues and PRs related to containerd integration kind/bugfix PR's that fix bugs status/2-code-review

Projects

Development

Successfully merging this pull request may close these issues.

3 participants