Skip to content

DockerFetcher should return the original payload even if it is non-JSON#7564

Closed
kzys wants to merge 1 commit intocontainerd:mainfrom
kzys:fix-4672
Closed

DockerFetcher should return the original payload even if it is non-JSON#7564
kzys wants to merge 1 commit intocontainerd:mainfrom
kzys:fix-4672

Conversation

@kzys
Copy link
Copy Markdown
Member

@kzys kzys commented Oct 19, 2022

While DockerHub returns the errors array, the fetcher should support non-JSON errors to return as much information as possible.

Fixes #4672.

@k8s-ci-robot
Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kzys kzys force-pushed the fix-4672 branch 2 times, most recently from d45e67f to 13095a4 Compare October 19, 2022 17:32
@kzys kzys marked this pull request as ready for review October 19, 2022 18:10
@kzys
Copy link
Copy Markdown
Member Author

kzys commented Oct 19, 2022

/retest

var registryErr Errors

buf := &bytes.Buffer{}
_, err := io.Copy(buf, resp.Body)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use io.LimitReader ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. How about liming the body around 1024 bytes? It is pretty arbitrary though.

@samuelkarp
Copy link
Copy Markdown
Member

/retest

if len(s) > 0 {
s = ": " + s
}
return fmt.Errorf("unexpected status code %v: %v%s", req.String(), resp.Status, s)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if it could make sense to return the response as-is in the "not valid JSON" case, wdyt?

(agreed on the limit reader though, to prevent really bad responses)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without unexpected status code ...? I'd like to keep the URL (from req.String) and HTTP status code. The response body may have these, but it isn't guaranteed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Tricky one indeed. I guess either makes sense ("we couldn't parse; we don't know so here's what the registry told us" vs adding the extra information). I guess all bets are off in either case, so 🤷‍♂️.

@dmcgowan
Copy link
Copy Markdown
Member

We rejected a similar PR in the past due to the high likelihood that the body is HTML. Do we have more examples of the non-JSON return bodies and which registries they are coming from?

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Oct 20, 2022

No. We don't.

That being said, if we have a bad HTML-returning registry, it would be better to tell callers that the registry is returning HTML rather than just having its HTTP code.

@kzys kzys force-pushed the fix-4672 branch 2 times, most recently from 0ee3dad to a826b59 Compare October 20, 2022 19:15
@dmcgowan
Copy link
Copy Markdown
Member

Agreed about better informing the callers. At some point these sort of responses from an HTTP endpoint need to go through more detailed debugging. Throwing the entire output into an error string is kinda risky. I would be more inclined to have an option which could write these out to a file or just have a better way of collecting an HTTP trace. Today I debug these using mitmproxy, but that only helps if I can reproduce the issue, we need a way for users to capture the trace.

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Oct 20, 2022

Hmm, right. I was mostly thinking about the DDoS risk (because of @AkihiroSuda's comment), but there is a potential information disclosure risk.

Writing that to a local file could be abused and it doesn't seem like containerd's job. Tracing (like OpenTelemetry) is also not great due to sampling...

@thaJeztah
Copy link
Copy Markdown
Member

Perhaps dumping first XX amount of data still wouldn't be "too bad". Seeing a <html><head><title>404 Not Found</title></head><body><h1>404 Not Found</h1> at least gives a clue "maybe I'm connecting with the wrong server". Not sure if there's a real attach vector there (at least not more than any output printed by a command)

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Oct 20, 2022

Sorry. I should have started from the end of the last discussion.

#4674 (comment)

this PR now has conflicts with main; given the last few comments, is there a path forward with @dmcgowan's suggestion of a typed error and/or logging the body content?

So returning a typed error or logging the body (with the context's logger) seems better than just using fmt.Errorf().

})
}

type HTTPError struct {
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't love to have this error under docker. Then I realized we have ErrUnexpectedStatus since #4523! So I'm going to update this PR tomorrow.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@kzys kzys force-pushed the fix-4672 branch 2 times, most recently from 9af83b2 to ba19663 Compare October 21, 2022 17:30
@kzys kzys force-pushed the fix-4672 branch 2 times, most recently from 8c14d0b to 5f6f3a0 Compare November 8, 2022 18:56
While DockerHub returns the errors array, the fetcher should support
non-JSON errors to return as much information as possible.

Fixes containerd#4672.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
@github-actions
Copy link
Copy Markdown

This PR is stale because it has been open 90 days with no activity. This PR will be closed in 7 days unless new comments are made or the stale label is removed.

@github-actions github-actions bot added the Stale label Mar 25, 2024
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2024

This PR was closed because it has been stalled for 7 days with no activity.

@github-actions github-actions bot closed this Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done
Archived in project

Development

Successfully merging this pull request may close these issues.

DockerFetcher swallows non-JSON errors returned by registries

6 participants