Skip to content

postImagesCreate does actually produce JSON sequence (rfc7464).#43157

Closed
ndeloof wants to merge 7 commits intomoby:masterfrom
ndeloof:json-seq
Closed

postImagesCreate does actually produce JSON sequence (rfc7464).#43157
ndeloof wants to merge 7 commits intomoby:masterfrom
ndeloof:json-seq

Conversation

@ndeloof
Copy link
Contributor

@ndeloof ndeloof commented Jan 16, 2022

- What I did
Changed Content-type produced by postImagesCreate to adopt RFC7464 as an explicit mime type application/json-seq. The produced event stream is not a valid JSON object, so application/json should not be used.

- How I did it
Updated produced mime type

- How to verify it
curl -v --unix-socket /var/run/docker.sock -X POST http:/v1.41/images/create?fromImage=busybox

- Description for the changelog

Make it more explicit images/create produces JSON event sequence by adopting RFC7464 mime type application/json-seq.

- A picture of a cute animal (not mandatory but encouraged)
image

@thaJeztah
Copy link
Member

Thanks! I seem to recall we made some changes in the content-type in the past that we had to revert as it broke some things. I'll have to dig into history for that though.

Perhaps it should perform content-negotiation, and depending on what the client advertises, use one or the other.

@ndeloof
Copy link
Contributor Author

ndeloof commented Jan 18, 2022

I also wondered about the potential impact changing the Content-type. Could probably only send this more relevant content type is client sends Accept header with explicit support for this mime type, wdyt?

@thaJeztah
Copy link
Member

Yes, I was thinking along those lines indeed. When doing so, I think there's some utility package that we already use somewhere to make sure we parse Accept headers correctly (as they can be hard to parse)

These responses were always a bit "hairy" because when we implemented them, it was not really an official RFC yet, so there was no official content-type (other than some conventions used by some).

@thaJeztah
Copy link
Member

Ah, yes, this was the one;

switch gddohttputil.NegotiateContentEncoding(r, []string{"gzip", "deflate"}) {

That same packages also has a NegotiateContentType() function. Not sure if it's the best package (but it's "somewhat" official), but at least good to stick to one implementation.

func NegotiateContentType(r *http.Request, offers []string, defaultOffer string) string {

@thaJeztah
Copy link
Member

(that code was part of the godoc.org code, and has now been archived; not a major concern, but perhaps we need to look if it's maintained elsewhere as well, but we can look at that later)

@ndeloof
Copy link
Contributor Author

ndeloof commented Feb 14, 2022

IIUC Kubernetes uses bitbucket.org/ww/goautoneg, so seems a reasonable dependency to be added to align with other projects in our ecosystem, wdyt?

@thaJeztah
Copy link
Member

Thanks! So ... it looks like the bitbucket.org/ww/goautoneg package went AWOL, and they switch to a fork of it in kubernetes/kubernetes@16fd72d (switching to https://github.com/munnerz/goautoneg).

What slightly concerns me is that it feels a bit like they "just picked a fork of the original one", which is in a personal repository (not an org) and the owner apparently just made a fork, but is not a real expert on the code itself munnerz/goautoneg#3 (review)

This seems good to me, although I'm not too well-versed in this code.

Not saying we shouldn't switch to that package, but based on the above, I think it'd be good to have a good look how the implementation differs from what' we're currently using (if it's considerably better) before deciding whether or not we should switch. We could continue to use the current implementation (for now), and look for a possible replacement in a follow-up.

@ndeloof
Copy link
Contributor Author

ndeloof commented Feb 14, 2022

oh ok, I didn't checked the status of this dependency, makes perfect sense to keep our current implementation then, until the other ones demonstrates to be somehow a better one (?). Thanks for digging into this

@ndeloof ndeloof force-pushed the json-seq branch 3 times, most recently from c1be211 to e777f4b Compare February 15, 2022 14:41
@thaJeztah
Copy link
Member

Thanks! We briefly went through this PR in the maintainers meeting, and we're good with this change in general.

I want to have another look, to look more into detail, but from an earlier glance (some of this could be in a follow-up);

  • the swagger now only documents the new content-type as "produces" (I think it's possible to document both content-types separately, but I need to double check how swagger expects "content-negotiation" to be mentioned)
  • there may be other endpoints that use the same "json-progress" output; we should probably make them all do the same (could be in a follow-up)
  • we should check if the client code needs to be updated (i.e., make sure the client sends Accepts header for both application/json and the new content-type, so that it can talk with both "old" and "new" engines, and that new engines actually return the new content-type)

@thaJeztah thaJeztah added this to the 21.xx milestone Feb 18, 2022
@ndeloof
Copy link
Contributor Author

ndeloof commented Feb 18, 2022

swagger doesn't offer an explicit way to declare content-negotiation, but allows to define multiple values for produced content that a consumer can use to define Accept header for preferred format.
I've identified /events as another producer for json sequence.
Would be easy to add more in a follow-up pull-request in cased we missed one.

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@tianon
Copy link
Member

tianon commented Feb 26, 2022

Doesn't application/json-seq / RFC 7464 technically suggest the use of a "Record Separator" byte between JSON documents in addition to line-feed? Aren't our APIs only using line-feed?

(Also, I'm not sure it matters technically, but the two swagger changes here appear to use different orderings in produces: that I think should probably be consistent.)

@ndeloof
Copy link
Contributor Author

ndeloof commented Mar 1, 2022

@tianon correct. I need to check how we can get streamformatter.progressOutput to not only append LF after progress messages but also prepend RS.

@thaJeztah
Copy link
Member

Ah! Forgot to respond; good catch, @tianon (that one time I didn't dive into the RFC 😬 😂).

Let me know if you need help, @ndeloof ! (although I'm not super familiar with the streamformatter either 😂)

@thaJeztah
Copy link
Member

Before I forget; now that this PR needs updating, can you also add a note to the API history, and mention the new content-types / response types for these endpoints? https://github.com/moby/moby/blob/master/docs/api/version-history.md

@ndeloof ndeloof force-pushed the json-seq branch 3 times, most recently from 7c11913 to 4a8ea79 Compare March 3, 2022 21:35
@ndeloof ndeloof force-pushed the json-seq branch 4 times, most recently from 12cd2f0 to 93b1731 Compare March 25, 2022 08:15
ndeloof added 7 commits March 29, 2022 09:33
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…t accordingly

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOL; had some comment pending it says

Comment on lines 7637 to +7855
- "application/octet-stream"
produces:
- "application/json"
- "application/json-seq"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does swagger allow specifying both content types here (like it does for consumes)?

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 11, 2025

Closing as obsolete

@ndeloof ndeloof closed this Sep 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants