There might be a case for metadata to live at the version level of a schema. For example, when the schema was created, who created it, etc. I wonder if we need separate endpoints for the metadata vs fetching the raw schema itself.

I think those extra metadata items can be exposed through metadata annotations or through an OPTIONS call if they are not part of the schema itself.

How do we envision this working exactly? Does this just mean implementations might follow a scheme that isn't simply a monotonically increasing Integer? This wording feels like we're opening it up for implementations to diverge from the spec without being specific about how they should do that.

This means that the schema document itself can use semver or some other "embedded" versioning notion while the API goes strictly by order or when changes have been added.

Understood, but wouldn't semver (or other versioning schemes) imply a different data type?

I was wondering the same.

seems there should be a MUST in here about "no version == latest"

I think I'm on the same track as Ryan, I'm wondering if we need to allow for the author to decide the version string so they can choose a simple int or a semvar pattern.

The goal of the server-assigned integer is for versioning not to complicate the API model. With automatic numbers, you can make all updates a plain POST on the schema URI and you can enforce the compatibility rules you want using service-side logic as the update happens.

Introducing breaking changes should really require the pain of a wholly new schema with its own backcompat versioning sequence.

Semver 1.x, 2.x, 3.x is really better captured by

/schemagroups/myapp/schemas/foo.1/versions/{n}
/schemagroups/myapp/schemas/foo.2/versions/{n}
/schemagroups/myapp/schemas/foo.3/versions/{n}

than by

/schemagroups/myapp/schemas/foo/versions/1.1
/schemagroups/myapp/schemas/foo/versions/2.2
/schemagroups/myapp/schemas/foo/versions/3.0

Those things under foo are not the same if they don't describe structurally compatible data.

By doing this you're basically asking the impl to be a document management system. Would it be so bad if we left that up to the author of the schema files and they could pretty much pick any URL pattern they wanted (within their permissions/scope)? Meaning, if they PUT over an existing one then it updates it. If they PUT to a new URL them they're creating a new one. Then the impl doesn't need any versioning at all, no saving of history, etc.. that's left up to systems that are built for that kind of thing and the results are pushed here for sharing/viewing.

Given the ability and suggestion to semver the dataschema or schema id, complex versioning schemes within a schema seem less necessary. A server-side check to prevent footguns should be possible even without versions, and this API doesn't actually prevent replacing a specified version with an incompatible document at the same URL (delete the existing version and/or schema, then re-create a schema with the same name and re-insert schemas until the version lines up, but the content doesn't).

Just discovered this project. Evaluating whether we could use this instead of writing our own schema registry. We want to be able to use semver to represent versions for a schema, where foo v1.y can be used as-is instead of foo v1.x (for y >= x), but foo v1.x and foo v2 are incompatible representations of the same data (and then maybe there'll be a converter registered in our system, etc.).

It's not clear to me which direction the conversation is leaning, but it'd be really lovely if it was possible to control the version id assigned to a new version of a schema. Otherwise, I mean, we can always add a proxy in front that remaps foo/versions/x.y to foo.x/versions/y, I guess. But it would be simpler if we could just control the version id to be created as we want. This is a very localized configuration point, too.

One interesting thing I was thinking about with respect to semver support for versioning, is that compatibility checking could perhaps be customized with a different configuration based on a real understanding of semver. So if a client tries to update the content of a schema that is currently at version 2.7.3, uploading content to new version 2.7.4 then that is a patch version release and the compatibility checker can be very strict. But if the client does the same thing to new version 3.0.0 then the compatibility checker can e.g. allow breaking changes.

Just a thought. :)

@chrish42 @EricWittmann the model here allows for a "trivial" case where the version is just a server assigned counter similar to how a GitHub commit identifier is a server assigned identifier for an entry in the sequential commit log. That does also not have deeper meaning. (The commit id is not a plain counter for different reasons). If you want to do something more sophisticated, you can always manage schema versions explicitly as "myschema:v1.1" and "myschema:v1.2" as separate schemas within the group and if you want to have the "latest" functionality, you also maintain a "myschema" that always returns the latest version. An implementation behind the protocol could easily provide that.

How do you see the query for all group getting interop w/o agreement on the first one?

"schemagroups" is really part of the path to the registry itself; everything defined here sits under that path. the segment could even be empty or could be multiple segments. I don't see there being interop issues, because you would always have it being at the root of the URI.

ah, I didn't realize you meant for "schemagroups" to be an impl choice - we might want to make clearer throughout then entire doc

Also update the OpenAPI to remove /schemagroups from paths. Would make it easier to generate servers/clients from the OpenAPI at some future point.

I was wondering the same.

Should DELETE actually be allowed? Did you consider a "decomission" option or "soft delete" alternatively? That'd e.g. prohibit to produce new events referencing such schema taken out of business, but existing events could still be decoded.

Yes, in general deletes are problematic and should IMO be either discouraged or prevented. Immutability has been a useful property in our registry for schema versions - we can deprecate and disable versions. And I wish we had done the same for the entire schema (deprecate/disable) rather than allow deletes.

This seems simple and pragmatic. Does it make sense to start with this approach?

What about constraining on a per-schema basis (giving each schema a "format" or "type"). The reason here is that I can imagine a logical group of schemas that aren't all the same technology. Especially if this were ever to expand beyond schemas and into e.g. API Designs as well - I would want to have a group that included perhaps multiple OpenAPI documents as well as some JSON Schemas... and my OpenAPI would likely have $refs to the JSON Schemas.

@EricWittmann I will add that as an XOR option, i.e. you can either define the formats at the group or schema level. If you define it at the group level, that is binding, meaning you can't override.

Would it be more familiar to specify this as a hostname / reg-name in section 3.2.2 ("Host"), rather than a path segment (which is slightly more permissive)?

I think segment-nz-nc better supports some common ways you might want to name your groups to indicate a hierarchy where none exists. Or an Organization + Project format - that sort of thing. I'm thinking things like how a lot of NPM packages are now being named...

How do recipients know what type of content to expect in the body? Is this based on the datacontenttype in the received message (plus some sort of lookup table to map datacontenttype to format)?

If so, making format the same value as datacontenttype would simplify the API.

@EricWittmann "SHOULD" leaves that up to what you think is right.

Do you want to include the ID that was served in this response, so that clients can retrieve the value again on subsequent calls?

Given the ability and suggestion to semver the dataschema or schema id, complex versioning schemes within a schema seem less necessary. A server-side check to prevent footguns should be possible even without versions, and this API doesn't actually prevent replacing a specified version with an incompatible document at the same URL (delete the existing version and/or schema, then re-create a schema with the same name and re-insert schemas until the version lines up, but the content doesn't).

maybe "CNCF Schema Registry API - wip" since it's not an 'rc' yet.

Just wanted to add that groups is a great concept that we don't (yet) have in Apicurio Registry, but may have been a mistake to not include. It's important I think to organize these things into groupings. I guess I'm just saying +1 to this concept from me. :)

+1 Good

Schema evolution is a pretty significant concept in a schema registry. I haven't seen how the spec facilitates configuring the compatibility (or validity) rules. This paragraph mentions a compatibility policy, but if that's mentioned anywhere else I missed it. :(

This might be an important enough feature to include in the spec. For Apicurio Registry we have the concept of "rules" that can be configured globally or per-artifact (in this spec I imagine rules could also be configured at the schemagroup level). Right now we have only two rules: Validity and Compatibility. But perhaps "rule" is a concept to consider?

@EricWittmann I believe this is an important feature of the registry, but not an important feature of the protocol. At the protocol level, a policy violation bubbles up as a plain conflict. If we were designing an implementation and the management API for that implementation, I would agree.

Fair enough. :)

It would be useful to keep some other meta-data for each artifact, particularly if anyone eve wants to create a Registry UI. Name, description, labels, creationTime, modifiedTime, etc.

Can validity be disabled or configured (e.g. syntax vs. semantic validity)? Treating Validity and Compatibility in similar ways has proven useful in Apicurio Registry.

Should be: if and only if format has not been set for the schema

Might want to add a MUST in here someplace....

When the URL to a schema is used without a version string, the implementation MUST return the latest version of that schema.

perhaps?

My OCD is kicking in... sometimes you have periods, sometime you don't on bulleted lists :-) can we choose one? I'd prefer no periods.

I am referring to the CE spec for the data types now. Do you think we need to copy them?

I was just referring to the lack of consistency on the bulleted lists - some ending in a period and some not. I don't have an opinion on copying vs referencing the data types

as above, I think we need a MUST here specifying the format/syntax - or define "Timestamp" in some kind of "data types" section

use of the word "set" here might confuse people since it's new. Did you mean "group" or "set of versions for one particular schema" ?

Can you elaborate on this last sentence? Do you mean they can add extensions that are unique identifier values? If so, why did you call this out? Just curious.

Any concrete schema document may also have a unique identifier for itself. There is either a path /group/xyz/schemas/abc/versions/1 or you could just address that exact doc with its ID. We want that to enable a URL shortener function.

How do you see this being used by a consumer of the registry?

aka.ms/s/{id} URL shortener option for greedy protocols.

Why isn't that just part of the URL shortener function's logic? E.g. tinyurl.com doesn't ask for an id - I just give it a full URL

Not sure if it matters, but the use of the word id got me wondering if it should be name instead. While the semantics would be the same either way, all examples of these IDs appear to be more like human friendly names rather than IDs (e.g GUIDs). So while it's not totally human friendly (eg no spaces, etc...), if we expect people to use meaningful "words" and not "random chars", then perhaps name would help guide them in that direction.

Just a thought