I don't think I understand enough about the future ramifications of not doing this. Could you clarify those for my benefit? Presumably, if we do some sort of Ursula level versioning, we may end up with a fragmented network that may also have implications with respect to sampling of Ursulas i.e. Alice's choice of Ursulas?

Let's start by understanding that new (de)serialization formats for the aforementioned entities will no longer work with previous versions of the code base. This creates several surfaces of backwards-incompatibility, for example:

1. Existing treasure maps generated by previous versions will no longer be usable by Alice or Bob.
2. Existing MessageKits will no longer be decryptable by Alice, Bob, or Ursula running an older version.
3. WorkOrders (now ReencryptionRequest)` generated by older versions will no longer be usable by Ursulas.
4. Offline revocations will no longer be deserializable by Alice or Ursula.

Provide backwards compatibility for unversioned entities?

Currently I expect the failure mode for using outdated bytes to be caused by a missing brand or invalid version bytes . We can potentially improve this behavior by detecting a missing header or invalid version bytes. One issue with this approach is that there is no way to determine if a given bytes blob is outdated or simply invalid since the start bytes will be random. We can potentially even attempt to maintain backwards-compatibility for old policy entities but think this is a bad idea right now since we can still make breaking changes without too much impact.

Ursula subclasses Versioned too? This will cause an additional major backwards incompatibility in node discovery.

To consider Ursula... if we implement the same versioning scheme we'll definitely need to provide backwards compatibility to prevent segmenting the network, sampling, etc. This will require some additional work in this PR if we decide that it belongs in this changeset.

An additional consideration that I'd prefer to address in a follow-up is supporting a range of backwards-compatible versions instead of just one. This approach might be implemented with semantic versioning but consider the trade off between complexity and fine-tuning.

Some general notes:

• the list in the description is missing RetrievalKit and AuthorizedKeyFrag, which are also versioned
• we may want to make Arrangement versioned as well
• Ursula info (see Ursula.payload_splitter()) is currently versioned independently - do we bring it to this scheme too?
• Another possible protocol object is the stuff returned from server.py::all_known_nodes() (currently not even an object)
• I am worried about possible transition to nucypher-core where we may want to use serde-specific versioning packages which will be hard to connect to this scheme

the list in the description is missing RetrievalKit and AuthorizedKeyFrag, which are also versioned
✔️

we may want to make Arrangement versioned as well
✔️

Ursula info (see Ursula.payload_splitter()) is currently versioned independently - do we bring it to this scheme too? Another possible protocol object is the stuff returned from server.py::all_known_nodes() (currently not even an object)

It certainly will not hurt. Shall I go ahead and also version Ursula and make a class for UrsulaInfo in this PR? I'm thinking "yes".

I am worried about possible transition to nucypher-core where we may want to use serde-specific versioning packages which will be hard to connect to this scheme.

It may be good to have a scheme that is compatible across different language implementations. Will serde add additional constraints in this regard, or can it be easily made compatible in python? I'm eager to get something delivered for versioning in the next release (hopefully as soon as this PR is merged) since many entities are already completely backwards incompatible.

This system is elegant and is working right now. Plus, it's a very simple scheme, why will it be hard to implement in nucypher-core?

It certainly will not hurt. Shall I go ahead and also version Ursula and make a class for UrsulaInfo in this PR? I'm thinking "yes".

I'm looking into it myself right now, to separate it into nucypher-core... and it's a massive can of worms. But perhaps if you just want to replace the existing versioning scheme, it will be easier to do.

It may be good to have a scheme that is compatible across different language implementations. Will serde add additional constraints in this regard, or can it be easily made compatible in python?

Well, that's why we wanted to have a core in Rust and bindings to other languages :) But really, it will depend on what we choose as the "reference" binary implementation, and on the specific versioning solution. My guess is it can be made compatible, but perhaps not with the specific scheme in this PR.

I'm eager to get something delivered for versioning in the next release (hopefully as soon as this PR is merged) since many entities are already completely backwards incompatible.

Yes, I agree with that. But I'm worried that if we do want to proceed with nucypher-core, we will have to make another major release for it.

I think using this as a reference implementation and attempting to replicate it in nucypher-core is the way to go. Hopefully any tweaks will not be substantial.

Keeping this in Python and not doing it in nucypher-core means that we need to copy/rewrite it in nucypher-ts and then in every other future language/platform forever.

I think using this as a reference implementation and attempting to replicate it in nucypher-core is the way to go. Hopefully any tweaks will not be substantial.

We will have to write our own serializer for that and not use any machinery that serde provides.

We will have to write our own serializer for that and not use any machinery that serde provides.

Do you mean we won't be able to use the serde versioning libraries? Or can't use serde at all?

Also, while this PR is not merged yet, and seeing how there's already talk about supporting "ranges" of versions, I would like to revisit the suggestion to use "major.minor" versions instead of a single number.

@fjarri, I've incorporated this suggestion in ba6a49e. I've made the assumption that when the major version changes, backwards-compatibility with older bytes strings is broken but maintained for minor versions. In this sense the minor version number is just added metadata. Do you have a different sense of how this might operate?

Do you have a different sense of how this might operate?

Basically what you implemented, but slightly more relaxed: I think, if bytestring.major == class.major and bytestring.minor > class.minor, we can pass the bytestring to _from_bytes_current() and expect it to handle it. In most cases, yes, it will mean that later versions only add some fields but don't change existing ones, but I do not exclude the possibility that sometimes it will be possible to add some "future-proofing" (most likely, default values for fields).

I am not sure that ideologically it makes sense to keep deserializers for old major versions in the class as opposed to failing right away if bytestring.major != class.major; is it really still a different major version if we continue to support it?

...if bytestring.major == class.major and bytestring.minor > class.minor, we can pass the bytestring to _from_bytes_current() and expect it to handle it...

I am not sure that ideologically it makes sense to keep deserializers for old major versions in the class as opposed to failing right away if bytestring.major != class.major; is it really still a different major version if we continue to support it?

@fjarri, Considering the above two points. if we expect a minor version increase to be compatible with the current major version (_from_bytes_current) and we do not maintain previous major version handlers... what's the point of having _old_version_handlers at all?

Basically what you implemented, but slightly more relaxed: I think, if bytestring.major == class.major and bytestring.minor > class.minor, we can pass the bytestring to _from_bytes_current() and expect it to handle it.

✔️

I am not sure that ideologically it makes sense to keep deserializers for old major versions in the class as opposed to failing right away if bytestring.major != class.major

✔️

if we expect a minor version increase to be compatible with the current major version (_from_bytes_current) and we do not maintain previous major version handlers... what's the point of having _old_version_handlers at all?

Ok, fair enough, you're right. I think the guideline is that X.Y can be deserialized by X.Z as long as Y >= Z, and when you can't guarantee that, you need to bump the major version. But we need some limit on how long the old major versions are supported. Maybe to the point that they can be dropped at any time without further notice.

indicate deprecation of non-versioned object serialization?

I'm a bit hesitant to do so since there are still some unversioned (or differently versioned) entities.

I suppose it can say something like "Removes support for almost all unversioned bytestrings." I'm open to suggestions on how to word this accurately.