Skip to content

Add microversion utilities#2791

Merged
EmilienM merged 1 commit intogophercloud:masterfrom
Nordix:lentzi90/microversion-utils
Nov 6, 2023
Merged

Add microversion utilities#2791
EmilienM merged 1 commit intogophercloud:masterfrom
Nordix:lentzi90/microversion-utils

Conversation

@lentzi90
Copy link
Copy Markdown
Contributor

@lentzi90 lentzi90 commented Oct 4, 2023

This adds GetSupportedMicroversions and MicroversionSupported utility functions for working with microversions.
The doc string for ChooseVersion is also updated to clarify how it works.

Prior to a PR being reviewed, there needs to be a Github issue that the PR
addresses. Replace the brackets and text below with that issue number.

Fixes #2791

Links to the line numbers/files in the OpenStack source code that support the
code in this PR:

TODO. I am not sure what is needed here

@github-actions github-actions bot added the semver:minor Backwards-compatible change label Oct 4, 2023
@coveralls
Copy link
Copy Markdown

coveralls commented Oct 4, 2023

Coverage Status

coverage: 77.939% (-0.02%) from 77.962% when pulling ad897c8 on Nordix:lentzi90/microversion-utils into 32dbd09 on gophercloud:master.

@lentzi90 lentzi90 force-pushed the lentzi90/microversion-utils branch from a1c043b to 1502c65 Compare October 4, 2023 12:31
@github-actions github-actions bot added semver:minor Backwards-compatible change and removed semver:minor Backwards-compatible change labels Oct 4, 2023
@dtantsur
Copy link
Copy Markdown
Contributor

dtantsur commented Oct 4, 2023

}

// GetSupportedMicroversions returns the minimum and maximum microversion that is supported by the ServiceClient Endpoint.
func GetSupportedMicroversions(client gophercloud.ServiceClient) (string, string, error) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Thinking aloud) maybe we could return a wrapper structure with something like

type SupportedMicroversions struct {
    MaxMajor int
    MaxMinor int
    MinMajor int
    MinMinor int
}

Then some of these functions become methods on it, like

func (vs SupportedMicroversions) Require(version string) error
func (vs SupportedMicroversions) IsSupported(version string) bool
func (vs SupportedMicroversions) Choose(versions []string) string

etc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: you want to pass the client as a pointer, not by value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would the idea be to hold on to this wrapper and pass it around? I'm worried it would become hard to use. If I need to check/require microversions in many different places in a code base it would probably be cumbersome. Alternatively, I would end up calling GetSupportedMicroversions all over the place.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only thing it will be more convenient to use methods on the wrapper rather than passing two strings around and parsing them each time. There is also a risk of people using the wrong order (what goes first, min or max)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point! I was still in my head thinking about how I did it in CAPO, where the min and max is stored in the ComputeClient and then these functions become methods on that. Maybe I can find some way to integrate this better with the ServiceClient 🤔

// ChooseVersion queries the base endpoint of an API to choose the most recent non-experimental alternative from a service's
// published versions.
// It returns the highest-Priority Version among the alternatives that are provided, as well as its corresponding endpoint.
// ChooseVersion queries the base endpoint of an API to choose the identity service version.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're sure this is limited to "identity service", i.e. Keystone?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has a reference to IdentityBase down the code, dunno why.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I got fooled by the doc string into believing this could be used for Nova. I was very confused about why it never could find a good version to use until @mdbooth pointed out that it is taking the ProviderClient as input, not the ServiceClient. It is indeed only checking the identity endpoint and picks a version for Keystone.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that gophercloud is working on a major release, maybe it's time to change that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that would be nice!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also be really good to have this in the v1 branch though, if possible. We have a bunch of potential near term use cases for it, one of which is described in kubernetes-sigs/cluster-api-provider-openstack#1715 (comment).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdbooth I thought you did not want this kind of negotiation of the version? I have reworked the CAPO PR for that reason to set an explicit version in all cases instead of using a function like this. In which cases do you think we should use this?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably got confused and thought I was commenting on the PR in general :)

You are correct, I don't want to do negotiation in CAPO: we either get a specific version or we don't execute the code. There may be cases where some code might run with one of a small number of specific microversions (e.g. GET server) but we should never, e.g. request microversion N and get microversion N+1.

That said, it might be useful to be able to specify a microversion limit for testing so we can test old microversions against a newer server. The microversion requested would still be at the point of use, though, and we would still either get it or not get it.

However, CAPO hopefully isn't the only consumer of this code. Negotiation might be useful elsewhere, e.g. a CLI utility.

@lentzi90 lentzi90 force-pushed the lentzi90/microversion-utils branch from 1502c65 to 01c9b43 Compare October 5, 2023 05:39
@github-actions github-actions bot added semver:minor Backwards-compatible change and removed semver:minor Backwards-compatible change labels Oct 5, 2023
@lentzi90 lentzi90 force-pushed the lentzi90/microversion-utils branch from 01c9b43 to 7da6734 Compare October 11, 2023 08:14
@github-actions github-actions bot added semver:minor Backwards-compatible change and removed semver:minor Backwards-compatible change labels Oct 11, 2023

return supportedMicroversions, nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd also love to see

func (ms SupportedMicroversions) Choose(options []string) string

@lentzi90 lentzi90 force-pushed the lentzi90/microversion-utils branch from 7da6734 to 1219bf6 Compare October 23, 2023 11:59
@github-actions github-actions bot added semver:minor Backwards-compatible change and removed semver:minor Backwards-compatible change labels Oct 23, 2023
@lentzi90
Copy link
Copy Markdown
Contributor Author

Is it possible to retrigger the failed test? It looks unrelated to me but I could be wrong.

go: creating new go.mod: module unit_tests
go: github.com/wadey/gocovmerge@master: github.com/wadey/gocovmerge@master: invalid version: Get "https://proxy.golang.org/github.com/wadey/gocovmerge/@v/master.info": EOF
Error: Process completed with exit code 1.

This adds GetSupportedMicroversions, MicroversionSupported,
RequireMicroversion and ParseMicroversion utility
functions for working with microversions.
The doc string for ChooseVersion is also updated to clarify how it
works.
@lentzi90 lentzi90 force-pushed the lentzi90/microversion-utils branch from 1219bf6 to ad897c8 Compare October 24, 2023 08:26
@github-actions github-actions bot added semver:minor Backwards-compatible change and removed semver:minor Backwards-compatible change labels Oct 24, 2023
Copy link
Copy Markdown
Contributor

@dtantsur dtantsur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works with Ironic in my testing, thank you!

@lentzi90
Copy link
Copy Markdown
Contributor Author

Thank you for testing!
I have also confirmed that the code works with CAPO in kubernetes-sigs/cluster-api-provider-openstack#1567 (although I had to backport it to the v1 branch first since that is what CAPO is using)

@lentzi90 lentzi90 marked this pull request as ready for review October 25, 2023 11:22
Copy link
Copy Markdown
Contributor

@EmilienM EmilienM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, I'll wait for others to approve or request changes.

@pierreprinetti
Copy link
Copy Markdown
Member

I want to look at this this afternoon; please go ahead and merge if I don’t

@pierreprinetti
Copy link
Copy Markdown
Member

I want to look at this this afternoon; please go ahead and merge if I don’t

I still believe I can get to that today 🤞

Comment on lines +117 to +122
type SupportedMicroversions struct {
MaxMajor int
MaxMinor int
MinMajor int
MinMinor int
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The internal properties of this struct (in particular: the fact that they're split into exactly major and minor) look to me like implementation details. Please consider un-exporting the properties (and expose getters and setters as necessary) to reduce the API surface of our compatibility promise.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the fact that they're split into exactly major and minor" is the hard fact that is embedded into the definition of microversions. I'm fine with hiding private details as long as I can still access them via accessors (not just the string representations - actual numbers).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

// GetSupportedMicroversions returns the minimum and maximum microversion that is supported by the ServiceClient Endpoint.
func GetSupportedMicroversions(client *gophercloud.ServiceClient) (SupportedMicroversions, error) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The returned type (SupportedMicroversions) contains a minimum and a maximum version. Are we sure that this is sufficient data to know what version is supported?

For example: consider the hypothetical situation in which service X on OSP Stegosaurus exposes microversions from 1.0 to 1.2 and from 2.0 to 2.3. A security issue is found that is so bad, it needs backport. And it also requires a slight change in the API. So after the fix, service X on OSP T-Rex publishes microversions 1.3 and 2.4. Knowing minimum 1.0 and maximum 2.4 is not a guarantee that 1.3 is available (unless you run more elaborate checks).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your example is too hypothetical :) First, it assumes having two major versions on one endpoint, which is not how things tend to be done. Second, it assumes two major versions with microversions which is a thing that is quite unlikely to happen because the very idea of microversions implies that you can do breaking changes without creating major versions. We do ponder Ironic v2 from time to time, but that's more of a wishful thinking (file it under "rewrite Ironic in Go").

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I see it, the assumption is to think that my example won’t happen. Which may or may not be a safe assumption.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a question of whether you want to try covering 100% of potential future cases. It's not a terrible idea if you do, but that also mean you will do much more work than you need to. With rust-openstack, I took the approach of "I'll think about it when it happens" (fwiw, openstacksdk does the same), which has its own pros and cons.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can see, this (min and max) is the only information we can get from a single endpoint so there is nothing more we can do.

Comment on lines +133 to +136
type response struct {
Version valueResp `json:"version"`
Versions []valueResp `json:"versions"`
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can assume that all services will either use version, versions, or no versions. What if we have service-specific logic instead? Have you considered having service-specific logic, maybe exposed in the ServiceClient through an interface?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory - no. In practice, everyone just copied what Nova did :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a data point, I've implemented a similar thing in rust-osauth long ago, and nobody complained so far: https://github.com/dtantsur/rust-osauth/blob/master/src/protocol.rs#L31-L39

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is Swift versioning its API?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it actually does. Of the core services, I think only Nova, Keystone and Cinder (and Ironic if you count it core) use microversioning.


// RequireMicroversion checks that the required microversion is supported and
// returns a ServiceClient with the microversion set.
func RequireMicroversion(client gophercloud.ServiceClient, required string) (gophercloud.ServiceClient, error) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this function signature. You give me a version, and I tell you if it's OK. This requires many less assumptions. Can we make it so that this is the only new exported symbol, and unexport all the rest?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't. GopherCloud is already bare bones enough to justify another high-level client built upon it.

Copy link
Copy Markdown
Member

@pierreprinetti pierreprinetti Oct 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be failing to see your intended use case. Do you want to share the one you have one in mind?

The use case I’m most inclined to think: a client has two, max three behaviours coded for a specific action, each corresponding to a specific API microversion. The client would ask for its preferred version, then if not available it would test the fallback behaviour, and so on. In this case the client knows what exact microversion it needs and asks Goohercloud whether it is available. Do you have a case where a client needs something different, like a range?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Largely, I want to know in advance what features I can and cannot use, so that I don't need to delay this decision until much later in the code. Our code is organized in the way that the controller layer does not know about the Ironic implementation details, and we'd rather keep it that way.

@pierreprinetti
Copy link
Copy Markdown
Member

Thank you for this addition. I think that the functionality is useful.

@pierreprinetti
Copy link
Copy Markdown
Member

A more general comment: the most Gophercloud-y solution for this problem would probably be to have a versions package in each relevant service, with a List function that makes the call to the service and returns a typed response. The caller would then adjust their ServiceClient microversion accordingly, with or without a helper function in Gophercloud.

I wonder if this "generic" solution that aims at encapsulating the calls and solving negotiation in one swipe, isn't going to feel like a trap going forward.

@pierreprinetti pierreprinetti dismissed their stale review October 27, 2023 19:27

I didn’t mean my review to be blocking; converting to comments

@pierreprinetti
Copy link
Copy Markdown
Member

I will not have the opportunity to come back to this PR in the next week; @EmilienM please go ahead and do as you see fit

@lentzi90
Copy link
Copy Markdown
Contributor Author

lentzi90 commented Nov 1, 2023

A more general comment: the most Gophercloud-y solution for this problem would probably be to have a versions package in each relevant service, with a List function that makes the call to the service and returns a typed response. The caller would then adjust their ServiceClient microversion accordingly, with or without a helper function in Gophercloud.

I wonder if this "generic" solution that aims at encapsulating the calls and solving negotiation in one swipe, isn't going to feel like a trap going forward.

Thank you for the comment! I appreciate the insights and would ask that the maintainers decide on what would be the preferred way forward here. I can see pros and cons witch both approaches.

I want to also highlight this function in the utils repo: https://github.com/gophercloud/utils/blob/80377eca5d56d1f07d0ebdd4c2a5ae000a66a90e/openstack/clientconfig/requests.go#L739
Would it make more sense to move the microversion utilities there? It also handles all services in one function so perhaps it would fit in better there.

@dtantsur
Copy link
Copy Markdown
Contributor

dtantsur commented Nov 6, 2023

the most Gophercloud-y solution for this problem would probably be to have a versions package in each relevant service, with a List function that makes the call to the service and returns a typed response

Could be, but currently Gophercloud is handling Microversions as a standard feature (which they are in a sense), e.g. they handling is hardcoded in the ServiceClient:

func (client *ServiceClient) setMicroversionHeader(opts *RequestOpts) {
. If we start treating Microversions as service features, we need to change that as well. Which is probably a lot of work and will make the change no longer backportable..

@dtantsur
Copy link
Copy Markdown
Contributor

dtantsur commented Nov 6, 2023

@lentzi90 @EmilienM @pierreprinetti so folks, what's the path forward here? I can leave with a reduced scope if that allows this change to move forward.

@EmilienM
Copy link
Copy Markdown
Contributor

EmilienM commented Nov 6, 2023

I feel like this is a great iteration moving forward.

From Pierre:

I wonder if this "generic" solution that aims at encapsulating the calls and solving negotiation in one swipe, isn't going to feel like a trap going forward.

I agree but I also think this would be doable in a future version. I want to unblock these folks for now.

@EmilienM
Copy link
Copy Markdown
Contributor

EmilienM commented Nov 6, 2023

I'll merge this for now, @pierreprinetti if you have strong concerns we can always revert if needed; also could we please track the proposal of "a versions package in each relevant service, with a List function that makes the call to the service and returns a typed response and the caller would then adjust their ServiceClient microversion accordingly, with or without a helper function in Gophercloud" (from Pierre).

@EmilienM EmilienM merged commit af1813e into gophercloud:master Nov 6, 2023
@lentzi90 lentzi90 deleted the lentzi90/microversion-utils branch November 6, 2023 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver:minor Backwards-compatible change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants