Handle various auth credentials, remove dashboard and network packages#39
Handle various auth credentials, remove dashboard and network packages#39austin-denoble merged 23 commits intomainfrom
dashboard and network packages#39Conversation
…y in target command and elsewhere with admin API. implement logic for caching ManagedKeys per-project in Viper, and properly creating keys for projects as needed when instantiating the go SDK for control and data plane operations, implement TokenManager and rename the oauth2 package to auth in utils, refactor some of the cruft around building clients and credentials
…cy for manual randomization for api key suffixes
…, allowing us to pass orgId to keep the targeted org sticky
…ice accounts mutually exclusive and clear the other on configuration, update auth status and add a presenter for printing the information, add TargetCreds and TargetUser to state, add pc auth clear command, fix logical issues with multiple auth paths
…ub-command to auth command, add functionality for listing out managed keys with json and reveal options
…d out functionality for cleaning up API keys that have been stored locally for use with the CLI
… and deleting locally stored keys on a delete operation, adjust printed output for auth local-keys list
…ing ProjectEditor permission
|
Claude finished @austin-denoble's task —— View job Code Review Completed ✅I've conducted a comprehensive review of this authentication refactoring PR. Here's my detailed analysis:
Security & Architecture✅ Strong Points
|
|
Claude finished @austin-denoble's task —— View job Code Review Completed ✅I've conducted a comprehensive review of this authentication refactoring PR. Here's my detailed analysis:
Security & Architecture✅ Strong Points
|
|
|
||
| type MarshaledProperty[T any] struct { | ||
| DefaultValue *T | ||
| DefaultValue T |
There was a problem hiding this comment.
I swapped to a value rather than a pointer here because it was a bit easier to reason about downstream when working with the properties, and also since we're writing to and marshaling out of a file, it didn't seem like we needed to pass a mutable ref around.
| name string | ||
| forceEncryptionWithCMEK bool | ||
| maxPods int | ||
| target bool |
There was a problem hiding this comment.
New flag allowing a user to immediately target a newly created project.
|
Claude finished @austin-denoble's task —— View job Claude finished reviewing this PR ✅ Code Review Progress
Comprehensive Code ReviewThis is a well-architected authentication refactoring that successfully addresses the original problems (token refresh, dashboard API removal) while maintaining backward compatibility. Here's my detailed analysis: Security & Architecture✅ Strong Points
|
| if access_token.AccessToken == "" { | ||
| msg.FailMsg("You must be logged in to set a target context. Run %s to log in.", style.Code("pc login")) | ||
| exit.ErrorMsg("You must be logged in to set a target context") | ||
| accessToken, err := oauth.Token(cmd.Context()) |
There was a problem hiding this comment.
The bulk of the changes in this file are refactoring around the removal of the dashboard package, and using the Go SDK resources directly.
| return orgName | ||
| } | ||
|
|
||
| type ListModel struct { |
There was a problem hiding this comment.
I extracted all of this into a reusable utility in the prompt package down in the /utils/ folder. It was generic enough, and could be reused elsewhere.
| var oauth2TokenMu sync.RWMutex | ||
| var oAuth2Token = configuration.MarshaledProperty[oauth2.Token]{ |
There was a problem hiding this comment.
Adding a mutex and un-exporting the MarshaledProperty itself.
| } | ||
|
|
||
| var ApiKey = configuration.ConfigProperty[string]{ | ||
| var GlobalApiKey = configuration.ConfigProperty[string]{ |
There was a problem hiding this comment.
I renamed this to "GlobalApiKey" because it acts as a kind of override, and I wanted to distinguish it from locally managed API keys.
|
|
||
| // ManagedKey represents an API key that is being actively managed by the CLI | ||
| // Either the CLI created it to work with a project, or a user created it and stored it explicitly | ||
| type ManagedKey struct { |
There was a problem hiding this comment.
This is the shape of the object we store locally to track keys.
|
|
||
| func GetProjectAPIKeys() map[string]string { | ||
| keys := ProjectAPIKeys.Get() | ||
| func GetOAuth2Token() oauth2.Token { |
There was a problem hiding this comment.
Getter/setter wrappers for things which leverage the mutex.
|
|
||
| func SetProjectAPIKey(projectId string, apiKey string) { | ||
| keys := ProjectAPIKeys.Get() | ||
| func SetProjectManagedKey(managedKey ManagedKey) { |
There was a problem hiding this comment.
My original implementation of this was never being called from the core CLI code, and was also inadequate in terms of what we actually need to store to manage things.
|
|
||
| import "github.com/pinecone-io/cli/internal/pkg/utils/pcio" | ||
|
|
||
| type TargetOrganization struct { |
There was a problem hiding this comment.
I moved these in here because they're a part of TargetContext and it felt like it made sense to me. I also added TargetUser which I don't think is being leveraged in a real way, and maybe was unnecessary. (possible TODO)
| DashboardUrl string | ||
| IndexControlPlaneUrl string | ||
| DashboardUrl string | ||
| PineconeGCPURL string |
There was a problem hiding this comment.
Just a rename to reflect things a bit better.
| // Parse token claims to get orgId | ||
| accessToken := secrets.OAuth2Token.Get() | ||
| claims, err := pc_oauth2.ParseClaimsUnverified(&accessToken) | ||
| accessToken, err := oauth.Token(ctx) |
There was a problem hiding this comment.
The refactoring in this command is to support the new TokenManager plumbing, and remove reliance on dashboard.
|
|
||
| // Token is a convenience function that gets the current OAuth2 token for the CLI | ||
| // It is a wrapper around the TokenManager.Token method | ||
| func Token(ctx context.Context) (*oauth2.Token, error) { |
There was a problem hiding this comment.
This is where all the TokenManager logic lives. I tried my best at a somewhat robust singleton factory, because it felt like trampling token credentials would be a possibility somewhere along the line.
This code also handles all of the token refreshing, so essentially whenever anything calls in for a token, it's refreshed if it needs to be.
| "github.com/pinecone-io/cli/internal/pkg/utils/pcio" | ||
| ) | ||
|
|
||
| type ListModel struct { |
There was a problem hiding this comment.
Interactive list selection utilities.
| "github.com/pinecone-io/go-pinecone/v4/pinecone" | ||
| ) | ||
|
|
||
| const ( |
There was a problem hiding this comment.
There are a lot of changes going on in here. This package is how we instantiate the Go SDK, and there's a bunch of logic in here for resolving which credentials we're using for which client. There's an AdminClient for admin API operations which needs either a user token or service account credentials. There's also the regular Client which needs an API key or a token with a Project ID (for X-Project-Id).
The logic for creating a new CLI-managed API key if necessary is here, along with the logic for pulling the map of existing keys to check if we need to create one at all, etc.
|
Claude finished @austin-denoble's task —— View job Claude finished reviewing this PR ✅ Code Review Progress
Comprehensive Code ReviewThis is a well-architected authentication refactoring that successfully addresses the original problems (token refresh, dashboard API removal) while maintaining backward compatibility. Here's my detailed analysis: Security & Architecture✅ Strong Points
|
…edentials so we can display it, etc
|
Claude finished @austin-denoble's task —— View job Claude finished reviewing this PR ✅ Code Review Progress
Comprehensive Code ReviewThis is a well-architected authentication refactoring that successfully addresses the original problems (token refresh, dashboard API removal) while maintaining backward compatibility. Here's my detailed analysis: Security & Architecture✅ Strong Points
|
|
Claude finished @austin-denoble's task —— View job Claude finished reviewing this PR ✅ Code Review Progress
Comprehensive Code ReviewThis is a well-architected authentication refactoring that successfully addresses the original problems (token refresh, dashboard API removal) while maintaining backward compatibility. Here's my detailed analysis: Security & Architecture✅ Strong Points
|
Problem
There are two big issues around authentication this PR is trying to address.
First, we're not refreshing our oauth user tokens after initially retrieving one on login (
pc auth login). This means that after 30 minutes, the token is no longer valid and the user will have to log in again. We need to make sure that we're calling for a refreshed token if the local token is expired. There were changes merged to the auth0-config to support this for the Pinecone CLI client.Second, the CLI previously relied on the non-public dashboard APIs in order to work with organizations, projects, and API keys. These implementations were handled manually through the
networkanddashboardpackages. In a previous PR, the code for working with admin resources was mostly replaced when implementing commands forproject,organization, andapi-key. However, we are still using dashboard APIs for theloginandtargetflows.Additionally, because of how our systems currently work with authentication, the CLI needs additional business logic for working with the control and data plane APIs when the user has authenticated via a user token (
pc auth login) or a service account (pc auth configure --client-id --client-secret). Because there are limitations on where these tokens can be used across APIs surfaces, the CLI needs mechanisms for creating its own API keys, and allowing users to "store" keys for specific projects. This in turn means we need to expose ways for users to manage these locally stored keys themselves.If you're curious or have any feedback, I've captured notes and thoughts here while working on an approach that will work for the CLI with our current authentication structure.
Solution
While it's a large PR, there haven't been any breaking changes made to the existing CLI interface. A lot of the credentials management logic is mostly abstracted from the user, but there are a number of new flags for existing commands that allow better ease-of-use when working with resources. Additionally, there has been new sub-commands added under
pc authto allow listing and pruning "local keys". I'll try and break things into sections below for review and testing focus.Supporting oauth user token refresh
Previously, the CLI would get an oauth user token on
pc auth login, store it once usingsecrets.OAuth2Token.Set(), and that was it. Since any code paths that needed the token for building the SDK would just pull it directly out ofsecrets.OAuth2Token.Get(), there was never any token refresh happening when expiry was reached.It's also not great that we have a lot of logic directly getting and setting against our Viper storage (local config files). This will lead to race conditions if multiple calls are made getting/setting within the same process, or if there are concurrent processes accessing these things (IE: a user having multiple shells opens running things using the CLI). Some of this is more forward-thinking, but I tried to be aware of it as I worked on an implementation to handle refreshing tokens.
Ultimately, I decided on writing a TokenManager singleton (
sync.Once), that exposes a methodTokenfor anywhere that needs access to the user token. This effectively acts as a wrapper for the secrets package where I've added async.RWMutexalong with getters and setters for the underlying ViperMarshaledProperty. I've also made the property itself local so it's not directly exported from the secrets package. We should most likely do this for anything we're storing locally, but I've held off refactoring everything for now.TokenManager. Add a public Token() function to theoauthpackage which returns TokenManager.Token(). Implement logic for refreshing when needed. Implement custom HTTP logic for calling for a refresh inrefreshTokenWithOrg. We need to do this because we need to passorgIdin the body or the refreshed token will default to a different org, and the Gooauth2package doesn't inherently support this.oauth2package tooauthin utils.secrets.OAuth2Token.Get()directly withoauth.Token().Handling multiple types of "auth"
There are a lot of changes and general refactoring in this PR to smooth out the experience of having several different forms of authentication that the user could be working with. The current primary forms of authentication are:
pc auth login)pc auth configure --client-id --client-secret)pc config set-api-keyorpc auth configure --api-key)X-Project-Idheader)X-Project-Idheader)Because of the variation in levels of access currently in our auth systems, the CLI needs to allow users to create new keys and store them locally for later use. Additionally, the CLI needs to be able to create its own managed keys if the user does not explicitly create keys before working with the control or data plane APIs. Currently, the console does a form of this by generating a "default" API key when you first log in. The console has access to unstable APIs that allow them to more easily work with API keys generally, so we have to make some concessions in the CLI where we're relying on the public admin APIs.
pc auth configure --api-keyorpc config set-api-keywill override the target organization, project, and any internally managed keys. This applies to operations involving the control and data plane (index, collections, etc). NOTE: We most likely need better documentation and messaging around this behavior. Any suggestions would be great.pc auth clear --global-api-key --service-accountto allow users a way to remove any configurations they've provided.pc auth statusto print out a detailed summary of the current authentication credentials that are configured.pc auth configure --client-id --client-secretafterpc auth loginand vice versa, it will log you out or clear the previously set credentials. This will also reset the current Organization and Project target because service accounts are scoped to a single organization.pc auth configure --client-id --client-secret, the CLI will use the admin API to fetch organization and project information, allowing the user to accurately target their org and project.Storing "managed" keys locally is a part of handling auth, and is explained a bit more below.
Managed Keys (locally stored
map[project_id]:ManagedKey)Because of limitations described previously, the CLI needs to have some concept of maintaining API keys for projects locally that can be reused across actions. There's no way to fetch an API keys value after the initial create call through the public Admin API. There's also no way to determine a keys organization or project if we just have access to the raw value, such as when a global API key is configured. There are also a lot of possible edge cases around an API key being used and then deleted outside of the CLI context, or manually by a user through the CLI.
My approach to resolving some of these issues and presenting a streamlined experience within the CLI is to introduce the concept of
ManagedKeys. These are keys that the CLI is essentially managing in state, and they can either be created manually by a user through the CLI, or the CLI will create one to manage itself if the user has not created one explicitly.The map of project keys is stored locally in a new
MarshaledPropertyin thesecretsViper store. Checksecrets.goto see the implementation of that along with the struct definition forManagedKey. The logic for creating "cli-managed" keys is in thesdkpackage, and is tied to when we instantiate thePineconeGo SDK. If the CLI needs to create its own key for management, this will be seamless to the user if they call something that requires it, likepc index list.In order to manage keys that are stored locally, there are new flags and commands exposed in a few places.
pc api-key create --name "your-key-name" --store- the--storeflag is new, telling the CLI to store this new key for the project it was created in locally. If there was a previously cli-managed key stored, it will be cleaned up.pc auth local-keys list- this will list all the project keys the CLI is maintaining locally in state.pc auth local-keys prune --dry-run --skip-confirmation- this prune command allows cleaning up locally maintained keys. It has flags for filtering what gets deleted:--dry-runwill output planned changes without applying anything,--skip-confirmationwill skip the deletion confirmation,--idallows cleaning up only one project, and--originallows cleaning up user-created keys, cli-created keys, or all keys.pc api-key delete- this command will now check to see if you're deleting the currently stored key for the project, and remove the locally cached key if you do.I don't think this approach is perfect, but I think for right now it gives us a pretty good overall experience while exposing options for managing your current state. Definitely open to feedback on this approach, and I think things will be evolving more as we move forward.
Removing
dashboardandnetworkpackagesThese packages are no longer needed due to the above changes, and we can safely remove all dependencies on the console admin API.
If you read all this, thank you!
Type of Change
Test Plan
The simplest way to test these changes locally are going to be checking out this branch (
adenoble/implement-credentials-logic) locally, and running goreleaser to build the binary. Then just run off of that as you normally would.$ git checkout adenoble/implement-credentials-logic $ goreleaser build --single-target --snapshot --clean # run commands against the binary $ ./dist/pc_darwin_arm64/pc auth loginI would recommend testing using each authentication approach. Using
pc auth statusandpc target --showto double check which organization and project you're targeting will be really helpful for validating things. When doing things like swapping your targeted project, you'll want to follow up with a control plane call likepc index list.Here's an example of a testing flow I've used locally to get started: