Capacity and utilization by just-be-dev · Pull Request #4696 · oxidecomputer/omicron

just-be-dev · 2023-12-14T19:19:28Z

This PR is a follow up to #4605 which adds views into capacity and utilization both at the silo and system level.

API:

op	method	url
silo_utilization_list	GET	/v1/system/utilization/silos
silo_utilization_view	GET	/v1/system/utilization/silos/{silo}
utilization_view	GET	/v1/utilization

I'm not entirely satisfied w/ the silo utilization endpoints. They could be this instead:

op	method	url
silo_utilization_list	GET	/v1/system/silos-utilization
silo_utilization_view	GET	/v1/system/silos/{silo}/utilization

Also take special note of the views

// For the eyes of end users
/// View of the current silo's resource utilization and capacity
#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
pub struct Utilization {
    /// Accounts for resources allocated to running instances or storage allocated via disks or snapshots
    /// Note that CPU and memory resources associated with a stopped instances are not counted here
    /// whereas associated disks will still be counted
    pub provisioned: VirtualResourceCounts,
    /// The total amount of resources that can be provisioned in this silo
    /// Actions that would exceed this limit will fail
    pub capacity: VirtualResourceCounts,
}

// For the eyes of an operator
/// View of a silo's resource utilization and capacity
#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
pub struct SiloUtilization {
    pub silo_id: Uuid,
    pub silo_name: Name,
    /// Accounts for resources allocated by in silos like CPU or memory for running instances and storage for disks and snapshots
    /// Note that CPU and memory resources associated with a stopped instances are not counted here
    pub provisioned: VirtualResourceCounts,
    /// Accounts for the total amount of resources reserved for silos via their quotas
    pub allocated: VirtualResourceCounts,
}

For users in the silo I use provisioned and capacity as the language. Their capacity is represented by the quota set by an operator. For the operator provisioned is the same but allocated is used to denote the amount of resources allotted via quotas.

Note: I had planned to add a full system utilization endpoint to this PR but that would increase the scope. Instead will ship that API as a part of the next release. We can calculate some version of the full system utilization on the client by listing all the silos and their utilization.

In RFD-427 we specify that for the initial implementation we want to require that all silos have a quota. In considering the API and implementaiton more carefully I decided that at least temporarily we should drop create/delete given that deletions should never happen and creations should only happen when the silo is created (or via some internal process which creates quotas for pre-existing silos)

There will be a follow up PR to add capacity/utilization to the API both at the silo and system levels. Given that, and the general lack of actionability of quotas to silo users, I've just dropped the quota view from the API.

…nt objectidentity

david-crespo · 2023-12-14T19:37:57Z

+        &self.identity().name
+    }
+}
+


Yeah, the constraint of implementing ObjectIdentity felt a little unnecessary. We have several of these not-really-resource resources and we should still be able to treat as if they were (even when they don't have a resource/asset identity)

karencfv

Almost there! Thanks for pushing through with this :)

karencfv · 2023-12-14T20:15:27Z

        }
      }
    },
+    "/v1/utilization": {


Should this not be /v1/system/utilization? It also has the tag "silos" like the other endpoints. Conversely, if we want to keep this one as /v1/utilization, perhaps we could change the other endpoints to be /v1/system/utilization. Otherwise it feels like one set of endpoints is very explicit and the other is not. WDYT?

This is the silo admin's view of the utilization inside the silo. So this is a developers way of seeing how many resources their silo can provision. I dropped /v1/system/utilization (which summarizes the entirety of the system utilization) from this PR as it was more complicated to deliver.

In my mind /v1/system/... means operator whereas /v1/... is inner silo / developer stuff

I dropped /v1/system/utilization (which summarizes the entirety of the system utilization) from this PR as it was more complicated to deliver.

Ah! I think this was the piece of information I was missing. If we want to have this endpoint in the future then the current implementation makes sense, thanks!

david-crespo · 2023-12-14T20:46:10Z

+    q.storage_bytes AS storage_allocated
+FROM
+    omicron.public.virtual_provisioning_collection AS c
+    RIGHT JOIN omicron.public.silo_quotas AS q 


Is this a right join because a silo which has never had a provisioning even would have no entry in virtual_provisioning_collection?

A provisioning collection can be the fleet, a silo, or a project. I want all silo_quotas and whatever entries in the virtual_provisioning_collection as match.

david-crespo

Very clean! Would like to see integration tests on the new endpoints. I figure you have those on the way.

david-crespo · 2023-12-14T21:08:57Z

+#[endpoint {
+    method = GET,
+    path = "/v1/system/utilization/silos/{silo}",
+    tags = ["silos"],


I think these should be system/silos instead of silos

david-crespo · 2023-12-14T22:07:39Z

+    /// Accounts for resources allocated to running instances or storage allocated via disks or snapshots
+    /// Note that CPU and memory resources associated with a stopped instances are not counted here
+    /// whereas associated disks will still be counted
+    pub provisioned: VirtualResourceCounts,


Are these supposed to have #[serde(flatten)] on them too? I think this is why VirtualResourceCounts is showing up in the spec. On the other hand it's kind of cool to not flatten it and share the structure, but it's possible the other language clients might not like it as much.

Whoops, I see now why these are not. They're supposed to be under these keys. In any case this is still probably why VirtualResourceCounts is making it into the spec.

just-be-dev and others added 30 commits November 20, 2023 23:05

Outline basic CRUD actions for quotas

b59092c

Start outlining quota database type

21ba99c

Simplify quota model to be flatter

0127dd1

Rough API shape of resource quotas

5c50ce5

Create quotas during silo creation

3346a33

Update comment for internal silo

501f5ac

Merge branch 'main' into add-resource-limits

1894eb3

Initial work

626a7f6

Drop inner silo quota view

9e51431

There will be a follow up PR to add capacity/utilization to the API both at the silo and system levels. Given that, and the general lack of actionability of quotas to silo users, I've just dropped the quota view from the API.

Convert storage/memory to bytecount instead of ints

caf371b

WIP quota check CTE

96877b2

Add quota checks for cpu, memory, storage

8025899

Join do_update and quota_check into a single CTE step

a4002ff

Try (and fail) to resolve CTE type issues

76c722f

Resolve CTE compliation issues... ???

f8e2afc

Convert quota detlas to sql types; fix borrow issues

3d4ebe2

Merge branch 'main' into add-resource-limits

d317629

Remove a todo

dcf974e

Wire up quota limit error handling

83c9872

Fix missing quota specifier

7e779a0

SELECT got you down? Try JOIN today!

9859e12

Whoops, removing test cruft

45eb0ef

Only perform quota checks if allocating resources

55ff0b2

Adjust some quotas in hopes of not failing all the tests

2ee6d36

Add WIP integration test setup for quotas

4c58f9e

Show that quota checks are failing when they shouldn't

3b014c8

Ensure default silo gets constructed w/ a quota

bd950bc

Flesh out basic quota tests

4b0d236

Remove unnecessary transaction

0ad4340

just-be-dev added 9 commits December 13, 2023 15:28

Add silo_utilization view

45a25ab

Merge branch 'main' into capacity-and-utilization

b219736

Print stderr for when the openapi test fails

35af7de

Ensure we can still paginate by name or ID even if it doesn't impleme…

c30a64d

…nt objectidentity

Add silo_name to SiloUtilization for NameOrID pagination

01850ac

Add datastore

264b43f

Update utilization pagination type; fix openapi errors

7bdc575

Fix quotas test

1cdd415

Gen API changes

c49e533

david-crespo reviewed Dec 14, 2023

View reviewed changes

Comment thread nexus/db-model/src/quota.rs Outdated

just-be-dev added 5 commits December 14, 2023 15:00

Push breaking endpoint test

7bd96b8

Drop potentially unused conversion???

3e1b5f6

As some extra docs on SimpleIdentity

17a206e

Merge branch 'main' into capacity-and-utilization

dd039e9

Add migration

377e370

karencfv reviewed Dec 14, 2023

View reviewed changes

Bump migration

9ccd452

david-crespo reviewed Dec 14, 2023

View reviewed changes

david-crespo approved these changes Dec 14, 2023

View reviewed changes

david-crespo reviewed Dec 14, 2023

View reviewed changes

Update utilization tags

aa4dcae

david-crespo reviewed Dec 14, 2023

View reviewed changes

Add e2e tests

81a7475

just-be-dev marked this pull request as ready for review December 14, 2023 22:36

morlandi7 linked an issue Dec 14, 2023 that may be closed by this pull request

Tracking Issue: Resource Allocation (Silo) #4554

Closed

Fix authz test

ce0366a

just-be-dev enabled auto-merge (squash) December 15, 2023 00:57

just-be-dev merged commit 5827188 into main Dec 15, 2023

just-be-dev deleted the capacity-and-utilization branch December 15, 2023 01:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capacity and utilization#4696

Capacity and utilization#4696
just-be-dev merged 73 commits into
mainfrom
capacity-and-utilization

just-be-dev commented Dec 14, 2023 •

edited

Loading

Uh oh!

david-crespo Dec 14, 2023

Uh oh!

just-be-dev Dec 14, 2023

Uh oh!

Uh oh!

karencfv left a comment

Uh oh!

karencfv Dec 14, 2023

Uh oh!

just-be-dev Dec 14, 2023 •

edited

Loading

Uh oh!

karencfv Dec 14, 2023

Uh oh!

david-crespo Dec 14, 2023

Uh oh!

just-be-dev Dec 14, 2023

Uh oh!

david-crespo left a comment

Uh oh!

david-crespo Dec 14, 2023

Uh oh!

david-crespo Dec 14, 2023

Uh oh!

david-crespo Dec 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

just-be-dev commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karencfv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

just-be-dev Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-crespo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

just-be-dev commented Dec 14, 2023 •

edited

Loading

just-be-dev Dec 14, 2023 •

edited

Loading