At the time of writing, #686 introduces the first usage of the "RSS", which makes requests to sled agent to create datasets/services.

Aside from the datasets necessary for initializing Nexus (that is, Nexus itself and CRDB), these service requests should be handled by Nexus "as much as possible" instead of the RSS.

Many operations can trigger a need to request these services partitions:

• Initializing a new sled
• Adjusting service location to maintain availability while turning down an old sled
• Adjusting capacity to serve request load
• Etc

All these conditions are ongoing, and best handled by Nexus, which maintains a "global" view of the rack and exists beyond initialization.

Fortunately, the APIs defined by the Sled Agent should (more-or-less) remain the same - this issue just addresses the matter of "who calls them".

• Datasets
  • Nexus should have RPW for provisioning CockroachDB
  • Nexus should have RPW for provisioning Clickhouse
  • Nexus should have RPW for provisioning Crucible
• Services
  • Nexus should have RPW for provisioning Nexus
  • Nexus should have RPW for provisioning Internal DNS
  • Nexus should have RPW for provisioning External DNS
  • Nexus should have RPW for provisioning Oximeter
  • Nexus should have RPW for provisioning Crucible Pantry
• Housekeeping
  • Nexus should pass request for "all services on a sled" as a single endpoint with a generation number #2977
    • This generation number should be stored in the on-M.2 ledger for services
  • Nexus should pass request for "all datasets on a sled" as a single endpoint with a generation number #2978
    • This generation number should be stored in the on-M.2 ledger for datasets