This topic was discussed in the latter half of https://drive.google.com/file/d/185bFxdvDo_1aA5-T5ywp9t3ZCJbc52B7/view , but I'll briefly summarize here:

• Currently, after sled agent boots zones, they require additional configuration before they can be made usable.
• This configuration includes: Creating IP addresses, setting up routes, setting SMF configuration options, etc.
• These configuration changes are made by invoking zlogin to access the zone.
• Across reboot, a portion of this information (namely the SMF properties) is saved into /var/oxide. This instructs the sled agent how to relaunch zones across reboots.

Instead, we should do the following:

• Make zones "self-assembling" as much as possible.
  • Before the zone is booted, the sled agent should inject a file into /var/svc/profile/site.xml to load an SMF profile with the run-time parameters
  • The zone should rely on services starting from the manifest-import service
  • These dynamic parameters can then be processed by a "method script" inside the zone
  • As a result, we should not need to call zlogin with a set of commands to run inside the zone

What are the advantages of doing this?

• Less likely to have race conditions with SMF. We have had issues in the past, like [sled-agent] Propolis server SMF service listen address is not always set correctly #1115 , related to asynchronous setting of SMF properties relative to launching the service. Additionally, we've dealt with race conditions between the sled agent and themanifest-import service. With this new mechanism, sled agent will set all configuration before starting the zone, so the ownership of the zone is more clear.
• Less configuration outside the zone to re-launch the zone. Management of zones across reboots is a fair bit easier, no longer requiring auxiliary information in /var/oxide to know how to re-launch the zone.
• Less calls to zlogin. This should be a minor efficiency boost.
• Sled agent should become simpler. Hopefully. Let's see!

Tracking pieces:

• Add support for dynamically generating an SMF profile (added within [sled agent] Create and use self-assembling zones for storage zones #1902)
• Create zone network setup service so individual zones don't do this themselves Zone network configuration service #4677
• Generate SMF profiles (and use method scripts) to launch the following services:
  • Storage
    • Cockroach: [sled agent] Create and use self-assembling zones for storage zones #1902
    • Clickhouse: [sled agent] Create and use self-assembling zones for storage zones #1902
    • Crucible: [smf] Allow Crucible Zones to be self-assembling crucible#498, [sled agent] Create and use self-assembling zones for storage zones #1902
  • General Services
    • Convert Internal DNS Zone to be self-assembling #2880
    • Convert External DNS zone to be self-assembling #2881
    • Convert Nexus zone to be self-assembling #2882
    • Convert Oximeter zone to be self-assembling #2883
    • Convert Switch Zone to be self-assembling #2884
    • Convert NTP to be self-assembling #2885
    • Convert Crucible Pantry to be self-assembling #2886
  • Convert Instance Zones to be self-assembling #2887
• Confirm that it's safe to reboot with transient VNICs #3493
  • This means a small amount of accounting may be necessary for them on reboot
• Make sure all code relating to non-self-assembling zones is removed [sled-agent] Remove non self-assembling zone code #6162