Skip to content

[nexus] Auto-update schema on boot#9030

Merged
smklein merged 3 commits into
mainfrom
auto-schema-update
Sep 24, 2025
Merged

[nexus] Auto-update schema on boot#9030
smklein merged 3 commits into
mainfrom
auto-schema-update

Conversation

@smklein

@smklein smklein commented Sep 16, 2025

Copy link
Copy Markdown
Collaborator

Fixes #8912

Should be merged after the rest of Nexus quiesce/handoff is complete.

@davepacheco davepacheco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update single-sled too?

In terms of impact to existing systems, as I understand it:

  • Systems that get updated via the MUPdate process from before this change to after: Nexus will immediately do the schema update on startup, rather than waiting for an operator to run the schema updater by hand. This should be fine because we've previously parked the rack so old Nexus instances shouldn't be running. But I'd like to check in with support about it.
  • Systems that get updated via the MUPdate process in the future: same as above.
  • For systems that get updated with self-service update: the handoff process + the new DataStore startup code ensures that we don't do this schema update until we're sure the old Nexus instances are never going to use the database again, so this is safe.
  • It's now dangerous to MUPdate one sled containing a Nexus instance to a newer image than the other Nexus instances while they're running, but we already knew that and documented in #8954 so no impact here.

Is all that correct?

@smklein

smklein commented Sep 23, 2025

Copy link
Copy Markdown
Collaborator Author

Do we need to update single-sled too?

I'll update it too (deba8cd). That one isn't the prod we deploy to customers, but might as well make these consistent.

In terms of impact to existing systems, as I understand it:

  • Systems that get updated via the MUPdate process from before this change to after: Nexus will immediately do the schema update on startup, rather than waiting for an operator to run the schema updater by hand. This should be fine because we've previously parked the rack so old Nexus instances shouldn't be running. But I'd like to check in with support about it.

Yup.

  • Systems that get updated via the MUPdate process in the future: same as above.

Yeah, because they'll be skipping the handoff stuff for MUPdate, by remaining "active" and the same nexus generation. So yeah, this will be the same as the case above.

  • For systems that get updated with self-service update: the handoff process + the new DataStore startup code ensures that we don't do this schema update until we're sure the old Nexus instances are never going to use the database again, so this is safe.

Yup, and we'll be in a world where all nexuses are quiescing cooperatively + checking their metadata on boot, to uphold that property that "no old Nexuses are hanging around when the new nexus takes over".

Is all that correct?

Yeah, seems correct to me!

@smklein smklein marked this pull request as ready for review September 23, 2025 02:53

@davepacheco davepacheco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me but I would still like to check in with support about it.

@smklein smklein merged commit 47aa2a9 into main Sep 24, 2025
16 checks passed
@smklein smklein deleted the auto-schema-update branch September 24, 2025 21:18
leftwo pushed a commit that referenced this pull request Sep 26, 2025
Fixes #8912

Should be merged after the rest of Nexus quiesce/handoff is complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let Nexus see schema_dir to self-update

2 participants