During #1880, the first issue we hit was that a reinstalled Omicron found and attempted to use a datasets for CockroachDB and Clickhouse that were left over from a previous install. I don't think this should ever come up in production, but I think we probably want to change the software to never do this anyway. In this case, we had a fully initialized CockroachDB zone (with database populated and maybe useful data in it) that we wiped with the intent to use in the new install. I don't think that's ever what we would want.
I'm not sure what the best way to avoid this is. We could have the uninstall step remove these datasets, and that might be a good idea for other reasons? But it seems like part of the problem here was that when RSS sent the ensure_dataset call, we found and used the old dataset. Maybe the RSS call should specify a uuid that's different each time RSS determines a plan? That way we'll never find and use and old dataset -- we'd always create a new one for the new install. (We still have to figure out what to do with the old dataset but I think that's related to a separate issue I'm about to file.)
During #1880, the first issue we hit was that a reinstalled Omicron found and attempted to use a datasets for CockroachDB and Clickhouse that were left over from a previous install. I don't think this should ever come up in production, but I think we probably want to change the software to never do this anyway. In this case, we had a fully initialized CockroachDB zone (with database populated and maybe useful data in it) that we wiped with the intent to use in the new install. I don't think that's ever what we would want.
I'm not sure what the best way to avoid this is. We could have the
uninstallstep remove these datasets, and that might be a good idea for other reasons? But it seems like part of the problem here was that when RSS sent the ensure_dataset call, we found and used the old dataset. Maybe the RSS call should specify a uuid that's different each time RSS determines a plan? That way we'll never find and use and old dataset -- we'd always create a new one for the new install. (We still have to figure out what to do with the old dataset but I think that's related to a separate issue I'm about to file.)