I've rebased this branch onto #3666

Please keep the comments and discussions in this PR related to the code (otherwise we'll get buried by comments!).

Please ask questions in the forum. Thanks a lot!

I've replaced the compress_repo_v2 migration with a new parameter for prune. That way we get all the functionality of prune like limiting how much data is rewritten and also avoid correctness issues from creeping in.

I wonder whether the upgrade_repo_v2 migration should first try to just overwrite the config file, all backends except the rest-server and the mem backend seem to allow that. It would result in slightly different codepaths depending on the backend, but would have the huge benefit for backends that allow atomic replacements, that there's less that can go wrong.

Oh, good idea! I didn't know that we allow overwriting files in the backends. I thought we open files (in sftp and local) with O_CREATE and O_EXCL... at least we used to do that at some point.

@fd0 And they should: I suspect what @MichaelEischer means is "create at temporary path" + "move into place".

Ah! It was dropped when we switched to temporary files. Hm. Maybe we should look into restoring that behavior with local and sftp backends (in a separate issue though).

@fd0 And they should: I suspect what @MichaelEischer means is "create at temporary path" + "move into place".

Since the changes to atomically create files in local/sftp, these backends already creates a temporary file and then move it into place. The Azure/GS/S3/Swift backends also support atomic replacements for files.

Maybe we should look into restoring that behavior with local and sftp backends (in a separate issue though).

For what purpose? The normal files which use a hash as filename don't care because the file content will be identical. And for the config file it shouldn't matter either. Either you're able to overwrite/delete that file or not.

Ah, I just noticed that sftp doesn't overwrite existing files. It's using Rename and not PosixRename. Rename uses the semantic where the target file must not exist yet. AFAIK the openssh implementation hardlinks the temporary file to its new name before deleting the directory entry for the temporary file. I'm not sure it's worth the trouble to replicate that behavior for the local backend.

And for the cloud backends we cannot prevent atomic overwrites either, we could check whether a file exists first, but that costs an additional API call. (and might lead to consistency problems on S3-like backends)

The repo format migration now sort of runs check before upgrading to ensure that the repository is in a good state. There is no interactive output yet as that requires some changes to how migrations works. The upgrade checks are ideally combined with #3730 which warns about a few legacy variants of repo v1.

The migrate command now directly runs check for upgrade_repo_v2. That way we get check progress information and don't have to reimplement check within a migration.