Why not let restic automatically assign pseudo-device ids as suggested in #3041 (comment) ? As rawtaz and I have commented (see e.g. #3041 (comment) ) --device-map looks like it requires rather a lot of setup to work.

The PR currently does not address device ID collisions at all, which can be a problem once users start fiddling with the device ids.

Hi @MichaelEischer, thanks for taking a look. I'd be happy to implement an automatic mapping algorithm, but I'm not sure how to go about it.

Consider a scenario with 4 files, spread across two devices:

/device1/a.txt
/device1/b.txt
/device2/c.txt
/device2/d.txt

and let's say that for the first backup, device1 has a DeviceID of 17, and device2 has a DeviceID of 18.
For the second run let's say Device1 has a DeviceID of 81 and Device2 has a DeviceID of 82

The question becomes: How do we map device1 and device2 to pseudo-ids which are consistent across runs?
Let's say you do something like hash the path, or the inode. So we hash /device1/a.txt and let's say that hashes to 3. So we have a pseudo-map that says "device 17 maps to pseudo 3". Now when we see /device1/b.txt, the mapping already exists so we can use the existing value. c.txt is missing a mapping, so you would hash it and then do something like "device 18 maps to pseudo 4"

But the problem with this approach is it isn't stable across backups, unless enumeration happens in the same order. For example, on the next backup, let's say that instead of backing up a.txt, the enumeration finds /device1/b.txt first (or maybe you deleted a.txt).

Now, the algorithm would see that there is no device mapping for ID 81, and so it would hash /device1/b.txt. But now this is a different file/inode so let's say that maps to pseudo id 11. And now we haven't solved anything because the pseudo-id's are not consistent across runs.

Some ideas I considered, but with obvious flaws:

• Find some way to do enumeration in-order of devices (I don't see how you can enforce this with hard links)
• Poke into the guts of the filesystem to get the mappings (e.g. look at /proc/mounts) and build a hash off the base mount path, rather than the file
• For each file, try to get it's parent, (and keep getting its parent), until the DeviceID changes. Use that folder path/inode as the input for the hash algorithm. You'll still run into issues with hard links though. For example, if you hard link /device1/folderA into /device2/folderA, then iyou would run into the same issue depending on if the backup code sees /device1/a.txt or /device2/folderA/c.txt. The former would point to /device1 as the root device, and the latter would point to /device2/folderA as the root node

The last approach seems most reasonable to me but also fairly involved. I don't know enough about all the nuances of file systems (including network drives) to know what gotcha's I'd run into trying to traverse up parent directories.

The benefit of the current approach is it's very iterative (code-wise) and results in a small diff with consistent, repeatable behavior. The last approach would involve a much larger diff and would be more likely to have a tail of "it's not stable in this configuration" etc.

Happy to hack on ideas more, once we have an idea of how the algorithm should work.

Thinking about this some more, it looks like findmnt gives me the set of device ID information needed to make a stable lookup. However at that point, I might as well just implement -use-fs-snapshot and be done with it. I'll think on this some more

Is this issue specific to Linux? I'm wondering how many systems have the issue but don't have findmnt.

I am using ZFS snapshots with restic and am affected by issue 3041. With the --device-map flag my strategy for using it would be to assign pseudo device IDs based on the mount point. These are very stable for my use case.
My experiments show that everything else is stable except the inode of the directory of the mountpoint.

#!/bin/bash
for m in $(findmnt -R -T . --json | jq --raw-output '.filesystems[0].children[].target') ; do
  SRCDEV=$(stat --format "%Ld" $m)
  DEVHASH=$(echo ${m//$(pwd)/} | b2sum --length=64 | cut -f1 -d" " | tr a-f A-F)
  PSEUDODEV=$(echo "obase=10; ibase=16; ${DEVHASH}" | bc)
  echo --device-map ${SRCDEV}:${PSEUDODEV}
done

But the problem with this approach is it isn't stable across backups, unless enumeration happens in the same order. For example, on the next backup, let's say that instead of backing up a.txt, the enumeration finds /device1/b.txt first (or maybe you deleted a.txt).

The backup sorts filenames before traversing a folder. Thus the traversal order is stable.

For example, if you hard link /device1/folderA into /device2/folderA, then iyou would run into the same issue depending on if the backup code sees /device1/a.txt or /device2/folderA/c.txt.

Hardlinks cannot point across devices. Or is there some subvolume related trickery that avoids that limitation?

The archiver component already traverses the filesystem starting from the root directory /. So we'd just have to call stat before entering a directory afterwards to detect when the mountpoint has changed.

Hardlinks cannot point across devices. Or is there some subvolume related trickery that avoids that limitation?

Although hardlinks cannot be made across devices, bind mounts can be made at different points in the filesystem. The bind mounts all share the same device ID.

With the traversal order being stable, bind mounts don't seem like an issue though. The mount that is seen first would be the path used for the pseudo identifier for the device. If this path is hashed in some way to produce the identifier it wouldn't matter how many other devices were seen before this device.

Hey, I've been successfully using this patch in unattended backups, though, on a filesystem without any bind mounts in it.
Rebasing the patch for new releases becomes increasingly hard, though, as the gap between the source tree and the patch widens.
May I ask for a review and merge, if possible?

Thanks!

As discussed in #3041 we want to have a solution that just works, without having users manually map device ids. That is the PR is just there as a temporary workaround, but is unlikely to get merged.

As discussed in #3041 we want to have a solution that just works, without having users manually map device ids. That is the PR is just there as a temporary workaround, but is unlikely to get merged.

I'm going to close this PR out. My personal opinion is that in the search of a perfect solution, we've sacrificed a good enough one. This issue has been open for 3 years now. It's frustrating that a low-risk, opt-in fix wasn't able to make it over the line (and a better one hasn't materialized either)

There's now #4006 .