Output of restic version

restic 0.16.4 compiled with go1.21.6 on windows/amd64

What backend/service did you use to store the repository?

Tested with rest server (https) and local folder. Probably doesn't matter.

Problem description / Steps to reproduce

Summary:
It seems that Restic is treating the encoding of a key file differently depending on whether the file is used for initializing/accessing the repository, or when a new key file is added. I discovered that issue when working with PowerShell on Windows. When piping command output to a file using PowerShell, the created file is encoded using UTF-16 by default.

Key issue:
Although adding an UTF-16 encoded key file as a key to the repository seems to work, this added key file cannot be used to access the repository. If a UTF-16 encoded recovery key would be added but not tested, one wouldn't be able to access any backups using this recovery key.

Steps to reproduce:
The example below uses PowerShell since this creates UTF-16 files by default. However, the issue probably can also be reproduced if the key file is created manually using the "problematic" encoding.

First. generate 2 simple key files (key1 and key2), initialize the repo, add the second key, and list all keys using key1:

PS C:\Users\dummy> "test1" > key1
PS C:\Users\dummy> "test2" > key2
PS C:\Users\dummy> restic.exe --password-file=key1 --repo ./data init
created restic repository b441efffee at ./data

Please note that knowledge of your password [...]

PS C:\Users\dummy> restic.exe --password-file=key1 --repo ./data key add --new-password-file key2
repository b441efff opened (version 2, compression level auto)
created new cache in C:\Users\dummy\AppData\Local\restic
saved new key as <Key of MYHOST\dummy@MYHOST, created on 2024-06-05 16:10:58.3340282 +0200 CEST m=+3.575316901>

PS C:\Users\dummy> restic.exe --password-file=key1 --repo ./data key list
repository b441efff opened (version 2, compression level auto)
 ID        User          Host     Created
-----------------------------------------------------
*8cb67921  MYHOST\dummy  MYHOST  2024-06-05 16:09:57
 997852d4  MYHOST\dummy  MYHOST  2024-06-05 16:10:58
-----------------------------------------------------

(Remark: For simplicity of the example I piped a String to file which of course is not recommended for several reasons. However, the same issue can be reproduced when output of another command is redirected to a file, e.g. Get-RandomPassword > password.txt)

So far everything seems to went fine without errors. However, when we now try to access the repo using key2, this key is not accepted, although it has been added before:

PS C:\Users\dummy> restic.exe --password-file=key2 --repo ./data key list
Fatal: wrong password or no key found

To further examine the issue, we will convert key2 to UTF-8 and save it as key2-UTF8. However, accessing the repo using the UTF-8 encoded file is also not working:

PS C:\Users\dummy> Get-Content key2 | Set-Content key2-UTF8
PS C:\Users\dummy> restic.exe --password-file=key2-UTF8 --repo ./data key list
Fatal: wrong password or no key found

Now key2-UTF8 is added as a key (key1 is used for this operation):

PS C:\Users\dummy> restic.exe --password-file=key1 --repo ./data key add --new-password-file key2-UTF8
repository b441efff opened (version 2, compression level auto)
saved new key as <Key of MYHOST\dummy@MYHOST, created on 2024-06-05 16:22:27.7545357 +0200 CEST m=+2.787216801>

Now, both key2 as well as key2-UTF8 can be used to access the repository:

PS C:\Users\dummy> Get-Content .\key2-UTF8
test2
PS C:\Users\dummy> restic.exe --password-file=key2-UTF8 --repo ./data key list
repository b441efff opened (version 2, compression level auto)
 ID        User          Host     Created
-----------------------------------------------------
*27276e81  MYHOST\dummy  MYHOST  2024-06-05 16:22:27
 8cb67921  MYHOST\dummy  MYHOST  2024-06-05 16:09:57
 997852d4  MYHOST\dummy  MYHOST  2024-06-05 16:10:58
-----------------------------------------------------

PS C:\Users\dummy> Get-Content .\key2
test2
PS C:\Users\dummy> restic.exe --password-file=key2 --repo ./data key list
repository b441efff opened (version 2, compression level auto)
 ID        User          Host     Created
-----------------------------------------------------
 8cb67921  MYHOST\dummy  MYHOST  2024-06-05 16:09:57
*27276e81  MYHOST\dummy  MYHOST  2024-06-05 16:22:27
 997852d4  MYHOST\dummy  MYHOST  2024-06-05 16:10:58
-----------------------------------------------------

Please note that in both cases, the key with the ID 27276e81 was used to access the repository.
Or with other words, the UTF-16 encoded file can now also be used to access the repository, which failed before:

PS C:\Users\dummy> file key*
key1:      Unicode text, UTF-16, little-endian text, with CRLF line terminators
key2:      Unicode text, UTF-16, little-endian text, with CRLF line terminators
key2-UTF8: ASCII text, with CRLF line terminators

PS C:\Users\dummy> Get-Content .\key1, .\key2, .\key2-UTF8
test1
test2
test2

Expected behavior

When adding an UTF-16 encoded file as a key, this added key should actually allow accessing the repository, or at least adding this key should fail, instead of siltently adding a corrupted key.

Actual behavior

Restic fails with "wrong password or no key found" when accessing a repository with a UTF-16 encoded key file that has been "successfully" (but apparently using the wrong encoding and therefore corrupted) added before.

Do you have any idea what may have caused this?

My guess is that when a repo is initialized or accessed, the key file is properly decoded. However, when an additional key file is added to the repository, the file is handled incorrectly and therefore later on cannot be used to access the repository.

Reason:

• When adding key2 (UTF-16) as a key to the repo, it cannot be used to access the repository.
• After adding key-UTF-8 (same content, but UTF-8 encoded), both files (UTF-8 and UTF-16) can then be used to access the repository.
• This leads to my assumption, that during init/access, restic probably decodes the file, but file encoding is handled differently during adding of a key.

Did restic help you today? Did it make you happy in any way?

I love Restic a lot! :) But I am also very happy that I discovered this issue before actually having to rely on a broken recovery key for restoring my data... 😄