I created a brand new VM and attached remote-ssh to it, then opened my ~ folder on it. I clicked to trust the folder, then immediately after I was prompted by the terminal to trust the folder again.

Every time I open a new folder thereafter I'm again double prompted for trust.