Overview of the Issue

Context: https://vitess.slack.com/archives/C0PQY0PTK/p1683010265492109

When specifying a timestamp for a snapshot keyspace, we expect all tablets coming up in that keyspace to use the backup at or before the snapshot time. https://vitess.io/docs/17.0/reference/features/recovery/

There seems to be a regression where the restore code is always using the latest backup regardless of what the snapshot time is.

Reproduction Steps

1. Pull the latest from repo.
2. make build & cd examples/local
3. ./101_initial_cluster
4. From GetTablets pick a replica
5. vtctldclient --server localhost:15999 Backup zone1-0000000101

vtctldclient --server localhost:15999 CreateKeyspace --type=SNAPSHOT --base-keyspace=commerce --snapshot-timestamp=2023-07-19T18:16:00Z pitrexample                                                            
Successfully created keyspace pitrexample. Result:
{
  "name": "pitrexample",
  "keyspace": {
    "served_froms": [],
    "keyspace_type": 1,
    "base_keyspace": "commerce",
    "snapshot_time": {
      "seconds": "1689790560",
      "nanoseconds": 0
    },
    "durability_policy": "none",
    "throttler_config": null,
    "sidecar_db_name": "_vt"
  }
}

7. Create another backup for a later timestamp vtctldclient --server localhost:15999 Backup zone1-0000000101

ls vtdataroot/backups/commerce/0                                                                                                                                                                                             
2023-07-19.181518.zone1-0000000100 2023-07-19.181623.zone1-0000000100

9. update vttablet-up.sh to have --init_db_name_override vt_commerce
10. Run CELL=zone1 TABLET_UID=300 ../common/scripts/mysqlctl-up.sh
11. Run CELL-zone1 KEYSPACE=pitrexample TABLET_UID=300 ../common/scripts/vttablet-up.sh
12. Navigate to localhost:15300/debug/vars and check RestoredBackupTime
    

Binary Version

latest on main - also in release 14

Operating System and Environment details

macOS

Log Fragments

No response