Bug description

This bug was investigated and validated with the assistance of Opus 4.6/Gpt 5.4. All reproduction scripts, patches, and test results were executed and verified on a real environment.

thv secret set silently loses secrets when another process writes concurrently. Once persisted, a secret is safe (all future snapshots include it), but if OAuth-enabled servers are already running, their token refreshes prevent new secrets from being added reliably. A contributing factor is high frequency oauth token refreshes.

SetSecret, DeleteSecret, and Cleanup in EncryptedManager contain a TOCTOU race condition. NewEncryptedManager reads and decrypts the secrets_encrypted file into an in-memory syncmap.Map at construction time, outside any lock. When SetSecret later acquires WithFileLock, it updates the stale in-memory map and writes the entire map back to disk, silently overwriting any changes written by another process between the initial read and the locked write.

Steps to reproduce

(while true; do
# Background writer simulating OAuth token refresh
  for k in 1 2 3 4 5; do
    echo "v$k" | thv secret set "_churn_$k" &>/dev/null
  done
  sleep 0.05
done) &
BG=$!

lost=0; total=50
for i in $(seq 1 $total); do
  thv secret delete my-secret &>/dev/null || true
  echo "value" | thv secret set my-secret &>/dev/null
  sleep 0.1
  thv secret get my-secret &>/dev/null || lost=$((lost + 1))
done

kill $BG 2>/dev/null; wait $BG 2>/dev/null
echo "Lost: $lost / $total"

8% loss on macOS with the background writer. 0% without it. Anecdotally it's closer to 50% after resuming from sleep / restarting toolhive.

Expected behavior

thv secret set should permanently retain a secret.

Actual behavior

• thv run <mcp server> fails with:

  Error: failed to validate workload parameters: error processing secrets:
  failed to get secrets: secret not found: <secret name>
  

• thv secret get <name> returns secret not found immediately after a successful
  thv secret set.
• thv secret list shows hundreds of OAuth refresh tokens but no <secret name> entries.
• Secrets work right after setting, then disappear after the next OAuth token refresh
  overwrites the file from its stale snapshot.

Environment (if relevant)

• OS/version: macOS 26.3.1
• ToolHive version: v0.11.3 (v0.12.5 tested less extensively)
• ToolHive Desktop: v0.23.0 (v0.25.0 tested less extensively)
• Secret provider: encrypted
• Config: provider_type: encrypted in config.yaml

Additional context

The bug is in pkg/secrets/encrypted.go. Three mutating methods are affected:

The file lock (pkg/fileutils/lock.go, using gofrs/flock) only serializes the
write. Each CLI invocation creates a fresh EncryptedManager that snapshots the file
at construction time. By the time the lock is acquired, the snapshot may be stale.

Workaround

Our current workaround for internally developed MCP server instructions is to write-then-verify with retries to reduce the probability (0/200 losses vs 17/200 without the workaround):

#!/bin/bash
# Usage: ./thv-secret-set.sh my-secret
set -euo pipefail
NAME="$1"
read -rsp "Enter secret value: " VALUE; echo

for attempt in $(seq 1 10); do
  printf '%s' "$VALUE" | thv secret set "$NAME" 2>/dev/null
  sleep 1
  if thv secret get "$NAME" &>/dev/null; then
    echo "✓ $NAME persisted (attempt $attempt)"
    exit 0
  fi
  echo "  attempt $attempt: $NAME not found, retrying..." >&2
done

echo "ERROR: $NAME failed to persist after 10 attempts" >&2
exit 1