problem

running any kopia command that requires accessing the repository (e.g. kopia snapshot list) results in an error like:

ERROR open repository: unable to open repository: unable to create shared content manager: error loading indexes: unable to open pack index "xn84_6260a73609a1d24fef712646899085f4-s71751e05a7275847138-c1": unable to open file despite retries: open /home/maynard/.cache/kopia/bfc036a6bbb0d4c1/indexes/xn84_6260a73609a1d24fef712646899085f4-s71751e05a7275847138-c1.sndx: too many open files

this has been reported multiple times already:

• https://kopia.discourse.group/t/cant-check-snapshot-list-in-client/2095
• https://kopia.discourse.group/t/too-many-open-files/3293
• https://kopia.discourse.group/t/average-epoch-time/3295
• https://kopia.discourse.group/t/error-failed-to-open-repository-too-many-open-files/1286/4
• Too many open files #2633
• unable to find snapshot manifests #3070

current workaround

the usual recommendation is to raise the open-files limit (ulimit -n).

for example, my defaults are:

$ ulimit -Hn
4096

i can make kopia work temporarily with:

sudo prlimit --pid "$$" --nofile=8192:8192

but this raises two questions:

1. will I need to keep increasing this number as the repository grows?
2. should kopia really require thousands of simultaneously open files?

observations

• the error involves cached index files at:
  .cache/kopia/${HASH}/indexes/${INDEX_ID}

• my cache directory contains over 4000 files:

ls -l ~/.cache/kopia/bfc036a6bbb0d4c1/indexes | wc -l
4148

• this slightly exceeds my default limit (4096), explaining the error.

• clearing the cache (kopia cache clear or deleting ~/.cache/kopia) doesn’t help — the cache quickly grows back to the same scale.

• it looks like kopia attempts to open all these files simultaneously rather than reading and closing them, which leads directly to hitting the system limit.

question

would it be possible for kopia to avoid holding all index files open at once? for example, by:

• loading them into memory and closing the file descriptors, or
• batching access to a smaller working set

this would make kopia behave better on systems with default limits, without requiring system-wide changes.

thanks 🙏

kopia is a fantastic tool and i’m very grateful for it. i’m opening this not just for myself but also so anyone else who runs into the same issue can find a single place with discussion and possible solutions.