I don't really understand the intuition behind saving an index once it's more than two minutes old (MinAge) and contains at least 50 blobs (MinBlobs). Assuming you have at least 1MByte/s upload speed and backup a large file. Then with 1.5MB blobs (the average size) this would still be 80 blobs within 2 minutes and therefore immediately trigger an index upload. So this would degrade to Full() == (age>MinAge or blobs>MaxBlobs).

I also didn't understand why upper and lower limits were needed 😉 Just wanted to fix it in a way that it more consistent and intuitive...

I'd propose to simplify this whole method instead and just check for (age>MaxAge or blobs>MaxBlobs) (notice the MaxAge instead of MinAge). And then maybe reduce MaxAge to 10 minutes.

I agree and will change that soon..

I changed the condition as suggested.

The new limit of MaxBlobs = 50k might seem to be just a round number that was picked arbitrarily, but actually it provides a bunch of useful properties:

• One blob entry in an index requires approx. 120 bytes, each listed pack file adds another 85 bytes. For small blobs we can basically ignore the per pack overhead, while for large blobs that overhead applies approximately once every for blobs. This yield 120-141 bytes per blob on average. With 50k blobs the resulting index has between 6 and 7 MB. This is slightly larger than the minimum pack size of 4MB.
• It reduces the memory required to load the (intermediate) index files. Go hashmaps use a load factor of 6.5/8 and allocate an array whose size is a power of two. For 50k entries this requires an array with 65536 slots. The maximum number of entries for that array size would be 53248, which is pretty close to the minimum blobs per pack of 50k. As pack files are added as a whole to the index and a pack can contain hundreds of blobs, we need the wiggle room of a few thousand blobs. This PR also obsoletes Optimize indexMaxBlobs for lower memory usage #2631 .