This reduces memory usage significantly, as in the common
case (no two documents share the same relative path), we will
end up processing one document at a time.

I've tested the new code with a 5.7GB Chromium index, and we're
able to process it with even 100MB of memory (at the cost of
increased GC pressure).

We need to iterate over the index twice, first to get all external symbols,
and then to process documents, as document processing requires access
to the external symbols list. This means we need the ability
to seek to the start again. I've implemented that as follows:

-   For small indexes, just read the index into a slice.
-   For large indexes, save the compressed index to a temporary file
    on disk, and rely on the GC and page cache to transparently
    drop pages earlier in the file when under memory pressure.
    I figured decompressing is cheap enough, that it doesn't make
    sense to have the extra I/O overhead of reading/writing the
    uncompressed index.

Other changes:

-   Documents are no longer sorted by relative path during iteration
    The order of iteration is still deterministic though as it matches the
    order of documents in the index.

(cherry picked from commit b98ca76)

…55160)

This should reduce memory usage significantly, as in the common
case (no two documents share the same relative path), we will
end up processing one document at a time.

I've tested the new code with a 5.7GB Chromium index, and we're
able to process it with even 100MB of memory (at the cost of
increased GC pressure).

We need to iterate over the index twice, first to get all external
symbols,
and then to process documents, as document processing requires access
to the external symbols list. This means we need the ability
to seek to the start again. I've implemented that as follows:
- For small indexes, just read the index into a slice.
- For large indexes, save the compressed index to a temporary file
   on disk, and rely on the GC and page cache to transparently
   drop pages earlier in the file when under memory pressure.
   I figured decompressing is cheap enough, that it doesn't make
   sense to have the extra I/O overhead of reading/writing the
   uncompressed index.

Other changes:
- Documents are no longer sorted by relative path during iteration
  The order of iteration is still deterministic though as it matches the
  order of documents in the index.

Questions:
- Should we add instrumentation see the memory usage at different
  stages of processing an index?
- How do we add a memory usage test (either here or in the upstream
scip bindings)? ([internal Slack
discussion](https://sourcegraph.slack.com/archives/C3B3SDBMY/p1687751363185919))

## Test plan

- [x] Update existing tests
- [x] Add low memory usage test -- added upstream.
scip-code/scip#181 <br> Backport
b98ca76 from #53828

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>