Merged
Conversation
Member
Author
|
Broken for packages which use |
Member
Author
|
Note that, on existing opam installs, this will work transparently and tar-gzip the repo dirs on the next |
cb2c91e to
278bd72
Compare
rjbou
reviewed
Feb 27, 2019
rjbou
reviewed
Feb 27, 2019
Collaborator
|
LGTM! thanks! |
Makes things *much* faster on older HDDs by keeping local repositories as .tar.gz instead of thousands of scattered files. The implementation isn't very nice at the moment though.
keep a closer look on the extracted dir lifespan, they are generally not needed so only expand/keep it when required.
Allowing for loading extra files or raw opam files whenever needed. Cleanup is done when releasing the repository state, or on exit.
We need to be more careful since they now require a finaliser to clean up their associated temporary directory.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repositories stored in
~/.opam/repocontain thousand of tiny filesand directories, and can take a huge amount of time to load on HDDs,
networked or old filesystems. This is not visible in normal use
because there is a marshalled cache, but can cause
opam updateto beextremely slow (it now needs to compute a diff on the files, which
requires re-loading the repository drom disk), or big lags when
changing your opam version (and the full file tree is not yet in the
OS memory cache).
The solution proposed here is extremely pragmatic, yet quite
efficient: we store the repository contents as
.tar.gzfiles in~/.opam/repoinstead. Rather than resorting to a complex in-memorystructure, we just untar them to
/tmpwhen they need to be read, andre-tar them after modification (
opam update, or format upgradeonly). Then we let the OS disk cache do the job: in normal operation,
the tree never needs to be flushed to disk, and loading the
.tar.gzis orders of magnitude faster than loading the individual files.
Note that this is done even for
rsyncorgitrepositories, whichis not particularly clean, but works. In the case of
git, it wouldbe possible to just store a bare repository, and use
gitto extractthe individual files (this could even be done explicitely, directly to
memory, see how
Camelusperforms). But it does not seem worth theadditional implementation cost at the moment.