Pre-allocate space in shared memory files#2267
Conversation
Replace usage of `ftruncate` with `posix_fallocate`. `ftruncate` sets the size of a file without actually allocating the backing storage, which is allocated on-demand. While this allows the true allocation to be "lazy", and thereby allows us to do large allocations without wasting resources if the whole allocation isn't used, it can cause unpredictable and confusing failure modes. With `posix_fallocate`, the space is eagerly allocated. If the call succeeds, we are guaranteed that the storage is actually available. When we run out of storage, the call fails and we can fail with a clear error message. We were previously allocating quite-large chunks of memory at a time; about 132 MB. While this probably wouldn't actually cause problems for a single shadow simulation (which typically only needs one such allocation), it ends up using a lot of space when running the shadow tests in parallel. This changes the default allocation to 1 MB, instead. Fixes shadow#2266
|
Running a benchmark: https://github.com/shadow/benchmark/actions/runs/2638637530 |
rwails
left a comment
There was a problem hiding this comment.
Looks good. I'm glad the default mem block file size is being reduced; I remember setting it to a high value when we were prototyping, but I didn't realize that the 100 MB value stuck!
I spot-checked to make sure that most of the allocations being made are under 500 k, and I believe this is the case. (Otherwise, we'd lose small-object allocation optimizations and the shmem allocator would make a shmem syscall for every allocation, which is how it handles large objects).
Thanks Jim!
It was completely reasonable when using
Thanks! Yeah the benchmark results looks ok too; run-time and memory usage didn't change compared to the previous nightly run. |
Replace usage of
ftruncatewithposix_fallocate.ftruncatesetsthe size of a file without actually allocating the backing storage,
which is allocated on-demand. While this allows the true allocation to
be "lazy", and thereby allows us to do large allocations without wasting
resources if the whole allocation isn't used, it can cause unpredictable
and confusing failure modes.
With
posix_fallocate, the space is eagerly allocated. If the callsucceeds, we are guaranteed that the storage is actually available.
When we run out of storage, the call fails and we can fail with a clear
error message.
We were previously allocating quite-large chunks of memory at a time;
about 132 MB. While this probably wouldn't actually cause problems for a
single shadow simulation (which typically only needs one such
allocation), it ends up using a lot of space when running the shadow
tests in parallel. This changes the default allocation to 1 MB, instead.
Fixes #2266