eio_linux: make read_dir faster#219
Merged
talex5 merged 1 commit intoocaml-multicore:mainfrom Jun 1, 2022
Merged
Conversation
This improves performance of directory reads by using a larger buffer and by performing all reads in a single systhread, rather than creating a new one for each chunk. - `Read 152448 items in 1.188156 seconds` with the old code. - `Read 152448 items in 0.217013 seconds` with the new code. It also ensures we can already read at least one item.
patricoferris
approved these changes
Jun 1, 2022
Collaborator
There was a problem hiding this comment.
Great! :)) Thanks for the fixes in #218 as well
Collaborator
Author
|
BTW, I found a few other interesting details in https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/readdir64.c;h=e876d84b0249d720d33fd01aa0299e23ff6a2546;hb=HEAD:
We can probably leave it for now and see if it causes any trouble. |
talex5
added a commit
to talex5/opam-repository
that referenced
this pull request
Jun 28, 2022
CHANGES: API changes: - `Net.accept_sub` is deprecated in favour of `accept_fork` (@talex5 ocaml-multicore/eio#240). `Fiber.fork_on_accept`, which it used internally, has been removed. - Allow short writes in `Read_source_buffer` (@talex5 ocaml-multicore/eio#239). The reader is no longer required to consume all the data in one go. Also, add `Linux_eio.Low_level.writev_single` to expose this behaviour directly. - `Eio.Unix_perm` is now `Eio.Dir.Unix_perm`. New features: - Add `Eio.Mutex` (@TheLortex @talex5 ocaml-multicore/eio#223). - Add `Eio.Buf_write` (@talex5 ocaml-multicore/eio#235). This is a buffered writer for Eio sinks, based on Faraday. - Add `Eio_mock` library for testing (@talex5 ocaml-multicore/eio#228). At the moment it has mock flows and networks. - Add `Eio_mock.Backend` (@talex5 ocaml-multicore/eio#237 ocaml-multicore/eio#238). Allows running tests without needing a dependency on eio_main. Also, as it is single-threaded, it can detect deadlocks in test code instead of just hanging. - Add `Buf_read.{of_buffer, of_string, parse_string{,_exn}, return}` (@talex5 ocaml-multicore/eio#225). - Add `<*>` combinator to `Buf_read.Syntax` (@talex5 ocaml-multicore/eio#227). - Add `Eio.Dir.read_dir` (@patricoferris @talex5 ocaml-multicore/eio#207 ocaml-multicore/eio#218 ocaml-multicore/eio#219) Performance: - Add `Buf_read` benchmark and optimise it a bit (@talex5 ocaml-multicore/eio#230). - Inline `Buf_read.consume` to improve performance (@talex5 ocaml-multicore/eio#232). Bug fixes / minor changes: - Allow IO to happen even if a fiber keeps yielding (@TheLortex @talex5 ocaml-multicore/eio#213). - Fallback for `traceln` without an effect handler (@talex5 ocaml-multicore/eio#226). `traceln` now works outside of an event loop too. - Check for cancellation when creating a non-protected child context (@talex5 ocaml-multicore/eio#222). - eio_linux: handle EINTR when calling `getrandom` (@bikallem ocaml-multicore/eio#212). - Update to cmdliner.1.1.0 (@talex5 ocaml-multicore/eio#190).
talex5
added a commit
to talex5/opam-repository
that referenced
this pull request
Jun 28, 2022
CHANGES: API changes: - `Net.accept_sub` is deprecated in favour of `accept_fork` (@talex5 ocaml-multicore/eio#240). `Fiber.fork_on_accept`, which it used internally, has been removed. - Allow short writes in `Read_source_buffer` (@talex5 ocaml-multicore/eio#239). The reader is no longer required to consume all the data in one go. Also, add `Linux_eio.Low_level.writev_single` to expose this behaviour directly. - `Eio.Unix_perm` is now `Eio.Dir.Unix_perm`. New features: - Add `Eio.Mutex` (@TheLortex @talex5 ocaml-multicore/eio#223). - Add `Eio.Buf_write` (@talex5 ocaml-multicore/eio#235). This is a buffered writer for Eio sinks, based on Faraday. - Add `Eio_mock` library for testing (@talex5 ocaml-multicore/eio#228). At the moment it has mock flows and networks. - Add `Eio_mock.Backend` (@talex5 ocaml-multicore/eio#237 ocaml-multicore/eio#238). Allows running tests without needing a dependency on eio_main. Also, as it is single-threaded, it can detect deadlocks in test code instead of just hanging. - Add `Buf_read.{of_buffer, of_string, parse_string{,_exn}, return}` (@talex5 ocaml-multicore/eio#225). - Add `<*>` combinator to `Buf_read.Syntax` (@talex5 ocaml-multicore/eio#227). - Add `Eio.Dir.read_dir` (@patricoferris @talex5 ocaml-multicore/eio#207 ocaml-multicore/eio#218 ocaml-multicore/eio#219) Performance: - Add `Buf_read` benchmark and optimise it a bit (@talex5 ocaml-multicore/eio#230). - Inline `Buf_read.consume` to improve performance (@talex5 ocaml-multicore/eio#232). Bug fixes / minor changes: - Allow IO to happen even if a fiber keeps yielding (@TheLortex @talex5 ocaml-multicore/eio#213). - Fallback for `traceln` without an effect handler (@talex5 ocaml-multicore/eio#226). `traceln` now works outside of an event loop too. - Check for cancellation when creating a non-protected child context (@talex5 ocaml-multicore/eio#222). - eio_linux: handle EINTR when calling `getrandom` (@bikallem ocaml-multicore/eio#212). - Update to cmdliner.1.1.0 (@talex5 ocaml-multicore/eio#190).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This further improves the performance of directory reads by using a larger buffer and by performing all reads in a single systhread, rather than creating a new one for each chunk.
Read 152448 items in 1.188156 secondswith the old code.Read 152448 items in 0.217013 secondswith the new code.(continues #218)