Skip to content

improve performance of TOML parsing by reading from memory instead of from file#1751

Merged
KristofferC merged 6 commits intomasterfrom
kc/parse_speed
Apr 7, 2020
Merged

improve performance of TOML parsing by reading from memory instead of from file#1751
KristofferC merged 6 commits intomasterfrom
kc/parse_speed

Conversation

@KristofferC
Copy link
Copy Markdown
Member

@KristofferC KristofferC commented Apr 6, 2020 •
edited
Loading

Reading a bunch of small things from a file tends to be much slower than just reading the whole file in one chunk and operating on memory (especially with Julia now locking the stream for every read). None of the files we parse are big enough that operating on a file makes sense.

Benchmark:

function parse_registry()
    path = joinpath(homedir(), ".julia/registries/General")
    for (root, dirs, files) in walkdir(path)
       for file in files
           if endswith(file, ".toml")
               TOML.parsefile(joinpath(root, file))
           end
       end
   end
end

Before:

696.417 ms (2111063 allocations: 157.45 MiB)

After:

355.687 ms (2164815 allocations: 166.51 MiB)

@KristofferC
Copy link
Copy Markdown
Member Author

KristofferC commented Apr 6, 2020

Ah, this caught a too loose try catch.

@KristofferC KristofferC merged commit 47784de into master Apr 7, 2020
@KristofferC KristofferC deleted the kc/parse_speed branch April 7, 2020 07:10
KristofferC added a commit that referenced this pull request May 10, 2020
@KristofferC
improve performance of TOML parsing by reading from memory instead of…
ae6e8ac 
… from file (#1751)

* improve performance of TOML parsing by reading from memory instead of file

(cherry picked from commit 47784de)
@KristofferC KristofferC mentioned this pull request May 25, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@StefanKarpinski StefanKarpinski StefanKarpinski approved these changes

Assignees

No one assigned

Labels

performance TOML

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@KristofferC @StefanKarpinski @fredrikekre