While investigating a performance issue in a library I am working on, I noticed that parsing a particular document (warning: very large file) causes the program to spend a lot of time in memchr (roughly 70% of its time). Pausing execution in gdb gives the following stack trace: (this is an image to keep the colors)
to the point that parsing is only ~3x faster than doing it with PyYAML. The speedup I get for other files is >10x.
It looks like the problem is that reading floats is done using c4::from_chars -> c4::atof -> sscanf which ultimately calls _IO_str_init_static_internal in the glibc implementation. As far as I understand, this essentially causes strlen to be called on the string that rapidyaml passes to it, essentially causing strlen to be called O(n^2) times.
Since there are a lot of floats throughout the entire document, parsing appears to become accidentally quadratic.
Not sure how to fix this. Even using the standard atof might not help since that function doesn't take the length either...
While investigating a performance issue in a library I am working on, I noticed that parsing a particular document (warning: very large file) causes the program to spend a lot of time in
memchr(roughly 70% of its time). Pausing execution in gdb gives the following stack trace: (this is an image to keep the colors)to the point that parsing is only ~3x faster than doing it with PyYAML. The speedup I get for other files is >10x.
It looks like the problem is that reading floats is done using c4::from_chars -> c4::atof -> sscanf which ultimately calls _IO_str_init_static_internal in the glibc implementation. As far as I understand, this essentially causes strlen to be called on the string that rapidyaml passes to it, essentially causing strlen to be called O(n^2) times.
Since there are a lot of floats throughout the entire document, parsing appears to become accidentally quadratic.
Not sure how to fix this. Even using the standard atof might not help since that function doesn't take the length either...