Refactor: Optimize create_test_file in benchmark.py#84
Refactor: Optimize create_test_file in benchmark.py#84lxp merged 7 commits intocfv-project:python3from
Conversation
|
Thanks for your pull request! Ideally, I would like to get rid of the old behavior at all, since it is very inefficient. Can you maybe come up with a middle way like reading and writing blocks of 64k? Line 40 in 79587c1 Please also fix the formatting error (see https://github.com/cfv-project/cfv/actions/runs/18862085679/job/54029609083?pr=84): |
|
Hello @lxp, I agree with your suggestion I changed the code to write files split by chunk, not the whole data. Also, I fixed the flake8 formatting error Thank you for quick review! |
lxp
left a comment
There was a problem hiding this comment.
Thanks for updating your pull request accordingly.
I made a few more suggestions with concrete changes.
We should be able to merge it with this changes (at least when there are not additonal flake8 formatting errors).
| while size: | ||
| f.write(b'%c' % random.randint(0, 255)) | ||
| size -= 1 | ||
|
|
There was a problem hiding this comment.
There is still a flake8 error on this line.
Run flake8 --exclude=build,venv --ignore= --max-line-length=200 --max-complexity=75 --show-source --statistics .
./test/benchmark.py:73:1: W293 blank line contains whitespace
^
1 W293 blank line contains whitespace
Error: Process completed with exit code 1.
See: https://github.com/cfv-project/cfv/actions/runs/18927372363/job/54136536126?pr=84
Co-authored-by: Lisa Gnedt <lisa@gnedt.at>
Co-authored-by: Lisa Gnedt <lisa@gnedt.at>
Co-authored-by: Lisa Gnedt <lisa@gnedt.at>
Co-authored-by: Lisa Gnedt <lisa@gnedt.at>
The original function used a while loop to write one random byte at a time, so I use
os.urandom(size)to efficiently generate the full size bytes of random data in memory.If the requested size is too large to fit in available system RAM (e.g., 50GB file on a 16GB machine), a MemoryError is caught. In this scenario, the function falls back to the original, memory-efficient byte-by-byte write logic.