split fix android memory kill in split#5940
Conversation
dd6a6ca to
29668b4
Compare
|
GNU testsuite comparison: |
|
I've only given this a cursory glance, but we do probably need to at least keep that |
I do not see the purpose. Its a hidden (undocumented?) parameter. Who will gona use it? Does GNU have it? |
|
Yeah GNU has it, and that's why we have it too. It's also used us some of their tests, so if we want to pass those we need it. |
|
ah, ok, I see. I'll add it again. |
|
How do we know what the purpose of the undocumented argument is about? |
|
Yeah it's mostly for testing I believe, but it'd be nice to make it useful if we have it :) |
2c7dfd8 to
8d2a917
Compare
ci:android fix android memory kill in split
ci:android fix android memory kill in splitsplit fix android memory kill in split
tertsdiepraam
left a comment
There was a problem hiding this comment.
I have a couple of questions/suggestions about this.
- It looks like we're only using the result as
u64, is that correct? Maybe we could just make it useu64instead ofusize. That'll allow us to get rid of the conversions and simplify the code a bit. - I don't think the stdin check is really correct, because we might be piping in a file with a well-defined size. So, I think the check should not just be checking for
-. - We should think about the naming of the functions and document all the public functions. Maybe
sane_blksize? I'd love to get rid of the distinction between the path and metadata versions too, but if that's not possible this is alright.
|
Thanks for reviewing and feedback :-)
I was switching to usize because the value we return should always fit into a usize type.
You are right. Thinking about it a bit more I came to the conclution that I can just remove that code as the function will anyway return the
When removing the special handling for "-", then the path variant will be purely for convenience and can be removed. Is this was you mean? |
|
Great! I think for code paths like these, The solution for I think the path version is adding something useful in the end, so let's keep it. |
3cadde7 to
4832118
Compare
8e51b86 to
b4a038f
Compare
tertsdiepraam
left a comment
There was a problem hiding this comment.
Looking good! Just a few more questions
b4a038f to
7e2a965
Compare
7e2a965 to
e68312c
Compare
|
Thanks! |
splitallocates quite some memory when reading from "files" with no real end like/dev/zero.On x86 its ~ 1GB and on x86_x64 its 2GB. This was cause frequent stability issues on android CI.
This change limits the initially read memory heavily to the
blksizeof the filesystem or 500MB at max if no blksize is specified. This is aligned with a solution used inhead