LVTextFileBase: fix parsing of multibytes encodings#515
Conversation
Some bytes/chars could be lost if, while reading, our buffer would end inside a multibyte character, possibly losing sync and causing garbled data.
No, no urgency, it's been like that for years, east asians using old encodings can wait :) |
| bool m_eof; | ||
|
|
||
| void checkEof(lvsize_t bytes_needed); | ||
| void checkEof(int bytes_needed); |
There was a problem hiding this comment.
my proper-data-type loving heart, it bleeds ;p.
There was a problem hiding this comment.
I tried :) but as we now substracts from it, and have a size_t possibly getting a chance to become a offset_t, we'd need another variable for that different type, and it makes it noisier and harder to grasp.
(But good to be there to remind you you have a working heart :)
There was a problem hiding this comment.
Yeah, I was hoping the switch was just because you started running into signedness issues, thanks ;).
| else if ( (ch & 0xE0) == 0xC0 ) { // Should be a 2-bytes wide char | ||
| checkEof(2); | ||
| } | ||
| else if ( (ch & 0xF0) == 0xE0 ) { // Should be a 3-bytes wide char | ||
| checkEof(3); | ||
| } | ||
| else if ( (ch & 0xF8) == 0xF0 ) { // Should be a 4-bytes wide char | ||
| checkEof(4); | ||
| } | ||
| else { // Should not happen? | ||
| checkEof(4); // be sure we don't get stuck | ||
| } |
There was a problem hiding this comment.
I feel like the answer is going to be "because it matches existing code" but I'll try anyway ;p.
Could be simplified as ch < 0xE0 for 2 bytes, ch < 0xF0 for 3 bytes, and a single else for 4 bytes ;).
There was a problem hiding this comment.
because it matches existing code !
Nice try anyway :)
|
I reviewed 4 commits and now they seem perfect. |
|
Thanks ! |
Some bytes/chars could be lost if, while reading, our buffer would end inside a multibyte character, possibly losing sync and causing garbled data.
See issue and discussion in koreader/koreader#10218.
This change is