Python3 convert braille drivers#9736
Conversation
Focusing specifically on bytes/str
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
LeonarddeR
left a comment
There was a problem hiding this comment.
Here are my comments for the diff until and up to the alva driver.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I've been thinking about the several cases where you know do something like b"".join([arg, int2byte(arg2)]). I think this can be really error prone if you don't have the proper linting tooling. Have you considered something like this? |
|
The usage of that will end up like: I think that it hides the problem. At least with |
LeonarddeR
left a comment
There was a problem hiding this comment.
Comments up to and including ecoBraille
|
unfortunately, there is no in line way to populate a bytearry with groups of bytes, only single bytes.
|
LeonarddeR
left a comment
There was a problem hiding this comment.
Comments for the eurobraille driver
LeonarddeR
left a comment
There was a problem hiding this comment.
Looks like I'm finished for now.
| manuBytes = payload[INFO_MANU_START:INFO_MANU_END].replace( | ||
| FS_BYTE_NULL, b"" | ||
| ) | ||
| self._manufacturer = manuBytes.decode() |
There was a problem hiding this comment.
I think it is safer to assume latin1 or cp1252 than utf-8. I think I prefer latin-1, as the 256 options in latin-1 match with the first 256 code points in Unicode.
There was a problem hiding this comment.
I think latin-1 would match Python 2's behavior. However, assuming UTF-8 might not be a bad idea if a display will have a name including UTF-8 characters in the future. As far as I know, the first characters of latin-1/utf-8 are the same?
There was a problem hiding this comment.
If the first characters are the same, then can we just leave it as UTF-8? Would it be more future proof?
| """ | ||
| @type data: bytes | ||
| """ | ||
| if not isinstance(data, bytes): |
There was a problem hiding this comment.
I think we should use collections.abc.ByteString here, so that bytearray is also allowed as a valid type. It is fully compatible.
There was a problem hiding this comment.
Perhaps, but I think it's outside of the scope of this PR to go down that road. EG finding the optimal types to allow the greatest flexibility. This PR should just be concerned with getting it working. If someone shows that it causes some perf concern to do bytes(myByteArray), or taking a bytes type only is too inflexible for some given case we can look at changing it.
There was a problem hiding this comment.
Note that before this pr, the write function just accepted everything you threwat it. You're now introducing this explicit check for bytes. I won't insist, but I really believe that it is too strict, especially given bytearray is fully compatible with bytes and they are both handled by the ByteString abstract class.
There was a problem hiding this comment.
I reduced the types it could take to simplify verifying correctness. Taking multiple types introduces uncertainty, for instance taking an int, the code must guess how to convert it (number of bytes, byte order, signed), these are device specific and should be decided in the driver. There is certainly an argument for accepting a bytearray, however I don't see that it's necessary, and doesn't greatly impact the code. While it's tempting to try to build a general purpose library, it's much easier to get the known cases correct if we focus on those only.
especially given bytearray is fully compatible with bytes
It should be noted that a bytearray is not convertible by ctypes.create_string_buffer and will not be automatically converted to a bytes object.
Some extra annotations in braille.py and bdDetect.py. Noticed that these files (braille.py and bdDetect.py) require lots more work.
Thanks @josephsl I'll look at this. |
@jcsteh I've been looking at the docs and haven't noticed this specified in relation to WriteFile There is this comment in the code: But it doesn't tell us what error or where to get information on it. I'll look into this more on Monday, perhaps there is more information in the original commit messages / PR. My concerns:
If write size must be a fixed size, then we should be altering the size of the buffer to match and padding with something, or doing multiple writes. |
|
This behaviour of WriteFile is specific to HID devices. Moreover, it isn't well documented and the behaviour differs between versions of Windows. I don't recall the exact error code returned from WriteFile. I do recall having to figure it out through testing with a device; i.e. the docs weren't enough.
It's reasonable to say that writing more than writeSize is an error. I think throwing an exception in that case is reasonable. Certainly, no drivers (current or future) should be doing this.
As for a buffer less than writeSize, we're not reading past the end of the buffer because we're creating a buffer of writeSize. When we put the Python string into the buffer, ctypes only reads as far as the length of the string, since it knows the length. I'm fairly sure ctypes zeroes out buffers it creates (resulting in zero padding). Even if it doesn't, the device should ignore the remaining bytes; the protocol specifies the length of each report and the remaining bytes are just to satisfy the silly Windows requirement of having to write to the maximum report size.
|
This was the only thing causing NVDA not to run in this branch, and will be introduced exactly the same by pr #9736 anyway.
|
Leonard comented among other coments:
In source/brailleDisplayDrivers/papenmeier_serial.py:
@@ -9,15 +9,14 @@
Could you find out whether this driver is still in use by some people? I
think that it is very unlikely that this will ever be tested before a
release, due to the age of these devices.
I know serveral German persons who use such displays. May
@michaelDCurran could provide some statistic.
|
|
We see a maximum of 8 users using papenmeier_serial over the last 30 days.
Of course this only counts users who opted in to stats gathering, so the
number could be 30% higher, perhaps 10 or 11 users. Plus, it only tracks
users from 2018.3 onwards.
|
|
I'm merging in current threshold_py3_staging, so we can check appveyor status for this |
…on3-convertBrailleDrivers
By creating the buffer manually, we can ensure that data smaller than OutputReportByteLength will be padded with null bytes. I tested ctypes marshalling by creating a C++ dll, loading it and calling it with ctypes. Small refactor, only hwIo.HID devices were using _writeSize. hwIo.Bulk took the parameter, but it was only set to 0 by the hims driver. This failed the Truth test, and len(data) was used in this case. The refactor allows the HID complexity to stay with the HID class. If it becomes necessary to support buffers larger than OutputReportByteLength further refactoring will become necessary.
… python3-convertBrailleDrivers
Use Optional instead of Union[sometype, None] Checked all usages of: - ord() - bytes() - bytearray() - str() Found and fixed several issues.
|
I've just re-requested reviews from @LeonarddeR and @bramd. Just a note, we would like to merge this as soon as we can to start getting broader testing. Please focus reviewing efforts on looking for bugs / mistakes. |
|
I have tested the following driver/display combinations over USB:
|
LeonarddeR
left a comment
There was a problem hiding this comment.
I looked at the diff from last review actions, and they look ok.
|
Was the idea for this pr to also convert brailleInput.py? @feerrenrut
requested me not to touch brailleInput.py when fixing unichr issues but
in this pr I'm getting the following traceback when typing on an Orbit
Reader braille display:
```
Traceback (most recent call last):
File "scriptHandler.py", line 189, in executeScript
script(gesture)
File "globalCommands.py", line 2059, in script_braille_dots
brailleInput.handler.input(gesture.dots)
File "brailleInput.py", line 247, in input
if self._translate(endWord):
File "brailleInput.py", line 129, in _translate
data = u"".join([unichr(cell | LOUIS_DOTS_IO_START) for cell in
self.bufferBraille[:pos]])
File "brailleInput.py", line 129, in <listcomp>
data = u"".join([unichr(cell | LOUIS_DOTS_IO_START) for cell in
self.bufferBraille[:pos]])
NameError: name 'unichr' is not defined
```
…On 6/27/2019 11:52 PM, Leonard de Ruijter wrote:
***@***.**** approved this pull request.
I looked at the diff from last review actions, and they look ok.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9736?email_source=notifications&email_token=ADL7THLJ5PDFUWJFSTMQQ3LP4TA3NA5CNFSM4HXZP3J2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB43K3QY#pullrequestreview-255241667>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADL7THIVW6532G3MUJI6543P4TA3NANCNFSM4HXZP3JQ>.
|
|
Successfully used an Orbit Reader to read and navigate with NVDA, in both bluetooth and USB HID. Was also able to type braille input with #9835 merged into this branch. |
|
Successfully read / navigated / typed on Baum Refreshabraille 18 with both USB and blutooth. |
Yes, actually, it seems I missed it. Perhaps we merge these two PR's then I'll take another look through |
|
I tested the following displays over USB:
These work fine after applying the fixes in #9839 |
…on3-convertBrailleDrivers
Link to issue number:
Related to #9637
Summary of the issue:
The braille driver framework, and many of the drivers use strings interchangeably with bytes.
Description of how this pull request fixes the issue:
Since quite a careful approach is required to track down which literals should be bytes / vs strings and much of the surrounding logic needs to be modified I am converting the braille drivers to python 3 files-wise rather than feature-wise as we are elsewhere in the code base.
To help with this I have been using typehints and a linter to find issues. This is not perfect, it's quite likely issues have slipped through.
Testing performed:
None, this will require testing from people with devices.
Several people have voluntered to test with different devices:
Known issues with pull request:
Change log entry:
Section: New features, Changes, Bug fixes