Skip to content

Composable Shell touch keyboard: support emoji panel and other features in Windows 10 Fall Creators Update and later. re #7273#7378

Merged
michaelDCurran merged 12 commits into
nvaccess:masterfrom
josephsl:i7273
Jul 19, 2018

Conversation

@josephsl

@josephsl josephsl commented Jul 11, 2017

Copy link
Copy Markdown
Contributor

Hi,

Revision: revised in July 2018 to base it on the template.

Link to issue number:

Fixes #7273

Summary of the issue:

Support modern keyboard features, including emoji panel, hardware input suggestions, dictation, and cloud clipboard.

Description of how this pull request fixes the issue:

Adds an app module for Composable Shell keyboard. This overlay app provides the following features:

Testing performed:

Searching for emojis, dictating text, cloud clipboard paste, entering text with hardware input suggestion turned on in various Windows 10 releases. Note that because it is an app module, it was tested with Windows 10 App Essentials add-on first.

Known issues with pull request:

None, although features are subject to change.

Change log entry:

Section: new features:
Added support for modern input features in recent Windows 10 releases. These include emoji panel (Fall Creators Update), dictation (Fall Creators Update), hardware keyboard input suggestions (April 2018 Update), and cloud clipboard (RS5).

Technical details:

  • The new XAML touch keyboard has its own app module.
  • If one has United States English keyboard layout and is running build 16215 or later, one can press Windows+period or Windows+semicolon to search for and enter emojis. The keyboard layout requirement was relaxed in build 17134.
  • One can also type emoji descriptions to find the exact emoji.
  • Each emoji candidate raises element selected event when selected.
  • In some cases, when the emoji panel first opens, the first emoji isn't announced because element selected event is not fired then, although window open event is fired.
  • This app was later exte3nded to cover dictation (Windws+H), cloud clipboard and announcing hardware input suggestion candidates.

Thanks.

michaelDCurran
michaelDCurran previously approved these changes Apr 15, 2018
michaelDCurran added a commit that referenced this pull request Apr 15, 2018
@josephsl josephsl changed the title Composable Shell touch keyboard: support emoji panel in Windows 10 Fall Creators Update. re #7273 Composable Shell touch keyboard: support emoji panel and other features in Windows 10 Fall Creators Update and later. re #7273 Jun 5, 2018
josephsl added a commit that referenced this pull request Jun 24, 2018
@josephsl

Copy link
Copy Markdown
Contributor Author

Hi,

Note that this app module also supports dictation, so we get that for free.

Thanks.

@josephsl

Copy link
Copy Markdown
Contributor Author

Hi,

Hmmm, I don't know why, but when I was assigned, the review status was set to "review required" when in fact this was reviewed before.

Thanks.

@josephsl

Copy link
Copy Markdown
Contributor Author

Hi,

Yet again, Microsoft has changed emoji panel in 17704, although in a subtle way: when browsing emoji, in addition to element selected event, name change event is fired by an ancestor control which displays selected emoji character and how many items there are in a category. Unfortunately, this element is the one that says things such as "emoji panel is closed", announces categories when browsing them in Redstone 5 and such. Also, name change is fired by emoji grid items when people/skin tone is selected.

Thanks.

josephsl added 8 commits July 12, 2018 20:50
…ll Creators Update. re nvaccess#7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.
…vaccess#7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.
…s#7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.
… Re nvaccess#7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.
… cloud clipboard suggestions. Re nvaccess#8189.

Modern keyboard is not just used for emoji panel entry: in version 1803, hardware keyboard input suggestions are lited there. In Redstone 5, cloud clipboard entries are shown through this panel. Thus add support for both.
… touch keyboard's name change event. Re nvaccess#7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.
…ries. Re nvaccess#7273.

Several changes as a result of deploying build 17700 series:
* Clipboard: NVDA will no longer announce label for clipboard candidates list.
* Emoji panel: NVDA will not respond to verbose name change events fired whenever items are selected.
@michaelDCurran

Copy link
Copy Markdown
Member

It would be great if you could rewrite your pr description to follow the template. Also, can you provide some clear steps to reproduce in order to test the features? I think I understand how emoji input works, but I'm not sure how to test dictation.

@josephsl

Copy link
Copy Markdown
Contributor Author

Hi,

To test dictation, you need a microphone and:

  1. Opoen Notepad.
  2. Press Windows+H to dictate something.

NVDA will then announce dictated text via name change event.

Thanks.

@michaelDCurran

Copy link
Copy Markdown
Member

Sometimes I am seeing the following traceback when starting to type a search string into the emoji panel:

ERROR - eventHandler.executeEvent (17:32:57.532):
error executing event: UIA_elementSelected on <NVDAObjects.UIA.ListItem object at 0x03AC4C50> with extra args of {}
Traceback (most recent call last):
  File "eventHandler.py", line 152, in executeEvent
    _EventExecuter(eventName,obj,kwargs)
  File "eventHandler.py", line 92, in __init__
    self.next()
  File "eventHandler.py", line 100, in next
    return func(*args, **self.kwargs)
  File "appModules\windowsinternal_composableshell_experiences_textinput_inputapp.py", line 37, in event_UIA_elementSelected
    if obj.UIAElement.cachedClassName == "ListViewItem" and obj.parent.UIAElement.cachedAutomationID != "TEMPLATE_PART_ClipboardItemsList":
AttributeError: 'ContentGenericClient' object has no attribute 'UIAElement'

I think some extra checks for UIA are required.

josephsl added 2 commits July 19, 2018 00:45
…tent generic client is the parent of the 'selected' item. Re nvaccess#7273.

Reviewed by Mick Curran (NV Access): sometimes, a traceback that ends with 'AttributeError' on line 37 is shown, caused by the fact that sometimes IAccessible content generic client window becomes the parent of the 'selected' item (tree traversal issue). Thus catch this early by making sure the 'parent' is indeed a UIA control.
@michaelDCurran michaelDCurran merged commit e409638 into nvaccess:master Jul 19, 2018
@nvaccessAuto nvaccessAuto added this to the 2018.3 milestone Jul 19, 2018
@josephsl josephsl deleted the i7273 branch July 30, 2018 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support the new emoji panel in windows 10

3 participants