Selective UIA event handling by LeonarddeR · Pull Request #11214 · nvaccess/nvda

LeonarddeR · 2020-05-27T17:51:32Z

Link to issue number:

Closes #11209
fixes #11077
Improves #11002
Follow up of #8742

Summary of the issue:

NVDA is listening globally to many UIA elements. This causes a major performance hit when many events are received (#11109)

Description of how this pull request fixes the issue:

This pr introduces an easier method for event registration by means of EventHandlerGroups. This feature was introduced in IUIAutomation6. I wrote a FakeEventHandlerGroup class to mimic this behavior on older versions of UIA. In short, this allows us to request for automation/ notification and property events in one call instead of three on newer versions of UIA.
For some events, we still keep listening globally. However, for many of them, I switched to local event registration, with that I mean only to objects of interest. Currently, these are the focus object and its ancestors.
When changing focus, we register for events fired at the focus object and its ancestors. This includes all focus changes, including these for non native UIA elements. This includes focus changes for IAccessible objects as well.

Testing performed:

@codeofdusk confirmed that UIA freezes after a large number of events is received #11002 has improved by this
I myself can confirm that the Nuget Package manager as well as other aspects of Visual Studio are much snappier.

Known issues with pull request:

@josephsl noted that announcements for modern input are somewhat broken, i.e. the announcement that the emoji panel is closed. I haven't yet looked into this. I'd like to have a basic review about the prototype and its performance improvements first.

Change log entry:

Bug fixes
- Major performance improvements in Microsoft Visual Studio and other UIAutomation based applications. (Restrict most event processing to objects of interest to the user #11077, UIA: selective event registration #11209)

LeonarddeR · 2020-05-27T17:52:19Z

Here is a try build

lukaszgo1 · 2020-05-27T18:18:58Z

@LeonarddeR I can confirm that the latest try build fixes my issues under Windows 7.

josephsl · 2020-05-27T20:03:23Z

Hi,

Regarding emoji panel and other modern input features, the fundamental issue to consider is focus change. A few months ago I tried to resolve overlay focus issues by faking a focus change event when emoji panel opens, but this resulted in NVDA's notion of "focus" being trapped once the panel was closed i.e. previously focused window isn't announced once the panel closes. In theory, this issue would also impact touch keyboard support on Windows 8 and later if it also suffers from focus trap issue, along with a possible impact on UIA notification event handling for classic Edge. I'm looking into resolving emoji panel issue from the app module side and to test this PR at the same time.

Thanks.

josephsl

It turns out focus change handler will add or remove events every time focus changes. This is more so when moving focus to a different element inside the same foreground window/application. This might be something to optimize later.

Apart from modern input support not working well, this looks promising. Thanks.

josephsl · 2020-05-27T22:58:24Z

Hi,

Found a way to restore emoji panel support - local event group must be added for input panel header. This means this PR must block #10378 - I'll package that PR and the follow-up one together.

Thanks.

codeofdusk · 2020-05-28T02:21:28Z

Found a bug:

In 21H1 (open) console with UIA, press alt+space to open the application menu.
Press e to open the edit menu.
Press p to paste.

Expected: auto-read continues functioning after this, with no need to switch windows.
Actual: a focus change is necessary to bring back autoread.

LeonarddeR · 2020-05-28T06:31:29Z

@josephsl wrote:

It turns out focus change handler will add or remove events every time focus changes. This is more so when moving focus to a different element inside the same foreground window/application. This might be something to optimize later.

This is currently intentional. #8742 implemented per foreground event registration, but as @jcsteh noted in #8742 (comment) , it wouldn't fix property event bombarding within the same foreground window.

@codeofdusk wrote:

Actual: a focus change is necessary to bring back autoread.

This is really weird. It looks like the local event registration isn't triggered when focus is back on the terminal window. I will investigate this.

josephsl · 2020-05-28T06:39:38Z

Hi, I see. By the way, for modern input, one must first remove local events before adding local event handler group, otherwise event groups will get large, leading to NVDA announcing things multiple times. This must be done for item selected and window open event – the former for emoji panel, the latter for dictation and hardware keyboard input, respectively. Thanks.

LeonarddeR · 2020-05-28T06:53:48Z

Ugh, so UIA allows you to register one handler to elements multiple times? That's horrific! I think we need something like eventHandler.requestEvents for UIA. If I'm correct, for modern input, we only need the name change anyway.

josephsl · 2020-05-28T07:18:40Z

Hi, yes for emoji panel, item selected event may be needed for hardware keyboard input. Eventually we need to do something about clipboard history context menu, but that’s beyond this PR. Thanks.

LeonarddeR · 2020-05-28T08:05:03Z

item selected event is listened to globally, as it is also required for search results.

LeonarddeR · 2020-05-28T16:37:13Z

@codeofdusk It looks like the issue with the UIA implementation of cmd is exactly the same issue as that requires us to implement focusRedirect. I will investigate a fix.

codeofdusk · 2020-06-01T07:25:00Z

More generally, I'd think you'd want to find a method of dealing with focus changes through redirects...

LeonarddeR · 2020-06-04T06:07:05Z

I will investigate whether it is possible to hide stuff behind a feature flag, so the initial version of this pull request can hopefully be merged sooner, with may be some issues remaining.

josephsl · 2020-06-27T18:52:25Z

Hi,

By the way, in order to properly test this, one must:

Change selective UIA events setting from Advanced settings panel.
Reload UIA handler thread as follows:

Open Python console, then:
import UIAHandler
UIAHandler.terminate(); UIAHandler.initialize()

The import statement is not needed if UIA handler was already imported.

Thanks.

josephsl · 2020-06-27T21:11:11Z

Hi,

My results so far:

Windows 7 and 8.1: no difference observed, although this was with no UIA intensive apps such as Microsoft 365 installed.
Windows 10 versions 1507 through 1709: no difference I can observe.
Windows 10 Version 1803: no hardware keyboard suggestions are announced when selective UIA events is on.
Windows 10 Version 1809 and later: no announcement of emoji panel categories when pressing tab from emoji panel.

Scenarios tested:

Opening, searching, and closing Start menu/screen.
Opening and closing Control Panel.
Opening, navigating, and closing Windows 10 Settings app.
File Explorer.
Modern input.

For hardware keyboard suggestions and emoji panel, one workaround is manually registering various UIA events for modern input apps. This does allow hardware keyboard suggestions to be announced and emoji categories to be heard when pressing Tab from emoji panel in 1809 and later.

Thanks.

josephsl · 2020-06-27T21:41:28Z

Hi,

Another thing to consider: in Windows 10's Start menu, after selecting a search result or moving focus away from the screen itself, suggestions closed sound is not heard (this is the case with at least 20H2). This is caused by the fact that when a search result is selected, focused item isn't search box anymore (search box is responsible for raising UIA controller for event). I'm designing a potential fix from my end.

Thanks.

michaelDCurran · 2020-06-28T23:11:28Z

@josephsl What specific UIA events are required for hardware keyboard and emoji selection support that was broken with this pr?

josephsl · 2020-06-28T23:26:14Z

Hi, “window open” and “item selected”. Thanks.

LeonarddeR · 2020-06-29T06:56:00Z

Personally, I think we shouldn't make these issues block this pr from going through, as the new behavior isn't the default. Also, in 2004 it works fine for me, at least when pressing escape to close the start menu.

josephsl · 2020-06-29T07:10:51Z

Hi, that I can confirm that. I think for now it would be preferable to let people test this for a long time and come up with edge cases as time goes. Thanks.

zstanecic · 2020-06-29T07:18:37Z

Yep, I agree. The more the testing cases from master,the more the possiblility for fixing issues.

zstanecic · 2020-07-02T18:16:28Z

How is situation with this?

LeonarddeR · 2020-07-03T07:09:52Z

i've been using this for the last month or so on a Windows 10 machine in production, as Visual Studio is almost unbearable without it.
Personally, if @michaelDCurran agrees with the current code, I'd appreciate a merge early in the 2020.3 alpha cycle so it can prove itself.

zstanecic · 2020-07-03T07:40:58Z

I am really for merging this, It really proves the performance boost

josephsl · 2020-07-03T07:44:26Z

Hi, or rather, gives us plenty of time to test edge cases and come up with proper workarounds and could serve as a stepping stone for a bigger things. I second early merge. Thanks.

michaelDCurran · 2020-07-03T07:46:53Z

I will look into reviewing / merging this on Monday.

codeofdusk · 2022-01-29T00:18:24Z

This feature has been disabled by default for several releases.

At this stage, what is blocking enabling by default?

Link to issue number: Partial mitigation for #11002. Follow-up of #11214 and #13297. Summary of the issue: NVDA 2020.3 introduced support for selective registration for UIA events based on the currently-focused provider, but this is disabled by default due to bugs in the Windows 10 task manager and emoji panel. These bugs have been fixed in Windows 11 SV2. Description of how this pull request fixes the issue: By default, enable selective UIA event registration on Windows 11 Sun Valley 2 (22H2). Testing strategy: Enable by default in master snapshots to allow for wider testing.

LeonarddeR and others added 4 commits May 27, 2020 08:12

Prototype

ba3ef89

Fix unhandled error when comparing elements

cae9c85

Fix windows 7

76a3ecf

Linting

cd4a325

LeonarddeR added performance api/uia Feature or bug requires an understanding of UIA labels May 27, 2020

LeonarddeR requested review from josephsl and michaelDCurran May 27, 2020 17:51

josephsl reviewed May 27, 2020

View reviewed changes

josephsl mentioned this pull request May 27, 2020

Emoji panel and clipboard history: locate correct items if controls are different due to recent changes #10378

Closed

LeonarddeR mentioned this pull request May 28, 2020

Investigate accessibility bugs reported by users NuGet/Home#9393

Closed

LeonarddeR mentioned this pull request Jun 3, 2020

Visual Studio IntelliSense overly verbose #8336

Closed

LeonarddeR added 2 commits June 4, 2020 18:16

Start with config flag

3fbdea0

Start with flag in dialog

43a02f3

This comment has been minimized.

Sign in to view

LeonarddeR added 2 commits June 5, 2020 18:55

Finish integration in gui

fe92ec5

Linter

cc3cba5

Update userdocs

bdab9b7

feerrenrut assigned michaelDCurran Jul 1, 2020

LeonarddeR requested a review from michaelDCurran July 3, 2020 07:10

michaelDCurran approved these changes Jul 6, 2020

View reviewed changes

michaelDCurran merged commit f15d3c9 into nvaccess:master Jul 6, 2020

nvaccessAuto added this to the 2020.3 milestone Jul 6, 2020

michaelDCurran added a commit that referenced this pull request Jul 6, 2020

Update what's new for pr #11214

e1c72f3

MarcoZehe mentioned this pull request Jul 8, 2020

Context menus in Firefox read inconsistently with selective UIA handling enabled #11354

Closed

codeofdusk mentioned this pull request Feb 1, 2022

Prepare for selective UIA event registration by default #13297

Merged

5 tasks

codeofdusk mentioned this pull request Aug 16, 2022

Selectively register for UIA events by default on SV2 #14018

Merged

5 tasks

This was referenced Nov 27, 2023

UIA freezes after a large number of events is received #11002

Closed

Avoid UIA event flooding in NvDA by moving UIA event handler COM objects into c++ #14888

Merged

cary-rowen mentioned this pull request Jun 11, 2025

NVDA freezes when typing in Chrome's Omnibox with the UiaProviderWin feature, triggered by setting UI Automation event registration to 'Selective' #18239

Closed

codeofdusk mentioned this pull request Jul 21, 2025

Replace nvda_dmp with an integrated implementation #18480

Merged

5 tasks

LeonarddeR deleted the selectiveUIAEventHandling branch August 23, 2025 06:28

Uh oh!

Conversation

LeonarddeR commented May 27, 2020

Link to issue number:

Summary of the issue:

Description of how this pull request fixes the issue:

Testing performed:

Known issues with pull request:

Change log entry:

Uh oh!

LeonarddeR commented May 27, 2020

Uh oh!

lukaszgo1 commented May 27, 2020

Uh oh!

josephsl commented May 27, 2020

Uh oh!

josephsl left a comment

Choose a reason for hiding this comment

Uh oh!

josephsl commented May 27, 2020

Uh oh!

codeofdusk commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LeonarddeR commented May 28, 2020

Uh oh!

josephsl commented May 28, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LeonarddeR commented May 28, 2020 via email

Uh oh!

josephsl commented May 28, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LeonarddeR commented May 28, 2020 via email

Uh oh!

LeonarddeR commented May 28, 2020

Uh oh!

codeofdusk commented Jun 1, 2020

Uh oh!

LeonarddeR commented Jun 4, 2020

Uh oh!

This comment has been minimized.

josephsl commented Jun 27, 2020

Uh oh!

josephsl commented Jun 27, 2020

Uh oh!

josephsl commented Jun 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelDCurran commented Jun 28, 2020

Uh oh!

josephsl commented Jun 28, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LeonarddeR commented Jun 29, 2020

Uh oh!

josephsl commented Jun 29, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zstanecic commented Jun 29, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zstanecic commented Jul 2, 2020

Uh oh!

LeonarddeR commented Jul 3, 2020

Uh oh!

zstanecic commented Jul 3, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josephsl commented Jul 3, 2020 via email • edited by feerrenrut Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelDCurran commented Jul 3, 2020 via email

Uh oh!

codeofdusk commented Jan 29, 2022

Uh oh!

Reviewers

Assignees

Labels

codeofdusk commented May 28, 2020 •

edited

Loading

josephsl commented May 28, 2020 via email •

edited by feerrenrut

Loading

josephsl commented May 28, 2020 via email •

edited by feerrenrut

Loading

josephsl commented Jun 27, 2020 •

edited

Loading

josephsl commented Jun 28, 2020 via email •

edited by feerrenrut

Loading

josephsl commented Jun 29, 2020 via email •

edited by feerrenrut

Loading

zstanecic commented Jun 29, 2020 via email •

edited by feerrenrut

Loading

zstanecic commented Jul 3, 2020 via email •

edited by feerrenrut

Loading

josephsl commented Jul 3, 2020 via email •

edited by feerrenrut

Loading