Skip to content

WIP: Use remote on Secure Desktop to send speech and braille to the user copy#18789

Closed
LeonarddeR wants to merge 5 commits into
nvaccess:masterfrom
LeonarddeR:sdRemote
Closed

WIP: Use remote on Secure Desktop to send speech and braille to the user copy#18789
LeonarddeR wants to merge 5 commits into
nvaccess:masterfrom
LeonarddeR:sdRemote

Conversation

@LeonarddeR

@LeonarddeR LeonarddeR commented Aug 23, 2025

Copy link
Copy Markdown
Collaborator

Link to issue number:

Fixes #2315

Summary of the issue:

Currently, braille doesn't work properly on secure screens because the braille display is held by the logged in copy.

Description of user facing changes:

Braille and speech will be send to the logged in copy. Apart from working braille, this has an additional benefit, namely that it is no longer necessary to copy third party braille display drivers and synth drivers to the system config. Unless I'm mistaken, this should therefore greatly reduce the need to have add-ons in the system config.

Description of developer facing changes:

The bridge module in _remoteClient has been removed as it is no longer used. I'm happy to restore it if desirable.

Description of development approach:

  1. When the secure desktop is activated from the user desktop view, a leader session is established that is scoped to the secure desktop handler. This session replaces the transport bridge used to propagate a remote leader's communication from the user copy to the secure desktop copy of NVDA.
  2. From the secure copy, when a secure desktop remote session is established, speech is set to no speech to avoid double speaking.
  3. Updated _remoteClient.transport.RemoteExtensionPoint to be able to provide a return type for deciders. Therefore we can ensure that waves and tones played on the secure desktop will only be propagated to the user copy and won't be playing within the sd copy.

Testing strategy:

  • Basic test with working speech
  • Test with braille output
  • Test with braille input
  • Test that a remote controller will still receive speech and braille from the secure desktop copy and can send keyboard keys and braille display gestures to it

Known issues with pull request:

  1. Opening the synthesizer dialog on the secure desktop copy will show that the no speech driver is selected. We might want to block this dialog when the secure desktop remote session is open.
  2. I'm not sure what to do when the user copy of NVDA freezes while the secure desktop is open. I suggested something with package acknowledgements, but that's more of a remote protocol change that probably deserves a broader scope than just this feature and may also be beneficial for other remote sessions.
  3. I've seen cases with user switching where Windows seems to mute the sound coming from the user copy. That would mean that the sd copy sends speech to the user copy, but as Windows muted the sound from that copy, there's no speech in the end. I've not found a reliable way to reproduce this.

Code Review Checklist:

  • Documentation:
    • Change log entry
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • API is compatible with existing add-ons.
  • Security precautions taken.

@LeonarddeR

LeonarddeR commented Aug 23, 2025

Copy link
Copy Markdown
Collaborator Author

@seanbudd and others, I think i'd first like to have a rough review on the approach before continuing, especially with regard to the known issues mentioned above.

Also cc @bramd

@seanbudd

seanbudd commented Aug 25, 2025

Copy link
Copy Markdown
Member

@SaschaCowley is probably the best reviewer here

@SaschaCowley

SaschaCowley commented Aug 25, 2025

Copy link
Copy Markdown
Member

@LeonarddeR did you investigate freeing the display in the user copy at secure desktop transitions?

@LeonarddeR

LeonarddeR commented Aug 25, 2025

Copy link
Copy Markdown
Collaborator Author

Sorry @SaschaCowley I didn't want to pass you, but recalled @seanbudd had done the most remote work.

@LeonarddeR did you investigate freeing the display in the user copy at secure desktop transitions?

No, but I think this has been tried in the past. The problem here lies in the reconnection part after closing the secure desktop, at least there won't be instant braille. I can prototype this though.
@bramd what do you think here?

@CyrilleB79

CyrilleB79 commented Aug 25, 2025

Copy link
Copy Markdown
Contributor

This probably means that we cannot configure consistent specific speech parameters for speech parameters on secure screens.
Probably not a problem that would block this work. But there is a use case that would not work anymore. Some people, myself included, use specific speech parameters / synth to detect if NVDA runs on secure screen or not; this seems not to be possible anymore with this architecture.

Also more generally (for speech or braille), what happens if two user sessions are currently opened with 2 NVDA running while on a secure screen?

@LeonarddeR

LeonarddeR commented Aug 25, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

This probably means that we cannot configure consistent specific speech parameters for speech parameters on secure screens. Probably not a problem that would block this work. But there is a use case that would not work anymore. Some people, myself included, use specific speech parameters / synth to detect if NVDA runs on secure screen or not; this seems not to be possible anymore with this architecture.

This should be possible when attaching a specific config profile to the secure screen. This is only possible when hacking profile triggers manually, though.

Also more generally (for speech or braille), what happens if two user sessions are currently opened with 2 NVDA running while on a secure screen?

The first secure screen initiates the control server, so the secure screen copy uses the first user's session.

@CyrilleB79

CyrilleB79 commented Aug 25, 2025

Copy link
Copy Markdown
Contributor

This probably means that we cannot configure consistent specific speech parameters for speech parameters on secure screens. Probably not a problem that would block this work. But there is a use case that would not work anymore. Some people, myself included, use specific speech parameters / synth to detect if NVDA runs on secure screen or not; this seems not to be possible anymore with this architecture.

This should be possible when attaching a specific config profile to the secure screen. This is only possible when hacking profile triggers manually, though.

Yes this remains possible with a hack, but clearly less user-friendly:

  1. Create a profile for secure screen
  2. Hack the trigger file to enable it on logon UI, so that this profile is active from secure screens opened from the current session
  3. Copy user config to system config (from general settings as usual) so that this config also applies when no user is logged
  4. For any other windows profile potentially using NVDA, copy this profile and the triggers file so that this config also applies on secure screens when those other windows profile are active.

Also more generally (for speech or braille), what happens if two user sessions are currently opened with 2 NVDA running while on a secure screen?

The first secure screen initiates the control server, so the secure screen copy uses the first user's session.

That's a good news: there is no mix.

And what happens in the following use cases:
UC1. User A opens session and launches NVDA, user B opens session (without closing user A's session) and launches NVDA, user A quits or restarts NVDA. What happens if secure screen pops up from user A's session? And from user B's session? I have not tested with today's NVDA.
UC2. User A opens session and launches NVDA, sighted user B opens their session without NVDA and locks it. Today, NVDA is not speaking on this logon screen; is it better with this PR?

@LeonarddeR LeonarddeR mentioned this pull request Aug 26, 2025
Merged
9 tasks
@LeonarddeR

LeonarddeR commented Aug 26, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

I actually don't think Braille works properly here yet, we need to set receiving braille properly on the user copy.

I created #18810 as an alternative approach. I think the current pr has too much pitfalls to deal with, especially the case where Windows mutes the user copy.

Leaving this open until comparisons / considerations of approaches are complete.

@LeonarddeR

LeonarddeR commented Aug 26, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

I'm convinced that #18810 is a better idea.

@LeonarddeR LeonarddeR closed this Aug 26, 2025
SaschaCowley pushed a commit that referenced this pull request Sep 5, 2025
@LeonarddeR
Free display on secure desktop switch (#18810)
e6557a7 
Fixes #2315

Summary of the issue:
Currently, braille doesn't work properly on secure screens because the
braille display is held by the logged in copy.

Description of user facing changes:
When NVDA is installed, it disconnects braille when entering secure
desktop and reconnects when exiting it.

Description of developer facing changes:
None

Description of development approach:
1. Added a secure desktop switch extension point to the braille handler.
2. Set a private class property on the braille display driver to ensure
the display isn't cleared, in order to make it clear that the `secure
desktop` was entered. This is communicated with an `ui.message` in
IAccessibleHandler.
3. When leaving secure desktop and auto detect was enabled, try to
reconnect to the last detected display before doing a full scan.

Testing strategy:
- [X] Tested via USB with a APH Mantis Q40
- [x] Tested via Bluetooth with a APH Mantis Q40 as the only Bluetooth
device available
- [x] Tested both automatic detect and manual display selection cases

Known issues with pull request:
1. After exiting the secure desktop, it takes some time before the
display is back in action. I must say though that with Bluetooth this
isn't considerably slower
2. When using a custom braille display driver bundled in an add-on, the
add-on needs to be available on the system copy of NVDA. This has always
been the case though, but is fixable with the approach in #18789.
3. When using manual braille display selection, reconnecting to the
display after closing the secure desktop happens on the main thread.
This can lead to a temporary freeze of NVDA, depending on how long it
takes for the driver to connect. Note however that the same freeze would
occur when starting NVDA.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Braille does not work on secure Windows screens while normal copy of NVDA is running

4 participants

@LeonarddeR @seanbudd @SaschaCowley @CyrilleB79