Skip to content

Able to toggle imageDesc while screen curtain is enabled#19178

Merged
seanbudd merged 1 commit into
nvaccess:masterfrom
tianzeshi-study:fix-imageDesc-with-curtain
Nov 6, 2025
Merged

Able to toggle imageDesc while screen curtain is enabled#19178
seanbudd merged 1 commit into
nvaccess:masterfrom
tianzeshi-study:fix-imageDesc-with-curtain

Conversation

@tianzeshi-study

Copy link
Copy Markdown
Contributor

Link to issue number:

Fixes #19157

Summary of the issue:

Unable to toggle AI Image Captioner while Screen Curtain is enabled

Description of user facing changes:

Able to toggle imageDesc while screen curtain is enabled

Description of developer facing changes:

None

Description of development approach:

None

Testing strategy:

Enable screen curtain, and toggle imageDesc via settings panel or gesture(need to assign ). Observe that toggle imageDesc script will work correctly.

Known issues with pull request:

None

Code Review Checklist:

  • Documentation:
    • Change log entry
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • API is compatible with existing add-ons.
  • Security precautions taken.

@tianzeshi-study tianzeshi-study requested a review from a team as a code owner November 6, 2025 05:03
@seanbudd seanbudd merged commit fa3dcfa into nvaccess:master Nov 6, 2025
27 of 29 checks passed
@github-actions github-actions Bot added this to the 2026.1 milestone Nov 6, 2025
derekriemer pushed a commit to derekriemer/nvda that referenced this pull request Nov 6, 2025
)

Fixes nvaccess#19157
Summary of the issue:

Unable to toggle AI Image Captioner while Screen Curtain is enabled
Description of user facing changes:

Able to toggle imageDesc while screen curtain is enabled
seanbudd added a commit that referenced this pull request Jan 9, 2026
SaschaCowley pushed a commit that referenced this pull request Jan 11, 2026
Reverts:
- #18475
- #19036
- #19024
- #19055
- #19057
- #19178
- #19243
- #19327
- Partial revert: #19342

### Issues fixed
Fixes #19298 

### Issues reopened
Reopens #16281

### Reason for revert / Can this PR be reimplemented? If so, what is
required for the next attempt

The current implementation of AI image descriptions yields low quality
captions from a 3 year old model (see #19298).
The current implementation also requires using numpy, which hogs RAM,
slows initialization, and increases the weight of the installer.
An attempt was made to convert this to C++ using WinML and Windows ONNX
runtimes as per #18662.
This would have removed numpy, and improved flexibility for using
different models in the future.
Unfortunately, this was not found to be feasible, as ONNX C++ fails to
work via 64bit emulation on ARM
(microsoft/onnxruntime#15403).

This means we have the following options for image descriptions:

1. Continue to use the python onnxruntime, and accept the RAM and
storage hits. Instead, improve the quality of the captioner with better
models such as
[git-base-coco](https://huggingface.co/microsoft/git-base-coco) or
[blip2](https://huggingface.co/Salesforce/blip2-opt-2.7b-coco).
2. Wait until MS builds ARM64EC into C++ ONNX (blocked by
microsoft/onnxruntime#15403)
3. Attempt to build our own fork of ONNX with ARM64EC
4. Build a separate ARM native installer of NVDA, offer as an
alternative to allow for ARM devices to do image descriptions with
numpy.
5. Release the feature on C++ without support for ARM devices.

All of these options require a significant amount of work.
As such, sadly this feature is not ready for a stable release.

Instead this code will be moved to a feature branch, until ONNX C++
matures such as fixing
microsoft/onnxruntime#15403.
Additionally, ONNX C++ runtimes are only available through the
experimental 2.0 version of the Windows App SDK, and requires you to
build your own headers from it.
I think this feature will be blocked until
microsoft/onnxruntime#15403 is implemented and
the 2.0 version of the Windows App SDK becomes stable.
Future re-implementations should also consider using higher quality,
more modern models.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to toggle AI Image Captioner while Screen Curtain is enabled

2 participants