Skip to content

[WIP] A11Y for blind players#3207

Closed
caturria wants to merge 181 commits intoHarbourMasters:developfrom
caturria:actor-accessibility-experiments
Closed

[WIP] A11Y for blind players#3207
caturria wants to merge 181 commits intoHarbourMasters:developfrom
caturria:actor-accessibility-experiments

Conversation

@caturria
Copy link

@caturria caturria commented Sep 17, 2023

This module provides accessibility features which can be used by blind players to navigate the game world independently. It is implemented by way of a "virtual actor" system. Virtual actors are spatialized sound sources which can be placed at arbitrary locations in a given scene. They can map directly to real actors or be entirely custom objects which identify a location to the player via sound.
Each virtual actor includes a policy which defines its behaviour (which sound effect it plays, how often, how far away it can be heard from, etc).
This mechanism is almost entirely decoupled from the game; there is just enough boiler plate to allow it to access information about actors and include extra audio in the game's output.
In addition to actors, the module also reports the location of specific terrain features (such as walls, hills, ledges and water), also via the virtual actor system.
Due to the limitations imposed on the number of concurrent sounds supported by the game's sound engine, a custom engine was developed (on top of MiniAudio), as well as a tool which extracts all of the sounds from the game and packs them into a new OTR file. Once extracted, the module can freely render the game's own sound effects without interfering with normal operation. It can also render the game's raw audio samples.
To maximize portability, MiniAudio is heavily stripped down: in particular, its I/O and threading features are excluded. Instead of opening a secondary audio handle, it simply mixes its audio in with the game's own audio output before it is sent out to Libultraship.
At this time, the game is playable up to the end of the Deku Tree using only sounds. Several behaviours have been defined for actors in other areas, however more are required in order to make subsequent content fully playable.
To use this module, do the following:

  1. Build as usual.
  2. set cvar gA11yAudioInteraction to 1.
  3. set gA11yTTS to 1.
  4. set gExtractSfx to 1.
  5. Close and reopen the game. You'll be asked to please wait as the sound effects are extracted. gExtractSfx will be automatically cleared when the process completes.
  6. Once done, close and reopen once more (we're hoping to streamline this in a future update).
  7. Load the game. Walk around. You'll hear sounds for many actors and terrain features, positioned relative to Link.
    we decided to make this public now as we have finished the first major hurdle: terrain
    detection. With terrain implemented as well as a powerful framework for defining accessible behaviours for actors, much of the project going forward will be adding definitions to new actors and brainstorming solutions to specific puzzles and minigames which will pose unique challenges.
    If you would like to get involved, please get in touch with us.

Build Artifacts

caturria and others added 30 commits May 22, 2023 19:51
@briaguya0 briaguya0 added the do not merge Not ready or not valid changes label Feb 17, 2024
@briaguya0
Copy link
Contributor

I think it's worth discussing a path forward for this. This PR is gigantic (miniaudio.h is ~91k lines, which means even without that this is an ~6k line PR), and quite far reaching.

I see 2 paths forward for this:

  1. See how this can be split up into more self contained and reviewable PRs.
  2. Have a fork be the new permanent home for this effort.

I personally feel that option 2 may be the better choice here. GitHub supports forking repos within the same org, so we could even discuss it being an "official" fork.

I'm looking forward to hearing what others think about this.

@serprex serprex mentioned this pull request Apr 16, 2025
// actors. Manually reset by aim assist provider.
f32 pitch; // Used to report whether Link is aiming higher or lower than the actor.
u8 frequency; // How often the sound will be played. Lower frequencies indicate that Link's vertical aim is
// closer to the actor. } aimAssist;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// closer to the actor. } aimAssist;
// closer to the actor.

@serprex
Copy link
Contributor

serprex commented Jun 2, 2025

@briaguya-ai I've created #5435 & have attempted to simplify the change / cleanup the code, while at same time extending cues into adult phase of game. It only depends on miniaudio as a submodule. I think this PR can be closed in favor of mine & I'd like to resume the discussion of a path forward here

I do think there are some blockers to merging, particularly the generation step needs to be improved (ideally it'd be a button, rather than having this reset-rewrite dance)

@briaguya0
Copy link
Contributor

closing as work has continued in #5435

@briaguya0 briaguya0 closed this Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Not ready or not valid changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants