Skip to content

WASM: Fix issues with play, pause, and browser autoplay blocking#774

Closed
DouglasDwyer wants to merge 4 commits intoRustAudio:masterfrom
DouglasDwyer:master
Closed

WASM: Fix issues with play, pause, and browser autoplay blocking#774
DouglasDwyer wants to merge 4 commits intoRustAudio:masterfrom
DouglasDwyer:master

Conversation

@DouglasDwyer
Copy link
Copy Markdown

This PR improves the behavior of audio streams on WASM in the following ways:

  • Calling play multiple times on a Stream only calls resume on the underlying AudioContext a single time, and only spawns background audio callbacks once. Previously, calling play from Rust multiple times in succession would spawn multiple closure instances and lead to stuttering audio.
  • Browsers block audio playback until the user interacts with the webpage, and force the AudioContext to remain in the Suspended state. This is problematic for cpal's WASM backend, because from the Rust side there is no way to determine whether audio playback has begun. This PR causes the Stream to read and discard audio frames while the context is blocked, so it's as if the audio stream is always playing (but muted until the user interacts) once play is called. Previously, the audio would simply hang if the user had not yet interacted with the page.
  • I've added support for running cpal with the +atomics target feature enabled. This is accomplished by adding a secondary Javascript-side array into which frames are copied before they are played, since you cannot send frames to Javascript from a SharedArrayBuffer directly.

I'm happy to make any improvements necessary to get this PR merged! I would like to see all of these behaviors available in cpal, because they are the best fit for my use case.

@est31
Copy link
Copy Markdown
Member

est31 commented Apr 14, 2023

Thanks for your PR! I think the play changes are good at least, and having SharedArrayBuffer support is also really neat.

Previously, the audio would simply hang if the user had not yet interacted with the page.

Can you explain why you prefer the new behaviour?

@DouglasDwyer
Copy link
Copy Markdown
Author

DouglasDwyer commented Apr 14, 2023

@est31 sure! Let me emphasize here that this new behavior should be non-breaking: it is a strict superset of the old behavior, and thus backwards compatible.
To reiterate, the current behavior is that any audio stream started before the user interacts with the page simply hangs. This remains true even after the user interacts; the audio stream continues to hang, and it's necessary to call play again to make it start. This means that any current users on WASM need to wait until a user interaction to call play on the stream.
These changes just extend this by allowing play to be called, and work correctly, before a user interacts. Consumers of cpal can still wait until user interaction to call play (which was previously required), but this gives them an extra option: they can just call play immediately and everything works.
For my personal use case, I'd like to embed cpal into a cross-platform game. It's much easier for me to begin an audio stream on startup, and not worry about web-specific nonsense like automute :)

@est31
Copy link
Copy Markdown
Member

est31 commented Apr 14, 2023

@DouglasDwyer my question was more why you think it's a good idea, not why it's allowed for us to do it. I agree that we are allowed to do it, but I want to hear arguments why the new behaviour is better than the old one. Why is it bad if it hangs? What does hanging mean? can you still interact? Why is it better that it is muted but plays in the background?

@DouglasDwyer
Copy link
Copy Markdown
Author

Okay. My argument above was meant to state that I think it's a good idea because it gives users an additional option: they can have the audio not hang if that's what they want. Surely having that option is better than not having it?

Anyway, let me try and answer your questions.

What does hanging mean? Can you still interact?

The audio stream is not started, and no audio frames are consumed from the output callback. The application otherwise functions normally.

Why is it bad if it hangs?

There's no way to detect the hanging behavior from Rust; currently, the play call still returns Ok(()). This strikes me as confusing - Rust crates typically use Results to indicate success, so I would expect that Ok(()) means that the stream was started.

Further, having the stream hang could lead to audio buffer overruns. If audio frames are being streamed into the application, but the output callback is never invoked, then these frames could accumulate and cause an out-of-memory issue.

Why is it better that it is muted and plays in the background?

Because as an end user, I expect that a successful call to play starts the audio stream, consuming audio frames. On other platforms, this is how play behaves. It's cumbersome that WASM functions differently. I think that playing (muted) more closely matches how cpal works on native platforms, and I think that cross-platform consistency is highly desirable.

@Ralith
Copy link
Copy Markdown
Contributor

Ralith commented Apr 15, 2023

As a user, I'd rather (web) applications whose audio I can't hear not be spending potentially large amounts of CPU time rendering said inaudible audio.

@DouglasDwyer
Copy link
Copy Markdown
Author

DouglasDwyer commented Apr 15, 2023

@Ralith, an alternative solution that I think would be acceptable would be for play to return Err if the user hasn't yet interacted with the page. That way, at least, the Rust application could respond by dropping audio frames. Unfortunately, it does not appear to be possible to synchronously detect a stream failing to start - the Javascript AudioContext's resume function returns an asynchronous promise.

@grantshandy
Copy link
Copy Markdown

I think this PR is extremely valuable for the users of cpal in WASM. My game engine starts on page load and the audio does not work without the changes included in this branch, even with user interaction. I believe this is the functionality users of the library (especially ones unfamiliar with the JS audio API) expect in the first place. Thank you for the great work here 👍🏽.

@Ralith mentioned issues with unused cpu usage while the audio is muted. I'd like to add that (from what I can) tell it would only be rendering silent audio in the second or two between the user opening the tab and clicking on the page.

@Davidster
Copy link
Copy Markdown

Davidster commented Jun 14, 2023

+1 for this PR getting merged. it should help me get multi-threaded audio loading working on the web

@Nek
Copy link
Copy Markdown

Nek commented Jun 15, 2023

+1 too as I was struggling with the same problems in WASM world.

@volodalexey
Copy link
Copy Markdown

Currently bevy 0.11.0 uses this library for sound.
Mobile browsers can not handle current implementation in WASM see my issue bevyengine/bevy#9211.

+1
I would like to see this merged!

@cybersoulK
Copy link
Copy Markdown

i just got this issue as well. i require +atomics for something else, but right now it's crashing on both chrome and firefox

@DouglasDwyer
Copy link
Copy Markdown
Author

Maintainers - please let me know if there is anything more that I can do to push this PR forward!

@cybersoulK
Copy link
Copy Markdown

i can confirm this PR solved it

@est31
Copy link
Copy Markdown
Member

est31 commented Sep 26, 2023

I'm still not really happy about the hang removal changes. But outside of that, I think the general idea of the PR looks good. I need to give it a closer look though.

@kettle11
Copy link
Copy Markdown
Contributor

kettle11 commented Jan 9, 2024

Maintainers (@est31?) can we find a way to get this merged? cpal crashing when atomics are enabled is blocking getting Bevy multithreaded on web.

@melody-rs
Copy link
Copy Markdown
Contributor

Any updates on this PR?

@Davidster
Copy link
Copy Markdown

Davidster commented Feb 23, 2024

@Speak2Erase looks like we might be good to go since the recent changes from @kettle11: #837! I will test on my project tonight. Thank you!!

Update: it works 😄

@est31
Copy link
Copy Markdown
Member

est31 commented Feb 24, 2024

Given #837 I'm closing this, but open for any of the non controversial changes to be submitted in a PR. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants