Skip to content

New package: @remotion/whisper-web#5043

Merged
JonnyBurger merged 89 commits intoremotion-dev:mainfrom
hunxjunedo:main
May 14, 2025
Merged

New package: @remotion/whisper-web#5043
JonnyBurger merged 89 commits intoremotion-dev:mainfrom
hunxjunedo:main

Conversation

@hunxjunedo
Copy link
Copy Markdown
Contributor

@hunxjunedo hunxjunedo commented Mar 21, 2025 •
edited by samohovets
Loading

fixes #5023

/claim #5023

todo:

  • scope models in storeFS
  • Make callbacks work in transcribe()
  • Make return value work
  • getLoadedModels() API
  • deleteModels() API
  • Reject if not enough space is available to load model
  • Remove all console.log's
  • Question 30 second audio limit
  • Is trimmed to 30 second audio on client side?
  • Document cross origin isolation restriction
  • Add comment why we added cross origin isolation restriction
  • Get return value as JSON
  • Is this abortable?
  • Do something with storage estimate
  • Scope modelState
  • Circular dependency warning when calling transcribe()
  • Document new methods
  • Need to reset if calling multiple times - state not shared
  • setTimeout of 100ms intended?
  • Make API for determining if model can be downloaded
  • Document that it cannot be called multiple times at once
  • Document onUpdate (maybe rename it)
  • "en" is hardcoded
  • Add a logger to allow to see the verbose logging

@vercel
Copy link
Copy Markdown
Contributor

vercel bot commented Mar 21, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
bugs ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 14, 2025 5:34am
remotion ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 14, 2025 5:34am

@hunxjunedo hunxjunedo marked this pull request as draft March 21, 2025 20:06
@hunxjunedo hunxjunedo changed the title new packages @remotion/whisper-wasm new package @remotion/whisper-wasm Mar 22, 2025
hunxjunedo
hunxjunedo commented Mar 22, 2025
View reviewed changes
@vercel vercel bot temporarily deployed to Preview – remotion March 24, 2025 12:26 Inactive
@samohovets
Copy link
Copy Markdown
Member

samohovets commented May 7, 2025 •
edited
Loading

  • Review docs once again for consistency
  • Why we don't support medium and large models? Test if it works and extend supported model list because they're not supported in the WASM implementation

@hunxjunedo
Copy link
Copy Markdown
Contributor Author

hunxjunedo commented May 7, 2025

  • Review docs once again for consistency
  • Why we don't support medium and large models? Test if it works and extend supported model list

We don't support them because of performance constraints, because the original wasm implementation of whisper-cpp doesn't support them. I think that'd be a long way, may require optimizing the cpp code in emscripten.cpp

@samohovets
Copy link
Copy Markdown
Member

samohovets commented May 9, 2025

Thanks for letting me know @hunxjunedo, just tested with medium.en model and got this error.

Keeping it here in case someone asks why larger models are not supported:

Aborted(Cannot enlarge memory arrays to size 2113515520 bytes (OOM). Either (1) compile with -sINITIAL_MEMORY=X with X higher than the current value 2097152000, (2) compile with -sALLOW_MEMORY_GROWTH which allows increasing the size at runtime, or (3) if you want malloc to return NULL (0) instead of this abort, compile with -sABORTING_MALLOC=0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JonnyBurger JonnyBurger Awaiting requested review from JonnyBurger

Assignees

@samohovets samohovets

Labels

🙋 Bounty claim

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

New package @remotion/whisper-wasm

3 participants

@hunxjunedo @samohovets @JonnyBurger