Skip to content

feat: implement captions editor integration#364

Merged
Brendonovich merged 17 commits intoCapSoftware:mainfrom
onyedikachi-david:feature/captions-editor-integration
May 26, 2025
Merged

feat: implement captions editor integration#364
Brendonovich merged 17 commits intoCapSoftware:mainfrom
onyedikachi-david:feature/captions-editor-integration

Conversation

@onyedikachi-david
Copy link
Copy Markdown
Contributor

/claim #361
Fixes #361

- Add CaptionsTab component with comprehensive caption management UI

- Implement caption store with state management using SolidJS

- Add model download and management functionality

- Integrate FFmpeg subtitle support

- Add export settings with subtitle toggle

- Implement real-time caption display in video player

- Add caption styling options (font, size, color, position)

- Add caption segments management with timestamps

- Implement localStorage backup for captions data
@richiemcilroy
Copy link
Copy Markdown
Member

hey @onyedikachi-david - thanks so much for the submission.

it doesn't seem to load for me, I get:


  thread 'main' panicked at /Users/macbookuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/cmake-0.1.54/src/lib.rs:1119:5:

  failed to execute command: No such file or directory (os error 2)
  is `cmake` not installed?

  build script failed, must exit now
  stack backtrace:
     0:        0x1029b6c1c - std::backtrace_rs::backtrace::libunwind::trace::h43235ac629484ce8
                                 at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
     1:        0x1029b6c1c - std::backtrace_rs::backtrace::trace_unsynchronized::h3bc78969d500b9fc
                                 at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
     2:        0x1029b6c1c - std::sys::backtrace::_print_fmt::h66db558f95871955
                                 at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/backtrace.rs:66:9
     3:        0x1029b6c1c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h6e25c13836847e70

Also, we need the captions to be baked into the video frames that come from the renderer. Would you be able to check?

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

Was the error during the export

@richiemcilroy
Copy link
Copy Markdown
Member

@onyedikachi-david it was on startup of the desktop app

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

@onyedikachi-david it was on startup of the desktop app

I am assuming this, not sure:

                    let status = Command::new("ffmpeg")
                        .args([
                            "-i", &output_path.to_string_lossy(),
                            "-i", &srt_path.to_string_lossy(),
                            "-c:v", "copy",
                            "-c:a", "copy",
                            "-c:s", "mov_text",
                            "-metadata:s:s:0", "language=eng",
                            "-movflags", "+faststart",
                            "-map", "0:v",
                            "-map", "0:a",
                            "-map", "1:0",
                            "-y",
                            &temp_output.to_string_lossy(),
                        ])
                        .status()
                        .map_err(|e| {
                            tracing::error!("FFmpeg command failed: {}", e);
                            format!("Failed to run FFmpeg: {}", e)
                        })?;

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

working quite well here:

image

Give me some time to find out what is wrong

@richiemcilroy
Copy link
Copy Markdown
Member

right now you are:

  • in the editor, rendering captions on top of canvas with html
  • in the export, adding a subtitle track to the mp4

we need the subtitles to be rendered in the GPU as part of the actual frame. Not a subtitle track, but a physical implementation in the frame. This would allow us to change the text size, text appearance etc in the editor.

You can use a crate like glyphon to help with this.

Signed-off-by: David Anyatonwu <davidanyatonwu@gmail.com>
- Refactor CaptionsTab component to improve state management and UI responsiveness.
- Introduce new caption settings and segment management features.
- Update caption rendering logic to ensure proper display and interaction.
- Remove deprecated code related to captions synchronization with project settings.
- Enhance user experience with improved caption styling options and segment operations.

Signed-off-by: David Anyatonwu <davidanyatonwu@gmail.com>
@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

right now you are:

  • in the editor, rendering captions on top of canvas with html
  • in the export, adding a subtitle track to the mp4

we need the subtitles to be rendered in the GPU as part of the actual frame. Not a subtitle track, but a physical implementation in the frame. This would allow us to change the text size, text appearance etc in the editor.

You can use a crate like glyphon to help with this.

Done, please review. Was a tough one.

onyedikachi-david and others added 4 commits March 18, 2025 19:40
Signed-off-by: David Anyatonwu <davidanyatonwu@gmail.com>
- Update default font in captions settings to "System Sans-Serif".
- Introduce new RgbInput component for color selection in caption settings.
- Implement scroll position preservation in the CaptionsTab for better user experience.
- Refactor caption settings management to improve synchronization with project data.
- Enhance UI for caption segments, including improved layout and interaction for adding and editing segments.

Signed-off-by: David Anyatonwu <davidanyatonwu@gmail.com>
…se of std::process::Command to call ffprobe and ffmpeg
@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

Screen.Recording.2025-03-19.at.10.46.32.mp4

@richiemcilroy Here is a demo

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

Hi, @richiemcilroy, have you had time to take a look.

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

What's up with this @Brendonovich

@Brendonovich
Copy link
Copy Markdown
Contributor

Apologies for not looking at this for a while. I'm going to merge so we can start playing with the model downloading and transcription. Got a few things I want to do before releasing this as a feature including redoing our rendering backend.

@Brendonovich Brendonovich merged commit 09c449c into CapSoftware:main May 26, 2025
5 of 6 checks passed
@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

Apologies for not looking at this for a while. I'm going to merge so we can start playing with the model downloading and transcription. Got a few things I want to do before releasing this as a feature including redoing our rendering backend.

It's all good.

@onyedikachi-david
Copy link
Copy Markdown
Contributor Author

Would love to help, paid or not, but I see that most issues/ are self assigned, do you allow contributions to such issues @Brendonovich

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subtitle/caption generation in Studio Mode recordings

3 participants