I have wrapped the shared texture handle to VideoFrame object successfully.

However, several major issue blocks:

1. The API needs to be in main world of renderer process (which I don't see a working sample in Electron), and VideoFrame doesn't works well with contextBridge. Currently I disabled contextIsolation and enabled nodeIntegration for testing.

2. The VideoFrame wants to take ownership of the handles we provided, on macOS and Linux this is just fine, macOS can retain the IOSurface (ref +1) and Linux calls dup on dma buf fd. But on Windows is can't DuplicateHandle because there's no handle to source process, we don't even know the pid either. Currently I slightly modified the DXGIHandle to prevent it takes ownership.

3. The generated VideoFrame is a GPU frame, when I tried to render to canvas, import to WebGPU, currently Chromium always tries to map to CPU memory to read pixels, and fail.

4. There's a zero copy path, which needs a SharedImage and now only supports NV12 for unknown reason. The SharedImage needs to create with a GPU Service, which seems hard to get one from anyehere.

5. I expect more Chromium patches, and these might not be able to merge to upstream.

Update:

I now got the SharedImageInterface from SharedGpuContext, but when I'm trying to call CreateSharedImage, the Renderer Process crashed without any debug message. Trying to use a debug build with symbol_level 2 to check again.

😞 Bad news:

SharedImage doesn't work well with this hack. It requires taking ownership of handles, and the renderer process seems having some issue with duplication.

😄 Good news:

Works with WebGPU (with hack)

It becomes more tricky, because the HANDLE on Windows has some internal design at Chromium:

When passing through ipcz, it DuplicateHandle automatically for target process. However it seems causing some error for D3D12 to import (mojo\core\ipcz_driver\transport.cc:221)

I love the good news 😄 ! What do you mean by 'hack'?
And... thank you for working on this!

😄 Good news:

I found hacking WebGPU also very ugly, so I truned my route back to VideoFrame.

After days of debugging, I found managing the handle on Windows very tricky, for example, when an FrameSinkVideoCapturer's output is first CreateSharedHandle on GPU process, and when it pass through IPC, the ipcz automatically DuplicateHandle for target process. Now we are at Main process, we need to pass the handle to Renderer process, but this time, we have no ipcz does the DuplicateHandle thing for us.

I then add a extra param to let user tell the handle's owner process id, and DuplicateHandle it internally to Renderer process. Later, the ipcz will clone the handle for us when passing it back to GPU Process. That's also the reason why Renderer process was crashing before (without any debugging message!), the handle belongs to Main process and it failed to duplicate the handle, making ipc serialization failure.

For other shared handle, user needs either use deprecated GetSharedHandle to create a non-NT handle, or pass the handle owner's pid, in order to successfully import handle on Windows.

At this step, I successfully imported as a VideoFrame, and successfully call WebGPU's importExternalTexture and get a valid object. But I found the rendering has some problem (full black). After some debugging, I found media::VideoFrame::WrapSharedImage will not take own of SharedImage (frame->shared_image_ = shared_image->MakeUnowned();), and cause it destroy immediately when scoped_refptr release. I now manually bind to ReleaseCallback and it works~!

If you're interested, here's a working sample: https://github.com/reitowo/electron-test-import-texture

That's huge! Thank you @reitowo

This now works like a charm in Windows.

The macOS's IOSurface also needs to pass by mach_port across processes. https://source.chromium.org/chromium/chromium/src/+/main:mojo/core/channel_mac.cc;drc=c7beb80e1c9ed11289a501218de27681143b9ba9;l=388

Passing mach_port is much more complicated than I thought. Maybe still need to think twice about the API.

I think the best option is to import at Main process, and use Mailbox to pass between processes in Chromium, which is same pattern in Chromium anyway.

I've done everything to make it work on Windows and macOS! Due to lack of development environment I choose drop Linux support for now.

Test also included, run npm run test -- -g sharedTexture to see. (It will fail (skipped) on CI for lack of GPU, except mac-arm64 which I think have a M series chip)

I'm also happy about current API design, especially zero upstream patch.

Looking forward to hear your advice! @codebytere @nikwen @MarshallOfSound

Binary for testing: https://github.com/reitowo/electron/releases/download/import-texture-v1/dist.zip

Working sample: https://github.com/reitowo/electron-test-import-texture

I found a bug when writing test on macOS that nativeImage was not able to compare pixel data correctly #46949. So I changed the compare method in test.

This is extremely cool @reitowo! The API looks great and should match my use case perfectly - I'll try this out soon

About performance: I created 16 browser 720P OSR send to 1 renderer process, and tested both use 16 canvas or 1 combined canvas render at and the results are same (16x 50fps). On my 3070 we can achieve:

The GPU process's CPU usage is high, I don't know how to optimize further, the GPU usage is not high.

@RenaudRohlinger Could you share more links to the reports by community? Sounds interesting.

Is it possible the video frame is going through the same path people are having issues with in https://issues.chromium.org/issues/406357270 for recent AMD GPUs? You could try passing the flags mentioned in that issue to check if it helps.

Thanks for the reference.

Yes, here those VideoProcessorGetOutputExtension … 0x80070057 errors seems to be
exactly what Chromium issue 406357270 tracks. They mean the driver can’t
open a hardware-secure video processor, so Chromium falls back to the
“clear” path and clamps every shared texture to 2 097 152 px (~1080p).

I tried the combo:

--disable-gpu-driver-bug-workarounds
--ignore-gpu-blocklist
--enable-features=HardwareSecureDecryption:force_support_clear_lead/true

In Chrome this lifts the limit only if the driver is actually able to
create a secure surface. On the affected RX 7900 / 6900 drivers it still
fails, the same 0x80070057 shows up, and the 2-MP cap remains.
So the flags are a good diagnostic step, but they don’t cure the bug.

The OSR relies on the VideoCapture path, so this bug affected us. I think since we can import external texture as a SharedImage we may someday no more rely on VideoConsumer and make our own OSR host (in the future)

I see. In the meantime, do you have a patch in mind for AMD, or is there nothing Electron can do here?

No, I have no AMD GPU on hand. And I think FrameSinkVideoCapturer doesn't involve video encode/decode, if it does, probably it is the oracle's bad again. It will be helpful if you can run with --no-sandbox and get some detailed log files to see if there's more error message.

Only error I have when logging with --no-sandbox:

source: node:electron/js2c/sandbox_bundle (8414)
[9196:0522/145414.813:VERBOSE1:media\gpu\windows\mf_video_encoder_shared_state.cc:145] Hardware encode acceleration is not available for vp9
[9196:0522/145414.969:ERROR:media\gpu\windows\mf_video_encoder_util.cc:552] Set output type failed, The parameter is incorrect. (0x80070057) 
{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 1920, height: 1032 },
    visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
    contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
    timestamp: 0,
    colorSpace: {
      primaries: 'bt709',
      transfer: 'srgb',
      matrix: 'rgb',
      range: 'full'
    },
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 0,
      isWebGpuCompatible: false
    },
    sharedTextureHandle: <Buffer 18 0a 00 00 00 00 00 00>
  }

Related source code:
https://github.com/chromium/chromium/blob/main/media/gpu/windows/mf_video_encoder_shared_state.cc#L144-L146
https://github.com/chromium/chromium/blob/main/media/gpu/windows/mf_video_encoder_util.cc#L546-L547

Hello @reitowo,

I did some extensive investigation on this issue, I even went as far as purchasing an RTX GPU to confirm whether the problem was related to AMD. To my surprise, it turns out the issue wasn't GPU-related at all, but display-related.

My current physical display is limited to a resolution of 1920x1080 with a device pixel ratio (DPR) of 1. Because of this, the capture API appears to be constrained by the physical resolution of the display, meaning it cannot exceed what the screen supports. This limitation seems to be the root cause of the problem.

I believe you haven’t encountered this issue because your display likely supports a higher resolution than mine.

Here are some improved logs that highlight the limitation:

[42300:0529/113219.677:VERBOSE1:components\device_event_log\device_event_log_impl.cc:200] [11:32:19.675] Display: EVENT: screen_win.cc:989 Displays updated, count: 1
[42300:0529/113219.677:VERBOSE1:components\device_event_log\device_event_log_impl.cc:200] [11:32:19.677] Display: EVENT: screen_win.cc:991 Display[4028745484] bounds=[0,0 1920x1080], workarea=[0,0 1920x1032], 

[46736:0529/113220.363:VERBOSE1:media\capture\content\capture_resolution_chooser.cc:205] Recomputed snapped frame sizes: 78x42 <--> 245x132 <--> 413x222 <--> 580x312 <--> 747x402 <--> 915x492 <--> 1082x582 <--> 1250x672 <--> 1417x762 <--> 1585x852 <--> 1752x942 <--> 1920x1032
[46736:0529/113220.363:VERBOSE1:media\capture\content\capture_resolution_chooser.cc:157] Recomputed capture size from 640x360 to 1920x1032 (100% of ideal size)
{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 1920, height: 1032 },
    visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
    contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
    timestamp: 0,
    colorSpace: {
      primaries: 'bt709',
      transfer: 'srgb',
      matrix: 'rgb',
      range: 'full'
    },
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 0
    },
    sharedTextureData: { ntHandle: <Buffer d4 0c 00 00 00 00 00 00> }
 
  }
}

Wow, nice catch! I do using a 4K monitor, and I successfully reproduced your problem. I'll see how to bypass that.

I think by design this should take effect and no constraint should take effect. I'll take a look tomorrow as my Chromium build went invalid :(

Oh, it suddenly become reasonable if electron somehow limits the width and height to the working area, and the resolution_chooser->SetSourceSize will be the final decision as the source frame is just 1920*1032, there's no point to be larger than that.

Yes, can be verified if you try calling osr.setSize(1920, 1080). You'll see the output is now correct. For me, it's 4K with 150% pixel ratio.

Indeed it now works! Thank you for finding this. I think it would be worth it mentioning this detail in the documentation.

Hi, guys.

This PR will be seperated for several minor PRs, and the PR is converted to an RFC for better outcome. electron/rfcs#17. I'll link all minor PRs in this PR for reference. Let's continue keep discussion here!

I've done everything to make it work on Windows and macOS! Due to lack of development environment I choose drop Linux support for now.

Test also included, run npm run test -- -g sharedTexture to see. (It will fail (skipped) on CI for lack of GPU, except mac-arm64 which I think have a M series chip)

I'm also happy about current API design, especially zero upstream patch.

Looking forward to hear your advice! @codebytere @nikwen @MarshallOfSound

Binary for testing: https://github.com/reitowo/electron/releases/download/import-texture-v1/dist.zip

Working sample: https://github.com/reitowo/electron-test-import-texture

output.mp4

Hi @reitowo

Thank you for your work. It’s exactly what we need! I would like to try your test project, but I encountered a 'not found' error with the download link. Could you please provide the latest compiled binary so that we can run the example?

Thank you!

Close this in favor of the last piece of the PR.

16x1080p@35fps

16x720p@54fps

When use with new managed API, testing with offscreen rendering's output, we can still archive 16x1080p@35fps, 16x720p@54fps rendered by WebGPU, which is actually better than before.

The performance could be better if we can elinimate the copy when creating ExternalTexture from VideoFrame. However I leave that for future works. (However, I don't see that will help as I think the limit might be CPU side, too much events).