We need to define the behavior of reading from WebGPU canvases using other Web APIs.
Most of these are "peek" operations that have no effect on the canvas contents:
There is also a "swap" operation:
OffscreenCanvas.transferToImageBitmap()
And "peek" operations that are done by the user (not by the page):
- Printing (to paper/pdf)
- Copy to clipboard
- "Save Image As"
Historically, the interaction between drawing operations inside the context and read operations outside the context is very "immediate" - in a way, all drawing operations on a page could be considered to happen on the same GPUQueue, so drawing operations are always available immediately. This results in some complications, especially regarding:
- Interaction with end-of-frame document compositing ("animation frame"), especially in WebGL with
preserveDrawingBuffer: false.
- Relationship between user operations like printing and
preserveDrawingBuffer - we previously agreed to avoid this in WebGPU.
- Resource synchronization inside implementations, e.g. transitioning the
[[currentTexture]] from WebGPU-writable, to WebGL-readable, back to WebGPU-writable.
We previously discussed "explicit" presentation APIs that would have e.g. required a swapChain.present() call in order for a frame to be submitted to the compositor. However this was rejected in favor of behavior that meshed better with the Web platform:
- Implicit destruction of the current swap chain texture on animation frame for
HTMLCanvasElements in the DOM and OffscreenCanvases created via transferControlToOffscreen (I'm not sure about HTMLCanvasElements which are not in the DOM - should match WebGL).
- I don't think we agreed on specific semantics for
OffscreenCanvases created via new OffscreenCanvas().
Proposals
These proposals straddles the boundary between the current mostly-implicit semantics and the earlier very explicit semantics, while preserving the properties (round-robin ordering) that are needed for implementations to be able to optimize using native swap chains.
Proposal 1
- Define "destroy" on canvas-presentation textures as an "early commit"1: the WebGPU device loses access, and the texture contents are preserved but become immutable.
- On (a) animation frame (for presented canvases), or (b) inside
OffscreenCanvas.transferToImageBitmap():
- Call
destroy() on the [[currentTexture]] (defined above).
- "Pop" the
[[currentTexture]] slot (leaving null).
- Hand off the preserved, now-immutable texture contents: to (a) the compositor, or (b) a new
ImageBitmap.
- Inside user-initiated "peek" operations:
[[currentTexture]] is ignored. Instead, use whatever contents are currently held by the compositor.2
- Inside programmatic "peek" operations:
- Throw if
[[currentTexture]] is null.
- Throw if
[[currentTexture]] is not already "destroyed" (defined above).
- Possibly throw if the WebGPU canvas is desynchronized (either using
desynchronized: true or transferControlToOffscreen()).3
- Copy the contents out of the texture.
Notes:
- 1 Despite having "early commit", it is still not possible to advance the "swap chain" more quickly than rAF:
destroy() doesn't affect [[currentTexture]], so getCurrentTexture() in the same frame will still return the same (destroyed) texture. Of course, people always ask to be able to do this, but I think a solution to that problem would be different (e.g. swapChain.discardCurrentTextureIfAny(), or present using transferToImageBitmap()+ImageBitmapRenderingContext).5
Proposal 2
An alternative that replaces validation with either extra copies or extra synchronization (implementer's choice).
- Define "destroy" on swap chain textures as a "discard"4, destroying the texture as usual and also setting
[[currentTexture]] to null.
- Note this means there will never be a
[[currentTexture]] that is in the destroyed state.
- Also note if this happens, the currently-presented frame will not be replaced, so this effectively "cancels" presentation.5
- In (a)
present the presentation context content to the canvas, or (b) OffscreenCanvas.transferToImageBitmap():
- If
[[currentTexture]] is non-null:
- "Pop" the
[[currentTexture]] slot (leaving null).
- "Move" the contents of the canvas out of the texture, marking it as
destroyed so the app can't use it.
- Hand off the contents: to (a) the compositor, or (b) a new
ImageBitmap.
- Inside user-initiated "peek" operations:
[[currentTexture]] is ignored. Instead, use whatever contents are currently held by the compositor.2
- Inside programmatic "peek" operations:
- Throw if
[[currentTexture]] is null.
- Possibly throw if the WebGPU canvas is desynchronized (either using
desynchronized: true or transferControlToOffscreen()).3
- Copy the contents out of the texture. (Requires synchronization to make sure that later writes from WebGPU are not seen. Implementations may instead simply copy the data out to a staging resource using a
copyTextureToTexture command here to easily ensure this.)
Proposal 3 (my current recommendation)
Slight refinement of proposal 2.
- In
getCurrentTexture, replace "if [[currentTexture]] is null" with "if [[currentTexture]] is null or destroyed".4
- In (a)
present the presentation context content to the canvas, or (b) OffscreenCanvas.transferToImageBitmap():
- If
[[currentTexture]] is not null and not destroyed:
- "Move" the contents of the canvas out of the texture, marking it as
destroyed so the app can't use it.
- Hand off the contents: to (a) the compositor, or (b) a new
ImageBitmap.
- Note if the app destroyed the texture, the currently-presented frame will not be replaced, so this effectively "cancels" presentation.
- Inside user-initiated "peek" operations:
[[currentTexture]] is ignored. Instead, use whatever contents are currently held by the compositor.2
- Inside programmatic "peek" operations:
- Fail if
[[currentTexture]] is null or destroyed.
- Need to decide whether this throws an exception, or returns a blank image (to avoid adding special cases to other APIs).
- Possibly fail if the WebGPU canvas is desynchronized (either using
desynchronized: true or transferControlToOffscreen()).3
- Copy the contents out of the texture. (Requires synchronization to make sure that later writes from WebGPU are not seen. Implementations may instead simply copy the data out to a staging resource using a
copyTextureToTexture command here to easily ensure this.)
Notes:
- 2 There still may be some cases where the compositor-owned contents cannot be read - e.g. a native swap chain is in use because the canvas is marked as
desynchronized or uses transferControlToOffscreen - but I'm not sure whether we can solve this. If we can't, we should consider trying to spec and test exact behavior in these cases, like that it's always blank.
- 3 We could require a specific, explicit flag in
GPUPresentationConfiguration to say "I want to be able to read from this canvas", and reading works IFF that is set. Can then be validated not to conflict with desynchronized: true/transferControlToOffscreen() as mentioned above. This also lets us safely make validation more lenient later on, rather than specifying the "always blank" behavior.
- 4 This does allow advancing the swap chain more quickly than rAF.
- This isn't necessary for outputting multiple images per frame (an important use case) - a clear could be used instead - but we have to define
destroy() behavior anyway.5
-
5 Triple buffering
Developers have expressed interest in "triple buffered" style rendering, rendering as fast as possible and having the latest result be displayed to the screen. I believe this is not feasible on the main thread, because it would fool the scheduler into thinking each frame is very expensive. However, this could be done on a background thread if there were a way to submit a frame and then start a new frame. destroy() could be a way to do that, but more likely either the GPUPresentationContext would have a new method for that, or it would be done with transferToImageBitmap()+ImageBitmapRenderingContext.1
We need to define the behavior of reading from WebGPU canvases using other Web APIs.
Most of these are "peek" operations that have no effect on the canvas contents:
CanvasRenderingContext2D.drawImage()WebGLRenderingContext.texImage2D()et al.HTMLCanvasElement.toDataURL()/.toBlob()/OffscreenCanvas.convertToBlob()(What is the toDataURL behavior for GPUCanvasContext? #1548)There is also a "swap" operation:
OffscreenCanvas.transferToImageBitmap()And "peek" operations that are done by the user (not by the page):
Historically, the interaction between drawing operations inside the context and read operations outside the context is very "immediate" - in a way, all drawing operations on a page could be considered to happen on the same
GPUQueue, so drawing operations are always available immediately. This results in some complications, especially regarding:preserveDrawingBuffer: false.preserveDrawingBuffer- we previously agreed to avoid this in WebGPU.[[currentTexture]]from WebGPU-writable, to WebGL-readable, back to WebGPU-writable.We previously discussed "explicit" presentation APIs that would have e.g. required a
swapChain.present()call in order for a frame to be submitted to the compositor. However this was rejected in favor of behavior that meshed better with the Web platform:HTMLCanvasElements in the DOM andOffscreenCanvases created viatransferControlToOffscreen(I'm not sure aboutHTMLCanvasElements which are not in the DOM - should match WebGL).OffscreenCanvases created vianew OffscreenCanvas().Proposals
These proposals straddles the boundary between the current mostly-implicit semantics and the earlier very explicit semantics, while preserving the properties (round-robin ordering) that are needed for implementations to be able to optimize using native swap chains.
Proposal 1
OffscreenCanvas.transferToImageBitmap():destroy()on the[[currentTexture]](defined above).[[currentTexture]]slot (leavingnull).ImageBitmap.[[currentTexture]]is ignored. Instead, use whatever contents are currently held by the compositor.2[[currentTexture]]isnull.[[currentTexture]]is not already "destroyed" (defined above).desynchronized: trueortransferControlToOffscreen()).3Notes:
destroy()doesn't affect[[currentTexture]], sogetCurrentTexture()in the same frame will still return the same (destroyed) texture. Of course, people always ask to be able to do this, but I think a solution to that problem would be different (e.g.swapChain.discardCurrentTextureIfAny(), or present usingtransferToImageBitmap()+ImageBitmapRenderingContext).5Proposal 2
An alternative that replaces validation with either extra copies or extra synchronization (implementer's choice).
[[currentTexture]]to null.[[currentTexture]]that is in thedestroyedstate.present the presentation context content to the canvas, or (b)OffscreenCanvas.transferToImageBitmap():[[currentTexture]]is non-null:[[currentTexture]]slot (leavingnull).destroyedso the app can't use it.ImageBitmap.[[currentTexture]]is ignored. Instead, use whatever contents are currently held by the compositor.2[[currentTexture]]isnull.desynchronized: trueortransferControlToOffscreen()).3copyTextureToTexturecommand here to easily ensure this.)Proposal 3 (my current recommendation)
Slight refinement of proposal 2.
getCurrentTexture, replace "if [[currentTexture]] is null" with "if [[currentTexture]] is null or destroyed".4present the presentation context content to the canvas, or (b)OffscreenCanvas.transferToImageBitmap():[[currentTexture]]is notnulland not destroyed:destroyedso the app can't use it.ImageBitmap.[[currentTexture]]is ignored. Instead, use whatever contents are currently held by the compositor.2[[currentTexture]]isnullor destroyed.desynchronized: trueortransferControlToOffscreen()).3copyTextureToTexturecommand here to easily ensure this.)Notes:
desynchronizedor usestransferControlToOffscreen- but I'm not sure whether we can solve this. If we can't, we should consider trying to spec and test exact behavior in these cases, like that it's always blank.GPUPresentationConfigurationto say "I want to be able to read from this canvas", and reading works IFF that is set. Can then be validated not to conflict withdesynchronized: true/transferControlToOffscreen()as mentioned above. This also lets us safely make validation more lenient later on, rather than specifying the "always blank" behavior.destroy()behavior anyway.55 Triple buffering
Developers have expressed interest in "triple buffered" style rendering, rendering as fast as possible and having the latest result be displayed to the screen. I believe this is not feasible on the main thread, because it would fool the scheduler into thinking each frame is very expensive. However, this could be done on a background thread if there were a way to submit a frame and then start a new frame.destroy()could be a way to do that, but more likely either theGPUPresentationContextwould have a new method for that, or it would be done withtransferToImageBitmap()+ImageBitmapRenderingContext.1