@Kaiido thank you for the review! I've fleshed things out more, renaming to drawHTMLElement() (because drawElement() will probably not be workable in WebGL) and adding the layoutsubtree attribute.

There's still some handwaving going on of course, in particular what causes the subtree to be laid out but not painted.

I've fleshed this out some more now, in particular the hit testing.

As I said in #11588 (comment) I still don't quite understand the rational to prevent using an ElementImage from another canvas. The sizing issue still does exist even with this limitation: A <canvas> can be resized after the ElementImage has been generated, the browser tab can be moved to a monitor with a different pixel density, the page can be zoomed in, the element can be detached from the document, etc. So there must be a solution for the sizing issues anyway, and it's really unclear to me why that solution couldn't also handle the case of drawing into another canvas.

drawElementImage() requires information from the containing canvas to determine the correct rasterization scale of the content. That canvas information is snapshotted along with the drawn element's paint record during regular rendering updates. Drawing an element into a different canvas poses (at least) two problems:

1. To get the necessary canvas information to compute rasterization scale at the time of invocation, we would need to perform a forced layout. That's a performance pitfall and we got strong pushback from Mozilla on that point.
2. OffscreenCanvas on a worker thread does not have access to the necessary canvas information.

As I said in #11588 (comment) I still don't quite understand the rational to prevent using an ElementImage from another canvas. The sizing issue still does exist even with this limitation: A <canvas> can be resized after the ElementImage has been generated, the browser tab can be moved to a monitor with a different pixel density, the page can be zoomed in, the element can be detached from the document, etc. So there must be a solution for the sizing issues anyway, and it's really unclear to me why that solution couldn't also handle the case of drawing into another canvas.

drawElementImage() requires information from the containing canvas to determine the correct rasterization scale of the content. That canvas information is snapshotted along with the drawn element's paint record during regular rendering updates. Drawing an element into a different canvas poses (at least) two problems:

1. To get the necessary canvas information to compute rasterization scale at the time of invocation, we would need to perform a forced layout. That's a performance pitfall and we got strong pushback from Mozilla on that point.
2. OffscreenCanvas on a worker thread does not have access to the necessary canvas information.

Oh, here's another one that is probably the big obstacle: the hit testing situation becomes much more complicated. I can't even think of a way to handle that sanely.

To get the necessary canvas information to compute rasterization scale at the time of invocation,

Why would you need to do that? The snapshot is created from the proper canvas, right before its onpaint event. The necessary info is available. We're talking about drawing back the ElementImage, not generating it.

OffscreenCanvas on a worker thread does not have access to the necessary canvas information.

Why isn't this part of the snapshot?

Oh, here's another one that is probably the big obstacle: the hit testing situation becomes much more complicated.

I believe it's fine that this one is broken. For all we know the rendered elements may not exist anymore, and if one decides to use it on another canvas I doubt they'll expect hit testing to match on the source one.

And still, how would you handle the case of the source canvas being detached from the document at the time the ElementImage is (re)drawn on the context? Certainly the "necessary info" wouldn't match anymore, right?

To get the necessary canvas information to compute rasterization scale at the time of invocation,

Why would you need to do that? The snapshot is created from the proper canvas, right before its onpaint event. The necessary info is available. We're talking about drawing back the ElementImage, not generating it.

I'm not sure what you mean by the "proper" canvas. The scenario is that element is a child of canvasA but you want to call canvasB.drawElementImage(element). When the snapshot is created, we collect the necessary information about canvasA because it is the DOM parent of element. At what point would we collect the necessary information about canvasB? We can't know a priori that you intend to call canvasB.drawElementImage(element). I'm using Element and ElementImage interchangeably here, the situation is the same either way.

OffscreenCanvas on a worker thread does not have access to the necessary canvas information.

Why isn't this part of the snapshot?

Information about canvasA is part of the snapshot, information about canvasB is not.

Oh, here's another one that is probably the big obstacle: the hit testing situation becomes much more complicated.

I believe it's fine that this one is broken. For all we know the rendered elements may not exist anymore, and if one decides to use it on another canvas I doubt they'll expect hit testing to match on the source one.

If you don't care about hit testing or accessibility or interactivity -- if you just want a pixel stamp of an element -- then you can simply: canvasA.drawElementImage(element); canvasB.drawImage(canvasA).

The scenario is that element is a child of canvasA but you want to call canvasB.drawElementImage(element).

Ok, so I think that's where the misunderstanding lies, I am not talking about drawing an Element, but an ElementImage. So the scenario is, that element is a child of canvasA, I call canvasA.captureElementImage(element) to generate img, and then I want to call canvasB.drawElementImage(img).

... then you can simply: canvasA.drawElementImage(element); canvasB.drawImage(canvasA).

This implies we dirty the "visible" canvasA, it also converts the drawing ops into pixels, and loses the vector-like resizing feature.

And this solution still does not handle the case of drawing an ElementImage on its source canvas, when said canvas is detached.

The scenario is, that element is a child of canvasA but you want to call canvasB.drawElementImage(element).

Ok, so I think that's where the misunderstanding lies, I am not talking about drawing an Element, but an ElementImage. So the scenario is, that element is a child of canvasA, I call canvasA.captureElementImage(element) to generate img, and then I want to call canvasB.drawElementImage(img).

That makes no difference. captureElementImage() doesn't do anything special, it just returns a wrapper around information that was generated internally during the rendering update. The only reason it exists is for sending to a worker thread, which cannot be done with Element.

... then you can simply: canvasA.drawElementImage(element); canvasB.drawImage(canvasA).

This implies we dirty the "visible" canvasA, it also converts the drawing ops into pixels, and loses the vector-like resizing feature.

We cannot do vector-like resizing without the information gathered during rendering update.

That makes no difference. captureElementImage() doesn't do anything special,

What? No, ElementImage and Element do behave differently: ElementImage does hold its snapshot whereas Element is just a key to the <canvas> live list of snapshots.
It has to be this way, otherwise doing something like

ctx.drawElementImage(img, x, y);
ctx.drawElementImage(img, x, y);

in a Worker could use 2 different snapshots.

That makes no difference. captureElementImage() doesn't do anything special,

What? No, ElementImage and Element do behave differently: ElementImage does hold its snapshot whereas Element is just a key to the <canvas> live list of snapshots. It has to be this way, otherwise doing something like

ctx.drawElementImage(img, x, y);
ctx.drawElementImage(img, x, y);

in a Worker could use 2 different snapshots.

Well, the premise is faulty because you can never have an Element on a worker thread. But I think this is a moot point; what I really meant is that there is no difference between Element and ElementImage as it pertains to the issue of drawing into a different canvas from the one containing the Element.

@szager-chromium I tried to contact you by email to avoid flooding this PR, but I must have found the wrong address.

Well, the premise is faulty because you can never have an Element on a worker thread.

First, in my last example, like in the one before, img is an ElementImage, not an Element.
This does matter, because the ElementImage keeps its one and only snapshot. (in specs) This snapshot is supposed to be accessible for at least as long as either the ElementImage is garbage collected, or its close() method is called.
This means that an ElementImage can outlive the element it represents, and may definitely not represent the current state of the element at the time it’s drawn.

For an Element on the other hand, we only pick in the <canvas>’s map of available snapshots (in specs), if there is one for this Element at the time drawElementImage(element, x, y) (or getElementTransform) is called.

So in code that would give

canvas.addEventListener("paint", (evt) => {
    // First event firing
    // Here we have a snapshot created for the element, let’s call it snapshot_1
    const img = canvas.captureElementImage(element);
    element.style.color = red; // trigger a new paint
    canvas.addEventListener("paint", (evt) => {
        // Second event firing
        // Here we have another snapshot created for the element, snapshot_2
        ctx.drawElementImage(img, x, y); // uses the old snapshot_1, because it’s the one held by the `ElementImage`
        ctx.drawElementImage(element, x, y); // uses the new snapshot_2, available in the live canvas map of snapshots
    }, { once: true });
}, { once: true });

You can’t have ElementImage work as a key over the live map like Element does, because in a worker thread you can’t know when the map content in the main thread will change.

canvas.drawElementImage(img, x, y); // uses snapshot_1
// the map on the main thread is updated, but in the worker we're sync
canvas.drawElementImage(img, x, y); // uses snapshot_2

This shouldn’t happen.

Now, this means the “necessary information” from the canvas also needs to be accessible for as long as the ElementImage holds it; even if you do maintain a same-canvas policy.
You weren’t very explicit about this “necessary information”, but I suppose it would change when the canvas is resized, moved to another monitor or even detached entirely from the document.
The current solution provided in the PR, to prevent drawing the ElementImage in other canvases, does not solve this issue. You still must keep the “necessary information” along with the ElementImage or the snapshot.

So, if that information is held even for the same-canvas policy, why can’t it be used by another canvas? Why would it work between canvasA and canvasA-after-a-resize, but not between canvasA and canvasB?

As I said in #11588 (comment) I still don't quite understand the rational to prevent using an ElementImage from another canvas. The sizing issue still does exist even with this limitation: A <canvas> can be resized after the ElementImage has been generated, the browser tab can be moved to a monitor with a different pixel density, the page can be zoomed in, the element can be detached from the document, etc. So there must be a solution for the sizing issues anyway, and it's really unclear to me why that solution couldn't also handle the case of drawing into another canvas.

When drawing an element from canvasA into canvasB, it is difficult to correctly update the DOM position (transform) of the element so that it matches the on-screen location, which is needed for web platform features like hit testing, accessibility, etc.

When drawing an element from canvasA into canvasB, it is difficult to correctly update the DOM position (transform) of the element so that it matches the on-screen location, which is needed for web platform features like hit testing, accessibility, etc.

But you face the same issues with an ElementImage that holds an out of date snapshot, whether it came from the same canvas or not. The content of the element in DOM may be completely different than what the old snapshot holds (I mean, the original element could even not exist in DOM anymore).
I personally think it's ok if this is broken, it still allows complex processing using multiple layers and vector sizing, and it's still possible to do the positioning in DOM yourself.

But if you believe it's a no-go, then there should probably be further limitation to ElementImage objects, because only preventing them from being drawn on a different canvas doesn't solve much.

You can’t have ElementImage work as a key over the live map like Element does, because in a worker thread you can’t know when the map content in the main thread will change.

canvas.drawElementImage(img, x, y); // uses snapshot_1
// the map on the main thread is updated, but in the worker we're sync
canvas.drawElementImage(img, x, y); // uses snapshot_2

This shouldn’t happen.

This can never happen, for all kinds of reasons that have nothing to do with HTML-in-canvas. There is no concurrent access of this kind anywhere in JavaScript, with the exception of SharedArrayBuffer. But again -- and I'm having a hard time making myself understood on this point -- the "liveness" of Element compared to ElementImage is really besides the point...

Now, this means the “necessary information” from the canvas also needs to be accessible for as long as the ElementImage holds it; even if you do maintain a same-canvas policy. You weren’t very explicit about this “necessary information”, but I suppose it would change when the canvas is resized, moved to another monitor or even detached entirely from the document. The current solution provided in the PR, to prevent drawing the ElementImage in other canvases, does not solve this issue. You still must keep the “necessary information” along with the ElementImage or the snapshot.

So, if that information is held even for the same-canvas policy, why can’t it be used by another canvas? Why would it work between canvasA and canvasA-after-a-resize, but not between canvasA and canvasB?

Among the canvas information required to determine the correct raster scale for drawElementImage() is the canvas' width and height attributes, its CSS box size, and its CSS zoom level. All that information is computed for canvasA and included in element's snapshot because element is a child of canvasA. If you call canvasB.drawElementImage(element), we don't have the snapshotted width/height, CSS box size, and CSS zoom level of canvasB available. There are pretty much two options in this situation: do a forced layout to get the necessary information for canvasB; or rasterize using the snapshotted information from canvasA, which can give screwy results. Maybe you don't care if the raster scale is wrong and you can live with the screwy results; but if that's true then canvasA.drawElementImage(element); canvasB.drawImage(canvasA) will work just as well.

As mentioned before, allowing an ElementImage to be drawn into a different canvas creates some potentially very confusing situations with respect to hit testing and rasterization scale, and it's very likely to be negative for accessibility as well. It also creates a possible weird loophole for privacy-preserving paint behavior: what should the browser do if you postMessage an ElementImage to a cross-origin context and draw it into a canvas there?

In my opinion, for all these reasons and for other reasons that I can't even think of right now, your stated reasons for wanting this behavior do not outweight the complexity and potential for confusion.

This can never happen, ...

I might be wrong, but I'm pretty sure it's still something we defend against when writing cross-context specs.
Besides, remember we're talking about two JS contexts working in parallel and having access to the same non-JS resources.

Among the canvas information required to determine the correct raster scale for drawElementImage() is the canvas' width and height attributes, its CSS box size, and its CSS zoom level.

And that information is gone as soon as the snapshot held by the ElementImage object is outdated. So you still face the same issues, even if you allow only drawing on the same canvas.

It also creates a possible weird loophole for privacy-preserving paint behavior: what should the browser do if you postMessage an ElementImage to a cross-origin context and draw it into a canvas there?

You can also send an ImageBitmap, or an ImageData of that snapshot rendered on the canvas to that cross-origin context. You can't prevent a page from leaking its data to another page if it wants to do so. What needs to be prevented is the cross-origin doc being able to read that info without the page's consent.

canvasA.drawElementImage(element); canvasB.drawImage(canvasA) will work just as well.

Once again, no. This prevents doing complex compositing in a Worker while maintaining the visible canvas clean with the last generated frame for the time the processing is done. It forces you to display an half-baked frame.

your stated reasons for wanting this behavior do not outweight the complexity and potential for confusion.

My main point isn't that this should work just for the sake of it, but rather that this limitation doesn't solve anything for an ElementImage with an outdated snapshot.

My main point isn't that this should work just for the sake of it, but rather that this limitation doesn't solve anything for an ElementImage with an outdated snapshot.

I sense that we have perhaps reached the limit of what can be resolved with this exchange. If you feel strongly on this point, I'd suggest you file a separate issue to bring it to the attention of the standards bodies. I don't think this PR should be blocked on resolving it.

I don't think this PR should be blocked on resolving it.

I think I disagree on this.
You came as the sole developer of this feature saying that the planned API had an issue. We have to trust you on this, and I do.
You proposed a solution that does bring some new limitations to the API. This goes against the UX > DX > impl > specs hierarchy, so as a reviewer, I think it's normal I challenge this proposed change. Given this is a whole new API and no actual user has voiced in favor of the particular use cases that would be prevented by this limitation, I do agree it would still be an acceptable change.

But, even with this limitation, the issue you identified still does exist in some other parts of the API as is shown in this fiddle (for the green text). This needs to be resolved too. We can't leave the API with a known broken path in it.
So please provide a solution to also fix the issue you've identified, but for the ElementImage objects that point to outdated snapshots,

From what I gather from your past comments, I am under the impression that it's mainly the magical 3 argument version of drawElementImage(element, x, y) that triggers the issue, and that the 5 argument version with explicit dw and dh wouldn't have the same problem. Is it so? Then maybe, this 3 argument version is the actual problem?

I don't think this PR should be blocked on resolving it.

I think I disagree on this. You came as the sole developer of this feature saying that the planned API had an issue. We have to trust you on this, and I do. You proposed a solution that does bring some new limitations to the API. This goes against the UX > DX > impl > specs hierarchy, so as a reviewer, I think it's normal I challenge this proposed change. Given this is a whole new API and no actual user has voiced in favor of the particular use cases that would be prevented by this limitation, I do agree it would still be an acceptable change.

But, even with this limitation, the issue you identified still does exist in some other parts of the API as is shown in this fiddle (for the green text). This needs to be resolved too. We can't leave the API with a known broken path in it. So please provide a solution to also fix the issue you've identified, but for the ElementImage objects that point to outdated snapshots,

From what I gather from your past comments, I am under the impression that it's mainly the magical 3 argument version of drawElementImage(element, x, y) that triggers the issue, and that the 5 argument version with explicit dw and dh wouldn't have the same problem. Is it so? Then maybe, this 3 argument version is the actual problem?

This is a known issue with the feature as specified: if you change the canvas grid and draw an outdated ElementImage it will have distorted proportions. We went through a very lengthy process of considering alternatives and working with other browsers to find something we all can agree on, and this is it. If you would like to suggest a different approach to the ones already considered please file an issue in the project repository. We do not have any clever ideas for how to accomplish this.

In my opinion this is not a significant shortcoming. As a point of comparison: the canvas is already implicitly clear()-ed when the canvas grid changes. One could argue that this is also a form of broken-ness -- one which your demo side-steps without complaint by doing all its painting after the resize. As a practical matter developers don't seem to have a problem with it. As for HTML-in-canvas, I don't think it's too onerous to require developers to draw with new element paint snapshots after resizing the canvas grid if they want to avoid distortion. This could perhaps be emphasized in the spec, but I don't agree that this constitutes "broken" behavior.

Specifying dx/dy will avoid the behavior but may introduce fuzziness, even if no projection is applied when drawing. Only the three-argument version gives reliably pixel-perfect results when compared to the same content placed outside the canvas.

(Drive by review; I haven't read the full patch or comment history)

Hey folks! I was working on some paint-timing spec cleanup (context: https://github.com/wicg/soft-navigations) and @shaseley pointed out that the "mark paint timing" steps are ambiguously expecting to start pre-paint but end post-paint (i.e., include paint time).

I think your PR nicely clarifies the update-the-rendering steps, but looks like it moves "mark paint time" to come after paint, and would AFAIK change the intent of the paintTime value that many specifications rely on for performance reporting.

See: w3c/paint-timing#126

I think the best solution would be to split "mark paint timing" into steps that come before paint, and steps that come after.

An alternative solution might be to mark the pre-paint timestamp in update-the-rendering and then pass that timestamp to "mark paint timing", similar to how we mark first, then pass unsafeStyleAndLayoutStartTime to "record rendering time".

@foolip , to address @mmocny 's point, we can go ahead and land this "breaking" change that introduces a paint step, and then ask @mmocny to follow up with a change to split paint timing into a pre-paint and post-paint steps, and w3c/paint-timing#126 can track this.