Hitting that CI Chrome error I think:

Windows 10 chrome dev (error)         
Error: {"status": 33, "sessionId": "ea17ac5e971a42afa2b5fabff307362e", "value": {"message": "session not created: Chrome version must be between 70 and 73\n  (Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 10.0.10586 x86_64)", "webdriver.remote.sessionid": "ea17ac5e971a42afa2b5fabff307362e", "hasMetadata": true}}, Timed out

• There can easily be scenarios where the saturation or brightness is unexpected without additional knobs for environment map intensity, exposure, and tonemapping. Using a bright HDR environment map on a reflective surface will, of course, result in a bright render.

We may want to expose... eh.. exposure in <model-viewer>.

Tonemapping is an opinionated change. I don't think chasing Filament's GLTF Viewer is worthwhile for pixel perfect comparison

For the time being, this is our standard for comparison. If we aren't chasing that, we need alternatives to consider (a custom Filament configuration, a selection of ray traced samples etc). But, keep in mind that <model-viewer> can afford to be more opinionated than Three.js so the Filament GLTF viewer is probably as good as anything for the time being.

It may be best to just stick to linear tonemapping for now until we expose knobs.

We need to pick a reasonable default. In time, we will expose knobs at some level. A conservative default is probably better so that we don't surprise users down the line when our opinions change.

Any opinion then on tonemapping for this change? Current one is used to match Filament (ACES Filmic Tonemapping, seems darker, more saturated colors), but the default (linear) will of course reduce visual delta.

General feedback follows:

Offline PMREM generation

Some of the qualities of this change (significant payload burden, high cost for realtime changes) suggest that PMREM generation would be well suited to a build time tool. Does this seem reasonable to you?

Script execution time

The script execution time cost of runtime PMREM generation seems high. On Chrome desktop, I measured an added cost in excess of 150ms for the PBR Spheres fidelity test with background-image. Even without a background-image, PMREM is still a cost factor and adds almost 100ms in script execution time. See timeline screenshots below (and note that in the no-background-image case pmremPass is actually invoked twice). Is it necessary to levy this cost on users by default? Should PMREM be opt-in?

Quick overview of the PMREM texture generator: Generating radiance maps is inherently a computationally intensive task -- each map represents the amount of light a facet receives from a given direction, essentially averaging all light sources for each texel(ish), and scaled down/blurred for various roughness values. I think I'm more accurately describing the process for importance sampling PR in OP, but that's the idea, computationally costly.

I can't imagine any production pipelines generating this at runtime unless for good reason (e.g. dynamic indirect lighting). The value/justification for this within three I think is:

• No well-known tools to do this that are compatible with three. There are a handful of tools that exist, but non in common circulation amongst web developers at least. cmgen has been getting some buzz and seems like it'd work well for this, so this may be a solved problem with some docs.
• The "PMREM" format used by three is arbitrary, I think, and coupled with the renderer. Although the texture could possibly use any 'layout' of the texture, assuming the mipmaps were set correctly. This makes it a little more challenging than the more ubiquitous mappings, like equirectangular or cubemaps.
• Radiance/PMREM maps are non-trivial to even understand (IMO), and runtime generation solves the problem of "nice defaults".
• pmremPass being called twice, even for default environment, sounds like some sort of race condition/bug. Note that this could be improved on a larger scale Cache generated environment map #336

For production use cases, this should certainly be generated offline, no doubt. This also assumes having 3D resources available and supporting pipelines. I'm not sure how well three handles that in general (I should look into what it'd take to use cmgen to output a texture that can be used directly in three, it should be possible, just a matter of matching up the 'format').

For more casual/prototype use cases, it seems more likely that they won't know what PMREM is, why its needed, or how to make it. It seems valuable to just handle this for developers. But of course, the costs!

• 100-150ms for generating a reasonably sized texture is a lot. On top of also generating the default environment maps. FWIW I didn't find it too noticeable, but that'll only be worse for lower power devices.
• This patch increases the minified cost by 12kb (887kb to 899kb) and the gzip cost by 3kb (188kb to 191kb). This doesn't bother me much, considering that difference could be shaken out via other means, I think (and accompanied by bandwidth-heavy assets). Precomputing a radiance map would be orders of magnitude larger in terms of bandwidth.

Some strategies for mitigating the computing cost:

• Only run for HDR images. This is where PMREM is most valuable anyway. This would prevent overhead on the more casual use cases of no background or sRGB background where it's less noticible.
• Always opt-in: maybe this is only for power users, but in that case, couldn't they precompute the radiance? Yes but it's still non-trivial and a very different workflow compared to other webby things. Maybe radiance attribute that can contain a URL to a radiance map, or if no value provided but the attribute is defined (el.hasAttribute('radiance')), then a radiance map should be generated if not provided.
• Only allow precomputed radiance maps: this forces the most efficient, production-grade path, but seems like a pretty big obstacle for most devs outside of an established pipeline.

Fidelity tests

Given the strong effect this change has on rendering, it would be nice to see a contribution to the fidelity tests as part of this change. On the one hand, PBR Spheres looks subjectively worse to me ("what happened to the skybox?"). On the other hand, it looks more like Filament, which is... good? My takeaway is that we aren't consciously measuring the right things.

These changes shock me and not sure why the skybox is changing..

Also note that cmgen is a part of Filament, and requires compiling the project -- even if cmgen is the perfect tool for the job, building the project can be a tall order for web devs unfamiliar with using a native build pipeline.

cmgen does not handle creating a single file with multiple mips AFAICT, would need to stitch that outside. Still doable, can stitch with convert or something, just more overhead.

Similarly, the reason why equirectangular images are used for background images is that it's a single image, whereas cubemaps require 6 different images (or a single image in a cross, but again, arbitrary), which faces a similar issue (in terms of "formats", not in terms of jank/computational costs)

Thanks for the thorough walk-through @jsantell

Some additional thoughts:

No well-known tools to do this that are compatible with three.
The "PMREM" format used by three is arbitrary, I think, and coupled with the renderer.
I should look into what it'd take to use cmgen to output a texture that can be used directly in three, it should be possible, just a matter of matching up the 'format'

Not suggesting we should do it in this change, but it seems like this is an area where we could help the community a bit while helping ourselves.

If we document the format and socialize it a bit I'm sure we could come up with something that others would make use of.

OTOH if we could somehow leverage cmgen, that would be a nice win for a broader swatch of the ecosystem than Three.js users. Although one thing I have noticed in my time using it: cmgen is quite slow compared to what we're doing at runtime.

I've filed #341 to track work on an offline map generation strategy.

Radiance/PMREM maps are non-trivial to even understand (IMO), and runtime generation solves the problem of "nice defaults"

Runtime generation definitely makes it easier to experiment. I think the other main advantage of runtime generation is for scenarios when the map might change frequently and generating the maps in advance is complicated.

It's not that much harder for someone to wield by specifying a URL in the radiance attribute. We can help them leap the conceptual gap with a visual configurator tool of some kind that produces a map for them to use by URL.

pmremPass being called twice, even for default environment, sounds like some sort of race condition/bug

Let's investigate / address this if we can before landing this change.

100-150ms for generating a reasonably sized texture is a lot. On top of also generating the default environment maps. FWIW I didn't find it too noticeable, but that'll only be worse for lower power devices.

I agree, I didn't notice this as much on my laptop. However, 100ms could easily be 1s on a less capable device. And, if I have more than one <model-viewer> on the page... ultimately, 100ms of blocking script execution time on Chrome Desktop is an unacceptable delta if you are working at a scale similar to one of Google's products.

For the future, I've filed #342 to track moving this work off of the main thread. I've also filed #343 to track generating the map lazily and progressively enhancing so as not to block rendering something while the map is being generated.

Maybe radiance attribute that can contain a URL to a radiance map, or if no value provided but the attribute is defined (el.hasAttribute('radiance'))

Good call. This is probably the best option to mitigate performance concerns for now.

These changes shock me and not sure why the skybox is changing..

Agreed, let's try to understand this better before we land the change.

SGTM, I'll check out:

• Investigate/eliminate redundant pmremPass calls
• Investigate why the skybox is different, not necessarily fix it -- although now I suspect it's most likely the tonemapping, since the skybox hasn't changed otherwise
• Gate PMREM generation on a radiance attribute
  • Perhaps should be pmrem? Environment maps are essentially radiance maps for roughness=0. Maybe environment= that can take equirect?
  • Possibly punt on pmrem=mypmrem.hdr since there does not exist any tools that generate something in this format AFAIK (and would also require UV generation, should be shareable across all pmrems?)

My bikeshed on attribute name is that we should avoid acronyms and also pick something that conveys both the breadth of the technical concept and the visual effect at the same time. Surely it will be easy to find a single word that satisfies these requirements. Surely.

• Investigate/eliminate redundant pmremPass calls

Tracked down to #344, filed (edit I think I fixed it in this patch now)

• Investigate why the skybox is different, not necessarily fix it -- although now I suspect it's most likely the tonemapping, since the skybox hasn't changed otherwise

Yup, tonemapping explains it.

• Gate PMREM generation on a radiance attribute

  • Perhaps should be pmrem? Environment maps are essentially radiance maps for roughness=0. Maybe environment= that can take equirect?
  • Possibly punt on pmrem=mypmrem.hdr since there does not exist any tools that generate something in this format AFAIK (and would also require UV generation, should be shareable across all pmrems?)

Gating on an experimental-pmrem attribute for now -- which can be usable by HDR and non-HDR, I'd wager that the LDR with PMREM is more "accurate", although visually a big change, and could be unexpected.

For example, here's the helmet with default environment map without, and with, PMREM:

And again without, and with, PMREM:

The white spheres we use often are way brighter as well -- which could be unexpected -- but, these are pure white albedo spheres that are lit on all sides by a whiteish environment, it's going to be a rather well lit sphere that absorbs all light. Above without PMREM, below with:

All images here have been tonemapped, although that affects the output much less than whether or not PMREM is enabled.

Remaining TODO: tests

My bikeshed on attribute name is that we should avoid acronyms and also pick something that conveys both the breadth of the technical concept and the visual effect at the same time. Surely it will be easy to find a single word that satisfies these requirements. Surely.

Saw this after my post and push, but with a name like experimental-pmrem it's clear that I have no idea what a good name for this is (especially since it may conveying format information e..g PMREM is a specific implementation of IBL lighting)

I have filed #346 to cover future work on naming. For now, let's just stick to experimental-pmrem.

I broke default environment maps in above shot. Updated helmet shot without and with PMREM on a default environment (note, rougher materials and non-default env maps benefit more from this)

Pushed changes and tests

Also fixes #344

I've opened a PR against this change that includes a fidelity test for HDR/PMREM. PTAL: #361

Amazing work on this. <model-viewer> looks so beautiful now... LGTM!

😄 thrilled to click merge