Ash runner: fix crash on resize/minimize and add shader hot reloading by Hentropy · Pull Request #299 · EmbarkStudios/rust-gpu

Hentropy · 2020-12-02T11:56:31Z

rebuild the swapchain when the window is resized, pipelines use dynamic state so they are not rebuilt.
F5 will rebuild the shader crate, load the spirv, and rebuild the pipeline all on a separate thread.

I also moved the ash setup code around quite a bit, so it looks like a big change when in reality it's like ~75% the same code.

khyperia

Looks good! I'd like someone else to take a look at it since I don't know ash very well, but seems good to me.

examples/runners/ash/src/main.rs

Jasper-Bekkers

Running with --debug-layer results in quite a few validation error messages on shutdown (I think it gets worse with more reloads, but I'm not 100% sure), I think those should get fixed before we merge this in.

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0xfe25f60000000013, type = VK_OBJECT_TYPE_SHADER_MODULE; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkShaderModule 0xfe25f60000000013[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0x51b3510000000014, type = VK_OBJECT_TYPE_PIPELINE_LAYOUT; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkPipelineLayout 0x51b3510000000014[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0x6269300000000015, type = VK_OBJECT_TYPE_PIPELINE; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkPipeline 0x6269300000000015[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0xb5f68b000000000e, type = VK_OBJECT_TYPE_RENDER_PASS; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkRenderPass 0xb5f68b000000000e[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0x1f91b40000000011, type = VK_OBJECT_TYPE_FRAMEBUFFER; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkFramebuffer 0x1f91b40000000011[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0xc6ac6a000000000f, type = VK_OBJECT_TYPE_FRAMEBUFFER; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkFramebuffer 0xc6ac6a000000000f[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

ERROR:
VALIDATION [VUID-vkDestroyDevice-device-00378 (1901072314)] : Validation Error: [ VUID-vkDestroyDevice-device-00378 ] Object 0: handle = 0x168e2411ca8, type = VK_OBJECT_TYPE_DEVICE; Object 1: handle = 0xedbd50000000010, type = VK_OBJECT_TYPE_FRAMEBUFFER; | MessageID = 0x71500fba | OBJ ERROR : For VkDevice 0x168e2411ca8[], VkFramebuffer 0xedbd50000000010[] has not been destroyed. The Vulkan spec states: All child objects created on device must have been destroyed prior to destroying device (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-vkDestroyDevice-device-00378)

I don't know if you intended to fix minimizing the window with this PR however I think it would be a small (but nice) addition to this. Right now we're getting validation layer messages around this as well.

ERROR:
VALIDATION [VUID-VkSwapchainCreateInfoKHR-imageExtent-01689 (320081257)] : Validation Error: [ VUID-VkSwapchainCreateInfoKHR-imageExtent-01689 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x13140d69 | vkCreateSwapchainKHR(): pCreateInfo->imageExtent = (0, 0) which is illegal. The Vulkan spec states: imageExtent members width and height must both be non-zero (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkSwapchainCreateInfoKHR-imageExtent-01689)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-width-00885 (-1231547098)] : Validation Error: [ VUID-VkFramebufferCreateInfo-width-00885 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xb6981526 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo width must be greater than zero. The Vulkan spec states: width must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-width-00885)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-height-00887 (-1513456701)] : Validation Error: [ VUID-VkFramebufferCreateInfo-height-00887 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa5ca7bc3 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo height must be greater than zero. The Vulkan spec states: height must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-height-00887)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-width-00885 (-1231547098)] : Validation Error: [ VUID-VkFramebufferCreateInfo-width-00885 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xb6981526 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo width must be greater than zero. The Vulkan spec states: width must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-width-00885)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-height-00887 (-1513456701)] : Validation Error: [ VUID-VkFramebufferCreateInfo-height-00887 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa5ca7bc3 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo height must be greater than zero. The Vulkan spec states: height must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-height-00887)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-width-00885 (-1231547098)] : Validation Error: [ VUID-VkFramebufferCreateInfo-width-00885 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xb6981526 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo width must be greater than zero. The Vulkan spec states: width must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-width-00885)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-height-00887 (-1513456701)] : Validation Error: [ VUID-VkFramebufferCreateInfo-height-00887 ] Object 0: handle = 0x1b041ce94f8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa5ca7bc3 | vkCreateFramebuffer(): Requested VkFramebufferCreateInfo height must be greater than zero. The Vulkan spec states: height must be greater than 0 (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkFramebufferCreateInfo-height-00887)

examples/runners/ash/src/main.rs

Jasper-Bekkers · 2020-12-04T11:20:37Z

examples/runners/ash/src/main.rs

+    pub fn new(window: winit::window::Window, options: &Options) -> Self {
+        cfg_if::cfg_if! {
+            if #[cfg(target_os = "macos")] {
+                let entry = ash_molten::MoltenEntry::load().unwrap();


It would be nice if somebody could run and test this on macOS before we merge it, mostly would like to see the resize code get tested there.

Linux should get checked again as well.

Jasper-Bekkers · 2020-12-04T11:23:38Z

examples/runners/ash/src/main.rs

+                _ => *control_flow = ControlFlow::Wait,
+            },
+            WindowEvent::Resized(_) => {
+                ctx.recreate_swapchain();


It looks like you're not resizing the swapchain (at least not passing through the new window resolution here)?

There should probably also be a condition around resizing down to a zero sized swapchain, this can happen at least on Windows when minimizing the window, but also when resizing the window to be really tiny.

Both issues here have been addressed. Viewports and scissors are now recreated when the window is resized, and rendering is paused when width or height is 0. Resolution still isn't passed in here though, the window is stored in the RenderCtx so the resolution is retrieved via a method on that.

Unrelated note about this same code location: recreate_swapchain is done both here as well as in response to the out of date error returns during rendering. On Windows, either of these is sufficient but this one takes precedence and produces nicer looking results (no black bars around the rendered portion on the frame drawn during a resize).

Hentropy · 2020-12-07T09:13:42Z

@Jasper-Bekkers Thanks for the review!
I've fixed the validation errors, resizing works properly now, and the multithreading is no longer racy af. Hopefully, I've fixed the Linux crash as well but I don't have a Linux machine to test that.

khyperia · 2020-12-07T09:35:53Z

Linux now works (as in doesn't crash, resizes properly), although there's still some validation errors when resizing the window in floating mode (both of these messages get spammed, here's two examples):

ERROR:
VALIDATION [VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 (2094043421)] : Validation Error: [ VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 ] Object 0: handle = 0x55ecdb5aafb8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x7cd0911d | vkCreateSwapchainKHR() called with imageExtent = (746,553), which is outside the bounds returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR(): currentExtent = (745,553), minImageExtent = (745,553), maxImageExtent = (745,553). The Vulkan spec states: imageExtent must be between minImageExtent and maxImageExtent, inclusive, where minImageExtent and maxImageExtent are members of the VkSurfaceCapabilitiesKHR structure returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR for the surface (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkSwapchainCreateInfoKHR-imageExtent-01274)

ERROR:
VALIDATION [VUID-VkFramebufferCreateInfo-pAttachments-00882 (-2119532822)] : Validation Error: [ VUID-VkFramebufferCreateInfo-pAttachments-00882 ] Object 0: handle = 0x55ecdb5aafb8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x81aa7eea | vkCreateFramebuffer(): VkFramebufferCreateInfo attachment #0 mip level 0 has dimensions smaller than the corresponding framebuffer dimensions. Here are the respective dimensions for attachment #0, framebuffer:
width: 742, 743
height: 549, 549
layerCount: 1, 1

Hentropy · 2020-12-07T10:09:25Z

@khyperia Sweet! Those validation errors should be cleared up now too.

khyperia · 2020-12-07T10:54:56Z

Hmm, looks like the second one got fixed, but the first one is still there.

ERROR:
VALIDATION [VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 (2094043421)] : Validation Error: [ VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 ] Object 0: handle = 0x55ffe0fc9fc8, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x7cd0911d | vkCreateSwapchainKHR() called with imageExtent = (194,178), which is outside the bounds returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR(): currentExtent = (196,178), minImageExtent = (196,178), maxImageExtent = (196,178). The Vulkan spec states: imageExtent must be between minImageExtent and maxImageExtent, inclusive, where minImageExtent and maxImageExtent are members of the VkSurfaceCapabilitiesKHR structure returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR for the surface (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkSwapchainCreateInfoKHR-imageExtent-01274)

There's no startup or shutdown warnings, though, which is hecking great!

rendering wasn't getting unpaused after being set to zero height and then made larger again

Hentropy · 2020-12-08T09:55:28Z

I'm not sure how to fix that last linux validation error. From what I can tell without a linux machine, the OS is resizing the window between the call to vkGetPhysicalDeviceSurfaceCapabilitiesKHR right before creating the swapchain and the second call to vkGetPhysicalDeviceSurfaceCapabilitiesKHR in the validation layer.

This fix still requires the user to set their default GPU to their dedicated one.

Hentropy · 2020-12-08T10:22:19Z

Those last two changes address an issue with wgpu on laptops with iGPU and dGPU. Users will still have to set their default 3d GPU to their dGPU though, or they will get a confusing error: "error X3000: unrecognized identifier shared_ShaderConstants" which is coming from DX12.
This is ready for a second review. Linux and macOS should both get checked again, though that one validation error will still be there.

Jasper-Bekkers

With the new changes it looks like I'm no longer getting validation messages on Windows, one of the discord members (DeanBDean) tested on macOS as well.

Like @khyperia mentioned, it's probably useful from a pedagogical point of view to have this just rely on spirv-builder instead of doing it's own thing since we expect our users to go through spirv-builder as well, but that can be done in a follow up PR.

…-hot-reloading

window resizing and hot reloading

8f67329

khyperia reviewed Dec 2, 2020

View reviewed changes

examples/runners/ash/src/main.rs Outdated Show resolved Hide resolved

examples/runners/ash/src/main.rs Outdated Show resolved Hide resolved

Hentropy added 5 commits December 2, 2020 07:32

remove windows paths and fix viewport creation

c8b0d37

readd spirv tools features

2d6ec24

re-remove features

943e0c6

and add them again

452975d

CI fix

8b2499d

Jasper-Bekkers reviewed Dec 4, 2020

View reviewed changes

proper-resizing-and-validation-fixes

a5c4a5d

linux validation fix

7f71637

fix zero height halting rendering

5200888

rendering wasn't getting unpaused after being set to zero height and then made larger again

Hentropy requested review from VZout and fu5ha as code owners December 8, 2020 09:50

Hentropy added 2 commits December 8, 2020 05:02

fix wgpu crash on laptops with both dedi and integrated GPUs

1b5b44c

This fix still requires the user to set their default GPU to their dedicated one.

METAL or VULKAN backend bit

aa7dd93

khyperia requested a review from Jasper-Bekkers December 9, 2020 09:31

rebuild pipelines after updating shaders

48660ac

Jasper-Bekkers approved these changes Dec 11, 2020

View reviewed changes

Jasper-Bekkers requested a review from khyperia December 11, 2020 11:11

khyperia approved these changes Dec 11, 2020

View reviewed changes

khyperia removed request for VZout and fu5ha December 11, 2020 13:31

Hentropy added 2 commits December 16, 2020 11:00

Merge remote-tracking branch 'upstream/main' into window-resizing-and…

9b6355d

…-hot-reloading

use spirv-builder, resolve merge conflict

e0ae372

fmt

5bebc14

mergify bot merged commit bc4c07e into EmbarkStudios:main Dec 16, 2020

Hentropy deleted the window-resizing-and-hot-reloading branch December 16, 2020 17:45

This was referenced Jan 15, 2021

feat: implemented swap chain recreation #103 #130

Closed

The example panics with ERROR_OUT_OF_DATE_KHR on a window resize. #103

Closed

Conversation

Hentropy commented Dec 2, 2020

Uh oh!

khyperia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jasper-Bekkers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jasper-Bekkers Dec 4, 2020

Choose a reason for hiding this comment

Uh oh!

Hentropy Dec 7, 2020

Choose a reason for hiding this comment

Uh oh!

Jasper-Bekkers Dec 4, 2020

Choose a reason for hiding this comment

Uh oh!

Hentropy Dec 7, 2020

Choose a reason for hiding this comment

Uh oh!

Hentropy commented Dec 7, 2020

Uh oh!

khyperia commented Dec 7, 2020

Uh oh!

Hentropy commented Dec 7, 2020

Uh oh!

khyperia commented Dec 7, 2020

Uh oh!

Hentropy commented Dec 8, 2020

Uh oh!

Hentropy commented Dec 8, 2020

Uh oh!

Jasper-Bekkers left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants