Skip to content

Switch to use embedded reflection#250

Merged
kpet merged 9 commits intokpet:masterfrom
alan-baker:clspv-reflection
Aug 14, 2020
Merged

Switch to use embedded reflection#250
kpet merged 9 commits intokpet:masterfrom
alan-baker:clspv-reflection

Conversation

@alan-baker
Copy link
Copy Markdown
Contributor

Change clvk to use clspv's embedded reflection in the SPIR-V binary.

  • Offline and online compilation now share parsing of reflection information
  • "binary" returned by clvk no longer contains the descriptor map
    • left the magic number to minimize overall number of changes
  • if the Vulkan implementation does not support VK_KHR_non_semantic_info then the reflection information is stripped after parsing
    • the stripped binary is stored separately so that the binary with embedded reflection may be returned for clGetProgramInfo
  • update submodules

* Descriptor maps for both online and offline compiles use embedded
  reflection information
  * replaced old parsers with a single unified parser
* If VK_KHR_non_semantic_info is not supported, the reflection is
  stripped from the binary before creating a shader module
* Descriptor maps are no longer written to saved binaries
* Add clspv include dirs for both modes
* Don't pass -desriptormap to clspv anymore
* update spirv-llvm-translator deps
* Update clspv
* Update SPIRV-LLVM-Translator (also point at KhronosGroup)
Copy link
Copy Markdown
Owner

@kpet kpet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks generally great! A few observations:

  • Changing the interface between clspv and clvk is now harder. One has to change the non semantic instruction set specification in the registry first, generate updated headers, update clspv and finally clvk. Changes are not that frequent so I think it's worth trading the added complexity for a cleaner interface.
  • I've pulled this change locally and run a CTS sweep using an offline build. test_compiler is crashing and write_image_{float,char,uchar} have regressed.
  • We'll probably want to do more renaming/refactoring but that can happen after this change.
  • Going forward we'll probably want clvk binaries to just be SPIR-V modules. I can't think of anything that couldn't be added to the SPIR-V at the moment. This would make offline compilation much easier as one could use clspv directly.

m_dmaps[entry.kernel_arg_data.kernel_name].push_back(arg);
spvtools::Optimizer opt(m_target_env);
opt.SetMessageConsumer(consumer);
opt.RegisterPass(spvtools::CreateStripReflectInfoPass());
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the pass factory should have had "clspv" in the name to be consistent with the instruction set name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reflection stripping pass existed before my instruction set and handles non-semantic instructions generally and also the HLSL reflection instructions and decorations.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, that's a bit counter-intuitive. I'm wondering whether generic stripping of all non-semantic instructions shouldn't be a separate pass. I'm not familiar enough with the HLSL bits to have an informed opinion though.

@alan-baker
Copy link
Copy Markdown
Contributor Author

I'll take a look at the CTS failures.

We'll probably want to do more renaming/refactoring but that can happen after this change.

Agreed, but I wanted to keep this change as minimal as reasonably possible.

@alan-baker
Copy link
Copy Markdown
Contributor Author

I believe the SPIRV-Tools linker/optimizer is having problems with the extended instructions. Need to investigate how to fix that.

@alan-baker
Copy link
Copy Markdown
Contributor Author

I believe the SPIRV-Tools linker/optimizer is having problems with the extended instructions. Need to investigate how to fix that.

SPIRV-Tools optimizer representation is reordering the module and producing an invalid binary.

* Update SPIRV-Tools to include fix handling of non-semantic
  instructions in the optimizer
* Improving naming
* Use extension name macro instead of raw string
@alan-baker alan-baker requested a review from kpet August 13, 2020 18:57
@alan-baker
Copy link
Copy Markdown
Contributor Author

The last push fixes the compiler conformance test crash. All those tests match the checked in results. With that and the sampler fix, CTS should be good now.

@kpet
Copy link
Copy Markdown
Owner

kpet commented Aug 14, 2020

All the usual CTS tests are now passing. All looking good, thanks!

@kpet kpet merged commit d118090 into kpet:master Aug 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants