spirv-fuzz: Support pointer types in FuzzerPassAddParameters#3627
spirv-fuzz: Support pointer types in FuzzerPassAddParameters#3627paulthomson merged 19 commits intoKhronosGroup:masterfrom
Conversation
| } else if (type->kind() == opt::analysis::Type::kPointer) { | ||
| // Pointer types with storage class Private and Function are allowed. | ||
| SpvStorageClass storage_class = | ||
| fuzzerutil::GetStorageClassFromPointerType(GetIRContext(), | ||
| type_inst->result_id()); | ||
| // Check if the pointee type is supported. | ||
| auto pointee_type = GetIRContext()->get_type_mgr()->GetType( | ||
| fuzzerutil::GetPointeeTypeIdFromPointerType(GetIRContext(), | ||
| type_inst->result_id())); | ||
| if (TransformationAddParameter::IsParameterTypeSupported(*pointee_type) && | ||
| (storage_class == SpvStorageClassPrivate || | ||
| storage_class == SpvStorageClassFunction)) { | ||
| type_candidates.push_back(type_inst->result_id()); | ||
| } |
There was a problem hiding this comment.
I think we can move this check into TransformationAddParameter::IsParameterTypeSupported.
There was a problem hiding this comment.
We could also add support for Workgroup storage class. It's the same as Private storage class but the initializer is 0. See TransformationAddGlobalVariable docs for more details.
| GetIRContext()->module()->ForEachInst( | ||
| [this, &available_variable_ids, | ||
| current_type_id](opt::Instruction* instruction) { | ||
| if (instruction->opcode() != SpvOpVariable) { | ||
| return; | ||
| } | ||
| if (instruction->type_id() != current_type_id) { | ||
| return; | ||
| } | ||
| available_variable_ids.push_back(instruction->result_id()); | ||
| }); |
There was a problem hiding this comment.
This part needs some refactoring. GetIRContext()->module()->ForEachInst iterates over all instructions in the module. This includes all global OpVariables and all local ones. The part when we iterate over all local variables needs some thought. See #3627 (comment) for more elaborate explanation.
| uint32_t initializer_id = | ||
| FindOrCreateZeroConstant(current_pointee_type_id, true); |
There was a problem hiding this comment.
I'm not sure whether this is a good or a bad practice to mark a variable initializer as irrelevant. @paulthomson, what do you think? In case we decide to make it irrelevant, it would be good to add a comment similar to
We mark the constant as irrelevant so that we can replace it with a more interesting value later.
| ApplyTransformation(TransformationAddLocalVariable( | ||
| GetFuzzerContext()->GetFreshId(), current_type_id, | ||
| function.result_id(), initializer_id, true)); |
There was a problem hiding this comment.
This will create a local OpVariable instruction in the function. This is somewhat tricky.
The initial idea is that the created variable will be used as an operand to OpFunctionCall instructions. These instructions can belong to different functions - we'll call them callers - (since we can call any function from any other as long as there are no recursion - SPIR-V doesn't support that). Thus, this piece of code should create a local variable in every caller of function.result_id(). This is somewhat tricky because some callers might already have a suitable local variable.
I propose to refactor this fuzzer pass as follows:
- Check the storage class of the variable. If it's
PrivateorWorkgroup, either find a global variable (IRContextmust have a method that returns only global instructions) or create a new one (as you do above). Otherwise... - Iterate over all callers of
function.result_id()(seefuzzerutil::GetCallers) and for each one either find a suitable local variable or create one.
I would also suggest to change the protobuf file to pass a repeated field of UInt32Pair (see fuzzerutil::Repeated*ToMap and fuzzerutil::MapToRepeated*). This field would represent a map from the result id of the caller to the result id of the local variable in that caller.
You would also need to adjust IsApplicable method in the transformation class to check that each caller has a local variable (if the added parameter is indeed a pointer with Function storage class). Hope this makes sense.
There was a problem hiding this comment.
I was thinking about the adding of a map to a protobuf file and for me it seems unneccessary. I came up with two reasons:
-
It would mean refactoring the code in cases where AddParameter doesn't add a pointer with Function storage class.
-
In the case where AddParameter is about to add the pointer with Function storage class, the information whether the transformation is applicable or not is already accessible, since we pass function_id. And as you said, we could iterate over all callers and check if there is a local variable in each calling function. Even if we pass the map in the protobuf we still have to recreate it in the IsApplicable method by iterating over all callers to check whether the resulting two maps are equal. For example, even if we have checked that each key in the passed protobuf map refers to a valid local variable, we might have missed some callers. The actual check is in iterating over the callers in IsApplicable method.
There was a problem hiding this comment.
Yes, you are right. It is indeed might be better to simply check in the transformation that the required parameter is available and in the fuzzer pass to ensure that the parameter is available. In this case, I would've created two functions in the fuzzerutil: MaybeGetLocalVariable and MaybeGetGlobalVariable (or some equivalent of these) that return result id of the local/global variable or 0 if no such variable exists.
You could then pass the type of the newly created parameter to the transformation and check that the variable (or the constant) of the required type exists.
There is one caveat, though. Currently, we do the opposite: we pass the value of the parameter and infer parameter's type out of it. You will need to change that.
There was a problem hiding this comment.
The function-creation part is up to you, though.
| ApplyTransformation(TransformationAddParameter( | ||
| function.result_id(), GetFuzzerContext()->GetFreshId(), | ||
| initializer_id, GetFuzzerContext()->GetFreshId())); |
There was a problem hiding this comment.
initializer_id is used to initialize the OpVariable instruction. There is also an initializer_id argument in the transformation's constructor that is used to initialize an operand in OpFunctionCall instruction. In our case, the latter is the result id of the OpVariable.
Sorry, it's a bit confusing but the bottom-line is that you should use the result id of the variable here instead of the initializer_id.
Vasniktel
left a comment
There was a problem hiding this comment.
You have the right idea about finding variables or creating them if none was found. The implementation needs some refactoring, though. Also, don't forget to add some tests in the end.
|
@antnkarp, you'll probably need to rebase since I merged some changes to the |
f48ae62 to
f692704
Compare
Vasniktel
left a comment
There was a problem hiding this comment.
Nice job! I've requested some changes in the comments.
| // It may not be OpTypeVoid. In case of new parameter being a pointer, this is the type id | ||
| // of the pointer type. Initializer id and pointer type id are mutually exclusive. | ||
| uint32 initializer_id_or_pointer_type_id = 3; |
There was a problem hiding this comment.
It would be better to make this just argument_type_id that accepts a type id of the argument. In this case, you would need to validate that the type is supported by the transformation and:
- If type is a scalar or a composite - check that there is an irrelevant constant in the module (
fuzzerutil::MaybeGetZeroConstant) - If it is a pointer - check that a suitable variable is present in the module (we haven't decided yet on what facts the variable should posses)
There was a problem hiding this comment.
Or maybe parameter_type_id to be consistent with the naming.
source/fuzz/fuzzer_util.h
Outdated
| // Returns the id of an available local variable that has the type |type_id| and | ||
| // is declared in the function that has the id |function_id|. If no such | ||
| // variable is available, returns 0. | ||
| uint32_t MaybeGetLocalVariable(opt::IRContext* ir_context, uint32_t type_id, | ||
| uint32_t function_id); | ||
|
|
||
| // Returns the id of an available global variable that has the type |type_id|. | ||
| // If no such variable is available, returns 0. | ||
| uint32_t MaybeGetGlobalVariable(opt::IRContext* ir_context, uint32_t type_id); |
There was a problem hiding this comment.
It would be good to add two boolean parameters to these functions:
pointee_value_is_irrelevantid_is_irrelevant
and make sure the the returned id is marked (or not marked depending on the value of parameters) with those two facts accordingly.
There was a problem hiding this comment.
Also, mention these parameters in the doc.
source/fuzz/fuzzer_util.cpp
Outdated
| } | ||
| } // namespace fuzzerutil |
source/fuzz/fuzzer_util.h
Outdated
| // Returns the id of an available local variable that has the type |type_id| and | ||
| // is declared in the function that has the id |function_id|. If no such | ||
| // variable is available, returns 0. | ||
| uint32_t MaybeGetLocalVariable(opt::IRContext* ir_context, uint32_t type_id, | ||
| uint32_t function_id); |
There was a problem hiding this comment.
Mention that type_id is required to be a result id of OpTypePointer instruction with Function storage class.
source/fuzz/fuzzer_util.h
Outdated
| // Returns the id of an available global variable that has the type |type_id|. | ||
| // If no such variable is available, returns 0. | ||
| uint32_t MaybeGetGlobalVariable(opt::IRContext* ir_context, uint32_t type_id); |
There was a problem hiding this comment.
Mention that type_id is a result id of an OpTypePointer instruction with Private or Workgroup storage class.
| function.result_id(), GetFuzzerContext()->GetFreshId(), | ||
| current_type_id, GetFuzzerContext()->GetFreshId())); | ||
|
|
||
| } else // Consider non-pointer parameters. |
There was a problem hiding this comment.
You can move the comment into the scope like so:
} else {
// comment| if (initializer_type->kind() == opt::analysis::Type::kPointer) { | ||
| uint32_t pointer_type_id = message_.initializer_id_or_pointer_type_id(); | ||
| auto storage_class = | ||
| fuzzerutil::GetStorageClassFromPointerType(ir_context, pointer_type_id); |
There was a problem hiding this comment.
Consider using opt::analysis::Type::AsPointer as I've described in the fuzzer pass.
| is_valid = false; | ||
| break; |
There was a problem hiding this comment.
You can get rid of is_valid and return false directly.
| case opt::analysis::Type::kPointer: { | ||
| auto storage_class = type.AsPointer()->storage_class(); | ||
| switch (storage_class) { | ||
| case SpvStorageClassPrivate: | ||
| case SpvStorageClassFunction: | ||
| case SpvStorageClassWorkgroup: { | ||
| auto pointee_type = type.AsPointer()->pointee_type(); | ||
| return IsParameterTypeSupported(*pointee_type); | ||
| } | ||
| default: | ||
| return false; | ||
| } | ||
| } |
| default: | ||
| break; |
There was a problem hiding this comment.
It would be good to make an assert here saying that storage class is not supported.
|
I think it will be good to make a design doc for this change and we can all clarify some things there before we proceed! |
|
@antnkarp, please change |
This reverts commit ac06f97.
f692704 to
b833c49
Compare
Vasniktel
left a comment
There was a problem hiding this comment.
Your approach looks good.
However, I would rather have a separate field with the parameter's type id than the current solution.
Also, feel free to resolve obsolete review comments.
paulthomson
left a comment
There was a problem hiding this comment.
I just added a few comments, mostly to reply to Vasyl's comments and point out some style issues. Please also check the design doc before addressing comments, as I think Vasyl is right that we should add a protobuf field for the type id of the new parameter.
paulthomson
left a comment
There was a problem hiding this comment.
Nice! It is getting very close!
Vasniktel
left a comment
There was a problem hiding this comment.
This looks great! Thanks for your hard work on this one. I left some minor refactoring comments.
Vasniktel
left a comment
There was a problem hiding this comment.
Thanks for your hard work on this one!
paulthomson
left a comment
There was a problem hiding this comment.
Awesome! Thanks a lot for your work on this.
…Group#3627) For FuzzerPassAddParameters, adds pointer types (that have the storage class Function or Private) to the pool of available types for new parameters. If there are no variables of the chosen pointer type, it invokes TransformationAddLocalVariable / TransformationAddGlobalVariable to add one. Part of KhronosGroup#3403
Roll third_party/glslang/ f257e0e..983698b (2 commits) KhronosGroup/glslang@f257e0e...983698b $ git log f257e0e..983698b --date=short --no-merges --format='%ad %ae %s' 2020-08-23 john Revert "Merge pull request KhronosGroup#2371 from RafaelMarinheiro/master" 2020-08-21 julius.ikkala Obey ENABLE_PCH CMake option Created with: roll-dep third_party/glslang Roll third_party/googletest/ adeef1929..1e315c5b1 (14 commits) google/googletest@adeef19...1e315c5 $ git log adeef1929..1e315c5b1 --date=short --no-merges --format='%ad %ae %s' 2020-08-20 absl-team Googletest export 2020-08-17 absl-team Googletest export 2020-08-12 robert.earhart Export LICENSE 2020-08-03 amatanhead Remove ThrowsMessageHasSubstr and fix some nits after review 2020-07-13 amatanhead Cleanup a bulky expression, document implementation details 2020-07-07 amatanhead Fix build under msvc 2020-07-07 amatanhead Update tests after changing an error message 2020-07-07 amatanhead Fix build under msvc 2020-07-07 amatanhead Add a test to ensure that the `Throws` matcher only invokes its argument once. 2020-07-07 amatanhead Add a test for duplicate catch clauses in throw matchers, fix a couple of nitpicks. 2020-07-03 amatanhead Add missing documentation piece 2020-06-20 amatanhead Small improvements: code style and property name 2020-06-20 amatanhead Add matchers for testing exception properties 2018-05-01 lantw44 Avoid using environ on FreeBSD Created with: roll-dep third_party/googletest Roll third_party/spirv-cross/ 4c7944bb4..685f86471 (6 commits) KhronosGroup/SPIRV-Cross@4c7944b...685f864 $ git log 4c7944bb4..685f86471 --date=short --no-merges --format='%ad %ae %s' 2020-08-24 post Run format_all.sh. 2020-08-24 post Work around annoying warning on GCC 10.2. 2020-08-21 post Overhaul how we deal with reserved identifiers. 2020-08-20 post HLSL: Fix FragCoord.w. 2020-08-20 post HLSL: Deal with partially filled 16-byte word in cbuffers. 2020-08-20 post HLSL: Fix bug in is_packing_standard for cbuffer. Created with: roll-dep third_party/spirv-cross Roll third_party/spirv-tools/ b8de4f5..4dd1223 (9 commits) KhronosGroup/SPIRV-Tools@b8de4f5...4dd1223 $ git log b8de4f5..4dd1223 --date=short --no-merges --format='%ad %ae %s' 2020-08-21 andreperezmaselco.developer spirv-fuzz: Add words instead of logical operands (KhronosGroup#3728) 2020-08-20 dnovillo CCP should mark IR changed if it created new constants. (KhronosGroup#3732) 2020-08-19 antonikarp spirv-fuzz: add FuzzerPassAddCompositeInserts (KhronosGroup#3606) 2020-08-19 antonikarp spirv-fuzz: Support pointer types in FuzzerPassAddParameters (KhronosGroup#3627) 2020-08-18 jaebaek Let ADCE pass check DebugScope (KhronosGroup#3703) 2020-08-18 andreperezmaselco.developer spirv-opt: Implement opt::Function::HasEarlyReturn function (KhronosGroup#3711) 2020-08-17 andreperezmaselco.developer spirv-fuzz: Check termination instructions when donating modules (KhronosGroup#3710) 2020-08-17 jackoalan Fix -Wrange-loop-analysis warning (KhronosGroup#3712) 2020-08-17 andreperezmaselco.developer spirv-fuzz: Check header dominance when adding dead block (KhronosGroup#3694) Created with: roll-dep third_party/spirv-tools
For FuzzerPassAddParameters, adds pointer types (that have the storage
class Function or Private) to the pool of available types for new
parameters. If there are no variables of the chosen pointer type, it
invokes TransformationAddLocalVariable / TransformationAddGlobalVariable
to add one.
Part of #3403