Skip to content

spirv-fuzz: Support pointer types in FuzzerPassAddParameters#3627

Merged
paulthomson merged 19 commits intoKhronosGroup:masterfrom
antonikarp:3403_support_pointer_types
Aug 19, 2020
Merged

spirv-fuzz: Support pointer types in FuzzerPassAddParameters#3627
paulthomson merged 19 commits intoKhronosGroup:masterfrom
antonikarp:3403_support_pointer_types

Conversation

@antonikarp
Copy link
Copy Markdown
Collaborator

@antonikarp antonikarp commented Jul 31, 2020

For FuzzerPassAddParameters, adds pointer types (that have the storage
class Function or Private) to the pool of available types for new
parameters. If there are no variables of the chosen pointer type, it
invokes TransformationAddLocalVariable / TransformationAddGlobalVariable
to add one.

Part of #3403

Comment on lines +46 to +59
} else if (type->kind() == opt::analysis::Type::kPointer) {
// Pointer types with storage class Private and Function are allowed.
SpvStorageClass storage_class =
fuzzerutil::GetStorageClassFromPointerType(GetIRContext(),
type_inst->result_id());
// Check if the pointee type is supported.
auto pointee_type = GetIRContext()->get_type_mgr()->GetType(
fuzzerutil::GetPointeeTypeIdFromPointerType(GetIRContext(),
type_inst->result_id()));
if (TransformationAddParameter::IsParameterTypeSupported(*pointee_type) &&
(storage_class == SpvStorageClassPrivate ||
storage_class == SpvStorageClassFunction)) {
type_candidates.push_back(type_inst->result_id());
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can move this check into TransformationAddParameter::IsParameterTypeSupported.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also add support for Workgroup storage class. It's the same as Private storage class but the initializer is 0. See TransformationAddGlobalVariable docs for more details.

Comment on lines +106 to +116
GetIRContext()->module()->ForEachInst(
[this, &available_variable_ids,
current_type_id](opt::Instruction* instruction) {
if (instruction->opcode() != SpvOpVariable) {
return;
}
if (instruction->type_id() != current_type_id) {
return;
}
available_variable_ids.push_back(instruction->result_id());
});
Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel Jul 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part needs some refactoring. GetIRContext()->module()->ForEachInst iterates over all instructions in the module. This includes all global OpVariables and all local ones. The part when we iterate over all local variables needs some thought. See #3627 (comment) for more elaborate explanation.

Comment on lines +117 to +118
uint32_t initializer_id =
FindOrCreateZeroConstant(current_pointee_type_id, true);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this is a good or a bad practice to mark a variable initializer as irrelevant. @paulthomson, what do you think? In case we decide to make it irrelevant, it would be good to add a comment similar to

We mark the constant as irrelevant so that we can replace it with a more interesting value later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it works.

Comment on lines +128 to +130
ApplyTransformation(TransformationAddLocalVariable(
GetFuzzerContext()->GetFreshId(), current_type_id,
function.result_id(), initializer_id, true));
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create a local OpVariable instruction in the function. This is somewhat tricky.

The initial idea is that the created variable will be used as an operand to OpFunctionCall instructions. These instructions can belong to different functions - we'll call them callers - (since we can call any function from any other as long as there are no recursion - SPIR-V doesn't support that). Thus, this piece of code should create a local variable in every caller of function.result_id(). This is somewhat tricky because some callers might already have a suitable local variable.

I propose to refactor this fuzzer pass as follows:

  1. Check the storage class of the variable. If it's Private or Workgroup, either find a global variable (IRContext must have a method that returns only global instructions) or create a new one (as you do above). Otherwise...
  2. Iterate over all callers of function.result_id() (see fuzzerutil::GetCallers) and for each one either find a suitable local variable or create one.

I would also suggest to change the protobuf file to pass a repeated field of UInt32Pair (see fuzzerutil::Repeated*ToMap and fuzzerutil::MapToRepeated*). This field would represent a map from the result id of the caller to the result id of the local variable in that caller.

You would also need to adjust IsApplicable method in the transformation class to check that each caller has a local variable (if the added parameter is indeed a pointer with Function storage class). Hope this makes sense.

Copy link
Copy Markdown
Collaborator Author

@antonikarp antonikarp Aug 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about the adding of a map to a protobuf file and for me it seems unneccessary. I came up with two reasons:

  1. It would mean refactoring the code in cases where AddParameter doesn't add a pointer with Function storage class.

  2. In the case where AddParameter is about to add the pointer with Function storage class, the information whether the transformation is applicable or not is already accessible, since we pass function_id. And as you said, we could iterate over all callers and check if there is a local variable in each calling function. Even if we pass the map in the protobuf we still have to recreate it in the IsApplicable method by iterating over all callers to check whether the resulting two maps are equal. For example, even if we have checked that each key in the passed protobuf map refers to a valid local variable, we might have missed some callers. The actual check is in iterating over the callers in IsApplicable method.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. It is indeed might be better to simply check in the transformation that the required parameter is available and in the fuzzer pass to ensure that the parameter is available. In this case, I would've created two functions in the fuzzerutil: MaybeGetLocalVariable and MaybeGetGlobalVariable (or some equivalent of these) that return result id of the local/global variable or 0 if no such variable exists.

You could then pass the type of the newly created parameter to the transformation and check that the variable (or the constant) of the required type exists.

There is one caveat, though. Currently, we do the opposite: we pass the value of the parameter and infer parameter's type out of it. You will need to change that.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function-creation part is up to you, though.

Comment on lines +134 to +136
ApplyTransformation(TransformationAddParameter(
function.result_id(), GetFuzzerContext()->GetFreshId(),
initializer_id, GetFuzzerContext()->GetFreshId()));
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initializer_id is used to initialize the OpVariable instruction. There is also an initializer_id argument in the transformation's constructor that is used to initialize an operand in OpFunctionCall instruction. In our case, the latter is the result id of the OpVariable.

Sorry, it's a bit confusing but the bottom-line is that you should use the result id of the variable here instead of the initializer_id.

Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have the right idea about finding variables or creating them if none was found. The implementation needs some refactoring, though. Also, don't forget to add some tests in the end.

@Vasniktel
Copy link
Copy Markdown
Collaborator

Vasniktel commented Aug 4, 2020

@antnkarp, you'll probably need to rebase since I merged some changes to the TransformationAddParameters in #3625. Let me know if you have any questions.

@antonikarp antonikarp force-pushed the 3403_support_pointer_types branch 2 times, most recently from f48ae62 to f692704 Compare August 11, 2020 08:02
Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job! I've requested some changes in the comments.

Comment on lines +758 to +760
// It may not be OpTypeVoid. In case of new parameter being a pointer, this is the type id
// of the pointer type. Initializer id and pointer type id are mutually exclusive.
uint32 initializer_id_or_pointer_type_id = 3;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to make this just argument_type_id that accepts a type id of the argument. In this case, you would need to validate that the type is supported by the transformation and:

  1. If type is a scalar or a composite - check that there is an irrelevant constant in the module (fuzzerutil::MaybeGetZeroConstant)
  2. If it is a pointer - check that a suitable variable is present in the module (we haven't decided yet on what facts the variable should posses)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe parameter_type_id to be consistent with the naming.

Comment on lines +496 to +504
// Returns the id of an available local variable that has the type |type_id| and
// is declared in the function that has the id |function_id|. If no such
// variable is available, returns 0.
uint32_t MaybeGetLocalVariable(opt::IRContext* ir_context, uint32_t type_id,
uint32_t function_id);

// Returns the id of an available global variable that has the type |type_id|.
// If no such variable is available, returns 0.
uint32_t MaybeGetGlobalVariable(opt::IRContext* ir_context, uint32_t type_id);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to add two boolean parameters to these functions:

  1. pointee_value_is_irrelevant
  2. id_is_irrelevant

and make sure the the returned id is marked (or not marked depending on the value of parameters) with those two facts accordingly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, mention these parameters in the doc.

Comment on lines +1385 to +1345
}
} // namespace fuzzerutil
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a new line.

Comment on lines +496 to +500
// Returns the id of an available local variable that has the type |type_id| and
// is declared in the function that has the id |function_id|. If no such
// variable is available, returns 0.
uint32_t MaybeGetLocalVariable(opt::IRContext* ir_context, uint32_t type_id,
uint32_t function_id);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention that type_id is required to be a result id of OpTypePointer instruction with Function storage class.

Comment on lines +502 to +504
// Returns the id of an available global variable that has the type |type_id|.
// If no such variable is available, returns 0.
uint32_t MaybeGetGlobalVariable(opt::IRContext* ir_context, uint32_t type_id);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention that type_id is a result id of an OpTypePointer instruction with Private or Workgroup storage class.

function.result_id(), GetFuzzerContext()->GetFreshId(),
current_type_id, GetFuzzerContext()->GetFreshId()));

} else // Consider non-pointer parameters.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can move the comment into the scope like so:

} else {
  // comment

Comment on lines +74 to +77
if (initializer_type->kind() == opt::analysis::Type::kPointer) {
uint32_t pointer_type_id = message_.initializer_id_or_pointer_type_id();
auto storage_class =
fuzzerutil::GetStorageClassFromPointerType(ir_context, pointer_type_id);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using opt::analysis::Type::AsPointer as I've described in the fuzzer pass.

Comment on lines +88 to +89
is_valid = false;
break;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can get rid of is_valid and return false directly.

Comment on lines +252 to +220
case opt::analysis::Type::kPointer: {
auto storage_class = type.AsPointer()->storage_class();
switch (storage_class) {
case SpvStorageClassPrivate:
case SpvStorageClassFunction:
case SpvStorageClassWorkgroup: {
auto pointee_type = type.AsPointer()->pointee_type();
return IsParameterTypeSupported(*pointee_type);
}
default:
return false;
}
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Comment on lines +101 to +102
default:
break;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to make an assert here saying that storage class is not supported.

@paulthomson
Copy link
Copy Markdown
Contributor

I think it will be good to make a design doc for this change and we can all clarify some things there before we proceed!

@Vasniktel
Copy link
Copy Markdown
Collaborator

Vasniktel commented Aug 13, 2020

@antnkarp, please change Fixes #... to Part of #... in the PR description so that the related issue is not closed when the PR is merged.

@antonikarp antonikarp force-pushed the 3403_support_pointer_types branch from f692704 to b833c49 Compare August 14, 2020 16:06
Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your approach looks good.

However, I would rather have a separate field with the parameter's type id than the current solution.

Also, feel free to resolve obsolete review comments.

Copy link
Copy Markdown
Contributor

@paulthomson paulthomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added a few comments, mostly to reply to Vasyl's comments and point out some style issues. Please also check the design doc before addressing comments, as I think Vasyl is right that we should add a protobuf field for the type id of the new parameter.

Copy link
Copy Markdown
Contributor

@paulthomson paulthomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! It is getting very close!

Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Thanks for your hard work on this one. I left some minor refactoring comments.

Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Copy link
Copy Markdown
Collaborator

@Vasniktel Vasniktel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your hard work on this one!

Copy link
Copy Markdown
Contributor

@paulthomson paulthomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks a lot for your work on this.

@paulthomson paulthomson changed the title spirv-fuzz: Add support for pointer types to FuzzerPassAddParameters spirv-fuzz: Support pointer types in FuzzerPassAddParameters Aug 19, 2020
@paulthomson paulthomson merged commit 582c276 into KhronosGroup:master Aug 19, 2020
dnovillo pushed a commit to dnovillo/SPIRV-Tools that referenced this pull request Aug 19, 2020
…Group#3627)

For FuzzerPassAddParameters, adds pointer types (that have the storage
class Function or Private) to the pool of available types for new
parameters. If there are no variables of the chosen pointer type, it
invokes TransformationAddLocalVariable / TransformationAddGlobalVariable
to add one.

Part of KhronosGroup#3403
dneto0 pushed a commit to dneto0/SPIRV-Tools that referenced this pull request Sep 14, 2024
Roll third_party/glslang/ f257e0e..983698b (2 commits)

KhronosGroup/glslang@f257e0e...983698b

$ git log f257e0e..983698b --date=short --no-merges --format='%ad %ae %s'
2020-08-23 john Revert "Merge pull request KhronosGroup#2371 from RafaelMarinheiro/master"
2020-08-21 julius.ikkala Obey ENABLE_PCH CMake option

Created with:
  roll-dep third_party/glslang

Roll third_party/googletest/ adeef1929..1e315c5b1 (14 commits)

google/googletest@adeef19...1e315c5

$ git log adeef1929..1e315c5b1 --date=short --no-merges --format='%ad %ae %s'
2020-08-20 absl-team Googletest export
2020-08-17 absl-team Googletest export
2020-08-12 robert.earhart Export LICENSE
2020-08-03 amatanhead Remove ThrowsMessageHasSubstr and fix some nits after review
2020-07-13 amatanhead Cleanup a bulky expression, document implementation details
2020-07-07 amatanhead Fix build under msvc
2020-07-07 amatanhead Update tests after changing an error message
2020-07-07 amatanhead Fix build under msvc
2020-07-07 amatanhead Add a test to ensure that the `Throws` matcher only invokes its argument once.
2020-07-07 amatanhead Add a test for duplicate catch clauses in throw matchers, fix a couple of nitpicks.
2020-07-03 amatanhead Add missing documentation piece
2020-06-20 amatanhead Small improvements: code style and property name
2020-06-20 amatanhead Add matchers for testing exception properties
2018-05-01 lantw44 Avoid using environ on FreeBSD

Created with:
  roll-dep third_party/googletest

Roll third_party/spirv-cross/ 4c7944bb4..685f86471 (6 commits)

KhronosGroup/SPIRV-Cross@4c7944b...685f864

$ git log 4c7944bb4..685f86471 --date=short --no-merges --format='%ad %ae %s'
2020-08-24 post Run format_all.sh.
2020-08-24 post Work around annoying warning on GCC 10.2.
2020-08-21 post Overhaul how we deal with reserved identifiers.
2020-08-20 post HLSL: Fix FragCoord.w.
2020-08-20 post HLSL: Deal with partially filled 16-byte word in cbuffers.
2020-08-20 post HLSL: Fix bug in is_packing_standard for cbuffer.

Created with:
  roll-dep third_party/spirv-cross

Roll third_party/spirv-tools/ b8de4f5..4dd1223 (9 commits)

KhronosGroup/SPIRV-Tools@b8de4f5...4dd1223

$ git log b8de4f5..4dd1223 --date=short --no-merges --format='%ad %ae %s'
2020-08-21 andreperezmaselco.developer spirv-fuzz: Add words instead of logical operands (KhronosGroup#3728)
2020-08-20 dnovillo CCP should mark IR changed if it created new constants. (KhronosGroup#3732)
2020-08-19 antonikarp spirv-fuzz: add FuzzerPassAddCompositeInserts (KhronosGroup#3606)
2020-08-19 antonikarp spirv-fuzz: Support pointer types in FuzzerPassAddParameters (KhronosGroup#3627)
2020-08-18 jaebaek Let ADCE pass check DebugScope (KhronosGroup#3703)
2020-08-18 andreperezmaselco.developer spirv-opt: Implement opt::Function::HasEarlyReturn function (KhronosGroup#3711)
2020-08-17 andreperezmaselco.developer spirv-fuzz: Check termination instructions when donating modules (KhronosGroup#3710)
2020-08-17 jackoalan Fix -Wrange-loop-analysis warning (KhronosGroup#3712)
2020-08-17 andreperezmaselco.developer spirv-fuzz: Check header dominance when adding dead block (KhronosGroup#3694)

Created with:
  roll-dep third_party/spirv-tools
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants