-
Notifications
You must be signed in to change notification settings - Fork 27.4k
Optest is unable to xfail on parametrized tests #112251
Description
It is currently not possible to xfail on a parametrized test that is marked e.g. as
"TestPSRoIAlign.test_faketensor__test_forward[x_dtype0-True-mps]": {
"comment": "AssertionError: Dtypes torch.int64 and torch.int32 are not equal!",
"status": "xfail"
},
in the failures_dict. (I thought #111797 was a fix, but it's not enough)
From what I can tell, the problem is that here
pytorch/torch/testing/_internal/optests/generate_tests.py
Lines 748 to 753 in cf5479b
| def get_status(self, qualname: str, test_name: str) -> str: | |
| if qualname not in self.data: | |
| return "xsuccess" | |
| dct = self.data[qualname] | |
| if test_name not in dct: | |
| return "xsuccess" |
test_name isn't found in dct (i.e. in the failures_dict) because test_name looks like TestPSRoIAlign.test_faketensor__test_forward, i.e. it's missing the [x_dtype0-True-mps] part. That's because this name is generated all the way up in
pytorch/torch/testing/_internal/optests/generate_tests.py
Lines 252 to 254 in cf5479b
| for attr in test_methods: | |
| for prefix, tester in test_utils.items(): | |
| construct_method(attr, prefix, tester) |
where attr is test_forward, but of course it's missing any parametrization info because this is only looking at the "static" test class object.
I'm not sure how to fix this without a significant rework of the whole opcheck logic, but perhaps @zou3519 has an idea?
Unfortunately, for now we cannot really use optests on the torchvision test suite. Note that we can't just put TestPSRoIAlign.test_faketensor__test_forward without the [...] part in the failure_dict, because that test actually passes on cpu and cuda, which would lead to an "unexpected success".
cc @zou3519