Skip to content

Conformance test denylist reduce#25442

Merged
asmorkalov merged 2 commits intoopencv:5.xfrom
Abdurrahheem:ash/comf-denylist-reduce
Apr 18, 2024
Merged

Conformance test denylist reduce#25442
asmorkalov merged 2 commits intoopencv:5.xfrom
Abdurrahheem:ash/comf-denylist-reduce

Conversation

@Abdurrahheem
Copy link
Copy Markdown
Contributor

@Abdurrahheem Abdurrahheem commented Apr 18, 2024

Comment out all passing tests in test_onnx_conformance_layer_filter_opencv_all_denylist.inl.hpp file.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@Abdurrahheem Abdurrahheem added this to the 5.0 milestone Apr 18, 2024
@Abdurrahheem Abdurrahheem self-assigned this Apr 18, 2024
@Abdurrahheem
Copy link
Copy Markdown
Contributor Author

Abdurrahheem commented Apr 18, 2024

@asmorkalov should I completely remove test names (instead of commenting)?

@Abdurrahheem Abdurrahheem marked this pull request as ready for review April 18, 2024 08:36
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 18, 2024

@Abdurrahheem I propose to remove and replace "FAIL" with some failure reason, e.g. exception message or some investigation result. Example: "dynamic shape support is required", "atribute qwerty is not supported for now", etc.

@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 18, 2024 •
edited
Loading

Also, please add reference to dependent PRs.

asmorkalov
asmorkalov approved these changes Apr 18, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@asmorkalov asmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Apr 18, 2024 •
edited
Loading

cc @alexlyulkov
OpenVINO fails list:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_example_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_random_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_example_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_random_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_0_NGRAPH_CPU, where GetParam() = (test_gather_0, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_1_NGRAPH_CPU, where GetParam() = (test_gather_1, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_2d_indices_NGRAPH_CPU, where GetParam() = (test_gather_2d_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_0_NGRAPH_CPU, where GetParam() = (test_gather_elements_0, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_1_NGRAPH_CPU, where GetParam() = (test_gather_elements_1, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_negative_indices_NGRAPH_CPU, where GetParam() = (test_gather_elements_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_negative_indices_NGRAPH_CPU, where GetParam() = (test_gather_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_maxunpool_export_without_output_shape_NGRAPH_CPU, where GetParam() = (test_maxunpool_export_without_output_shape, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_axis_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_duplicate_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_duplicate_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_negative_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_reduction_max_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_reduction_max, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_reduction_min_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_reduction_min, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_without_axis_NGRAPH_CPU, where GetParam() = (test_scatter_elements_without_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_with_axis_NGRAPH_CPU, where GetParam() = (test_scatter_with_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_without_axis_NGRAPH_CPU, where GetParam() = (test_scatter_without_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_NGRAPH_CPU, where GetParam() = (test_scatternd, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_add_NGRAPH_CPU, where GetParam() = (test_scatternd_add, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_max_NGRAPH_CPU, where GetParam() = (test_scatternd_max, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_min_NGRAPH_CPU, where GetParam() = (test_scatternd_min, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_multiply_NGRAPH_CPU, where GetParam() = (test_scatternd_multiply, NGRAPH/CPU)

@asmorkalov asmorkalov merged commit b009a63 into opencv:5.x Apr 18, 2024
@asmorkalov asmorkalov mentioned this pull request Apr 22, 2024
Open
48 tasks
@Abdurrahheem Abdurrahheem mentioned this pull request Apr 23, 2024
Merged
6 tasks
asmorkalov pushed a commit that referenced this pull request Apr 24, 2024
@Abdurrahheem
Merge pull request #25480 from Abdurrahheem:ash/comf-denylist-reduce
bbe86e6 
Add logs of test failure to test_onnx_conformance_layer_filter_opencv_all_denylist.inl.hpp #25480

### Pull Request Readiness Checklist

This PR add logs to test failures to  `test_onnx_conformance_layer_filter_opencv_all_denylist.inl.hpp` and it continuation of #25442

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asmorkalov asmorkalov asmorkalov approved these changes

@dkurt dkurt Awaiting requested review from dkurt

Assignees

@asmorkalov asmorkalov

Labels

test

Projects

None yet

Milestone

5.0-alpha

Development

Successfully merging this pull request may close these issues.

2 participants

@Abdurrahheem @asmorkalov