MulticlassNms/MatrixNms: transformations and CPU implementation by luo-cheng2021 · Pull Request #6653 · openvinotoolkit/openvino

luo-cheng2021 · 2021-07-15T02:06:16Z

Details:

Transformation: convert MulticlassNms/MatrixNms dynamic shape output to static shape
CPU implementation

Tickets:

56900
56916
56917
56901

…o_impl implemented multiclass_nms reference.

TODO: more test cases.

Yi3/ops matrix nms

support dynamic shape in test

…1/openvino into cecilia/multiclass_nms/algo_impl

…o_impl Cecilia/multiclass nms/algo impl

support sort_result_across_batch

luo-cheng2021 · 2021-07-22T06:56:23Z

General
MulticlassNMS and MatrixNMS have many of the same class members and do many the same action.
So maybe better create some NmsBase class and move to it:

Same class members

Extracting ngraph NMS attributes and conversion into mkldnn attributes.

Checks for number of inputs/outpus, precisions and shapes for inputs/outpus

Final sorting according to sort_result attribute

Outputs filling

Yes, it's a nice idea and we'll refactor it in a separate PR.

Question
Are all three outputs required for MulticlassNMS and MatrixNMS?

The output format remains consistent with NonMaxSuppression_5 and the third output is useful when using static shape - can easily get the real valid numbers per batch.

inference-engine/src/mkldnn_plugin/nodes/mkldnn_matrix_nms_node.cpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.cpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.hpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.cpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_matrix_nms_node.cpp

mandrono · 2021-07-23T09:25:03Z

General
MulticlassNMS and MatrixNMS have many of the same class members and do many the same action.
So maybe better create some NmsBase class and move to it:

Same class members

Extracting ngraph NMS attributes and conversion into mkldnn attributes.

Checks for number of inputs/outpus, precisions and shapes for inputs/outpus

Final sorting according to sort_result attribute

Outputs filling

Yes, it's a nice idea and we'll refactor it in a separate PR.

Question
Are all three outputs required for MulticlassNMS and MatrixNMS?

The output format remains consistent with NonMaxSuppression_5 and the third output is useful when using static shape - can easily get the real valid numbers per batch.

could you create ticket for refactoring please?

luo-cheng2021 · 2021-07-26T00:53:15Z

General
MulticlassNMS and MatrixNMS have many of the same class members and do many the same action.
So maybe better create some NmsBase class and move to it:

Same class members

Extracting ngraph NMS attributes and conversion into mkldnn attributes.

Checks for number of inputs/outpus, precisions and shapes for inputs/outpus

Final sorting according to sort_result attribute

Outputs filling

Yes, it's a nice idea and we'll refactor it in a separate PR.

Question
Are all three outputs required for MulticlassNMS and MatrixNMS?

The output format remains consistent with NonMaxSuppression_5 and the third output is useful when using static shape - can easily get the real valid numbers per batch.

could you create ticket for refactoring please?

Created, ticket 61088.

fix bug

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.hpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.cpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_matrix_nms_node.cpp

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.hpp

inference-engine/src/transformations/include/ngraph_ops/nms_static_shape_ie.hpp

...c/transformations/src/transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.cpp

mandrono · 2021-07-28T11:07:04Z

@iefode could you take a look at test part please?

GlebKazantaev

Transformations part looks good!

dmitry-gorokhov

CPU part looks good

…vinotoolkit#6653) * init version, need revise: opset7 * add convert testcase * multiclass_nms support spec * init version * matrixnms support spec * init support for matrix_nms * impl matirx_nms * implemented multiclass_nms reference. TODO: more test cases. * support dynamic shape in test * update to spec 0611 * update to spec 0611 * fixes. * fix: now sort by class_id and score work. * fix clang check error * more test cases verified. * fixes in ref impl. * attribute nms_eta works * test cross_batch and output_type i32. * enable multiclass-nms cpu plugin fallback ngraph * keep topk typo * enable matrix-nms cpu plugin fallback ngraph * support sort_result_across_batch * Add matrix_nms unit test * Add cross batch test cases * fix typo * move multiclass to opset8 * move matrixnms to opset8 * Reference implementations for MulticlassNms and MatrixNms ops * fix name conflict * remove unused var sort_result_across_batch default set to false * avoid float overflow * fix clang check error * info for mac fail * change testcase due to unstable sort * nms add 'normalized' attribute * multiclass cpu test support 'normalized' * nms add 'normalized' attribute * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fix sort order in matrix_nms * fix review comments * add matrix_nms MKLDNN extension layer * parallel in matirx nms * separate filtered_box * separate class_nms result * parallel in class * parallel in batch * partial new nms * partial remove useless function * debug & fix * debug in indexing * fix test cases * remove logging * fix code-style * fix typo * add matrix_nms extension * nms python api * remove unused testcases * refactor transformation * transform dynamic shape to static shape * Update inference-engine/src/transformations/include/ngraph_ops/nms_static_shape_ie.hpp Co-authored-by: Ilya Churaev <ilyachur@gmail.com> * remove register_pass call * [MKLDNN]migrate matrix_nms to MKLDNNNode * bug fix in matrix_nms * padding on matrix_nms * remove logging * test case refine * merged transform_matrix_nms branch * refine matrixnms testcase * multiclass nms cpu plugin implement for static shape, rebased on Reference implementations PR * rebase to new multi-classs transform provided by lc * Name style algin with matrix-nms * static shape padding style to batch inside,new unit test method, real classnum shape * fix format * fix ci error * multi-class NMS modification based on PR reviewer opinion: code format, copyright, delete unused include and funciton way * explicit template instantiation due to mac ci fail * Yi3/fix review (#16) * fix coding style * use parallel_for2d * fix ci fail * unify 'copyright 2021' * mkldnn_multiclass_nms node update based on PR review (#17) * [MKLDNN] apply suggestion for matrix_nms (#18) * fix bug * apply review comments * apply review comments * apply review comments * apply review comments * skip only Nms test, not MatrixNms MulticlassNms test Co-authored-by: Zhang Yi3 <yi3.zhang@intel.com> Co-authored-by: jialipen <cecilia.peng@intel.com> Co-authored-by: mangguo <mang.guo@intel.com> Co-authored-by: Ilya Churaev <ilyachur@gmail.com> Co-authored-by: liubo-intel <bo4.liu@intel.com>

luo-cheng2021 and others added 30 commits June 3, 2021 09:39

init version, need revise: opset7

0fff1e0

add convert testcase

25cceb7

multiclass_nms support spec

9b914c6

init version

c4a1686

matrixnms support spec

6271fdf

init support for matrix_nms

b4d1439

Merge pull request #2 from ceciliapeng2011/cecilia/multiclass_nms/alg…

b62f4ea

…o_impl implemented multiclass_nms reference.

impl matirx_nms

90a94b4

implemented multiclass_nms reference.

972b429

TODO: more test cases.

Merge pull request #3 from luo-cheng2021/yi3/ops_matrix_nms

9c5d6df

Yi3/ops matrix nms

support dynamic shape in test

18cee3e

Merge pull request #4 from luo-cheng2021/yi3/dynamic_shape_test

6527760

support dynamic shape in test

update to spec 0611

8559224

Merge branch 'luocheng/ops_multiclass_nms' into luocheng/ops_matrix_nms

b58a0b0

update to spec 0611

242497a

fixes.

bbbf861

fix: now sort by class_id and score work.

347c146

fix clang check error

a8f6fef

more test cases verified.

02dacff

fixes in ref impl.

7790d83

attribute nms_eta works

db6e9d4

Merge branch 'luocheng/ops_multiclass_nms' of github.com:luo-cheng202…

3b12ce1

…1/openvino into cecilia/multiclass_nms/algo_impl

test cross_batch and output_type i32.

7727aa5

Merge pull request #5 from ceciliapeng2011/cecilia/multiclass_nms/alg…

c740425

…o_impl Cecilia/multiclass nms/algo impl

enable multiclass-nms cpu plugin fallback ngraph

954a363

keep topk typo

d87d2c2

Merge branch 'luocheng/ops_multiclass_nms' into luocheng/ops_matrix_nms

5b0c1e7

enable matrix-nms cpu plugin fallback ngraph

097d801

support sort_result_across_batch

c24deb8

Merge pull request #6 from luo-cheng2021/yi3/sort_result_across_batch

ba95b2d

support sort_result_across_batch

liubo-intel and others added 2 commits July 22, 2021 15:57

mkldnn_multiclass_nms node update based on PR review (#17)

7eaec60

[MKLDNN] apply suggestion for matrix_nms (#18)

51e4600

luo-cheng2021 requested a review from mandrono July 23, 2021 05:00

mandrono reviewed Jul 23, 2021

View reviewed changes

zhangYiIntel and others added 3 commits July 26, 2021 10:57

fix bug

8c69804

Merge pull request #19 from luo-cheng2021/yi3/refactor_matrix_nms

9ebd506

fix bug

apply review comments

c65e8ac

luo-cheng2021 requested a review from mandrono July 26, 2021 03:41

mandrono reviewed Jul 26, 2021

View reviewed changes

apply review comments

9a9ac72

luo-cheng2021 requested a review from mandrono July 27, 2021 01:21

mandrono approved these changes Jul 27, 2021

View reviewed changes

inference-engine/src/mkldnn_plugin/nodes/mkldnn_multiclass_nms.hpp Outdated Show resolved Hide resolved

mandrono assigned dmitry-gorokhov and unassigned mandrono Jul 27, 2021

apply review comments

6a5b677

ilyachur approved these changes Jul 28, 2021

View reviewed changes

apply review comments

591fd14

mandrono requested a review from iefode July 28, 2021 11:07

GlebKazantaev approved these changes Jul 28, 2021

View reviewed changes

luo-cheng2021 added 2 commits July 29, 2021 09:15

Merge remote-tracking branch 'upstream/master' into luocheng/cpu_nms_ops

4b9859a

skip only Nms test, not MatrixNms MulticlassNms test

08745df

dmitry-gorokhov approved these changes Jul 30, 2021

View reviewed changes

ilyachur merged commit 86bb056 into openvinotoolkit:master Jul 30, 2021

Conversation

luo-cheng2021 commented Jul 15, 2021

Details:

Tickets:

Uh oh!

luo-cheng2021 commented Jul 22, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mandrono commented Jul 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luo-cheng2021 commented Jul 26, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mandrono commented Jul 28, 2021

Uh oh!

GlebKazantaev left a comment

Choose a reason for hiding this comment

Uh oh!

dmitry-gorokhov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

mandrono commented Jul 23, 2021 •

edited

Loading