Skip to content

[tools] new api and support dynamic models in python benchmark app#8582

Merged
akuporos merged 246 commits intoopenvinotoolkit:masterfrom
AlexeyLebedev1:alexeyl1/tools/update_benchmark
Dec 2, 2021
Merged

[tools] new api and support dynamic models in python benchmark app#8582
akuporos merged 246 commits intoopenvinotoolkit:masterfrom
AlexeyLebedev1:alexeyl1/tools/update_benchmark

Conversation

@AlexeyLebedev1
Copy link
Copy Markdown
Contributor

@AlexeyLebedev1 AlexeyLebedev1 commented Nov 14, 2021

Tickets:

  • 68569

Details:

  • rewrite benchmark app with API 2.0.
  • support dynamic models.

Support dynamic models includes:

  • -shape parameter can be dynamic
  • -data_shape parameter represents input data shape
  • specifying -data_shape parameter isn't required in case image input, images will be processed with original shapes
  • -batch_size can be dynamic
  • -pcseq tool reports performance for each shape (group of shapes in case multi input network)
  • -legacy_mode - enable legacy scenario with inputs filling only once before measurements, default for static case
  • -full mode - include inputs filling into loop measurement, default for dynamic models.
  • Max/Avg/Min latency for both cases - dynamic and static, median latency is valid for static case only

Some changes in benchmark app:

  • Number of processed input files doesn't depend on number of infer requests anymore. Max number of images that is multiple number of data shapes will be used.
  • User able to pass path to directory in input file mapping.

nosovmik and others added 30 commits September 6, 2021 21:18
Includes API definition for trivial mean/scale operations (which don't require layout)

Mean/scale with 'layout' support will be done under separate task together
 with Layout

Current test code coverage: 100%
@AlexeyLebedev1 AlexeyLebedev1 requested a review from a team November 28, 2021 17:24
@AlexeyLebedev1 AlexeyLebedev1 force-pushed the alexeyl1/tools/update_benchmark branch from af1400e to a49f4f5 Compare December 1, 2021 16:16
@akuporos akuporos merged commit f9bd740 into openvinotoolkit:master Dec 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: Core OpenVINO Core (aka ngraph) category: inference OpenVINO Runtime library - Inference category: tools OpenVINO C++ / Python tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.