I'm very interested in this feature. Does GPU plugin support Import/Export capability?

I'm very interested in this feature. Does GPU plugin support Import/Export capability?

Thanks a lot for your interest in this feature :-) Right now model caching is enabled in 'master' for GNA (PR #4889) and Myriad (PR #4868) plugins, planned to be available in next official release.
For GPU, as far as I know, there is no clear plans to add such model caching support so far

Thank you for reply.

For GPU, as far as I know, there is no clear plans to add such model caching support so far

Are there any reason for that? Do you recommend not to use GPU plugin? Or are there any difficulties around model caching for GPU plugin?

Are there any reason for that? Do you recommend not to use GPU plugin? Or are there any difficulties around model caching for GPU plugin?

Of course, it is not the reason to not use GPU plugin :-) This plugin already has a lot of other useful features for people to use and GPU team have a lot of new features for performance improvement in a pipeline :-)
By the way, can you please share some more information about your use case? Seems like your model takes quite a lot of time to load on GPU and you believe that 'model caching' feature will significantly improve this, right?

@nosovmik Thank you for quick and kind reply.

Seems like your model takes quite a lot of time to load on GPU and you believe that 'model caching' feature will significantly improve this, right?

Yes, I have a custom pose estimation model, and I want to run it in an embedded device (robot). Since CPU resource is limited, hopefully I want to run the model with GPU. But while CPU plugin takes less than 1 second to load the model, GPU plugin takes more than 30 seconds for that. So I want to cache the result of model optimization for GPU plugin.

@ledmonster Did you get this 30 seconds for model loading with kernels cache enabled? Import/Export API is not implemented yet for GPU, but if you enable CACHE_DIR option, then GPU plugin will enable caching of compiled OCL kernels. And since the OCL kernels build is the most expensive stage on model loading, this caching may significantly improve loading time.

if you enable CACHE_DIR option, then GPU plugin will enable caching of compiled OCL kernels.

Thank you. I didn't know that. I'll try it next week.

By the way, I can find that option in API Reference, but it is not listed in GPU Plugin documentation. Is the document missing some config parameters?

By specifying CACHE_DIR like this:

ie.set_config(config={"CACHE_DIR": "/tmp/openvino/gpu/"}, device_name="gpu")

cache files are created on first run, which is about 70MB, and initialization got faster on second run. While it took about 40 sec on first run, on second run, it took only 0.5 sec.

Thank you so much!!

@andrew-zaytsev @ilya-lavrenov Can you please review this PR? Should I add someone else to this review?