Inference engine as a static library by emmanuelattia · Pull Request #3219 · openvinotoolkit/openvino

emmanuelattia · 2020-11-19T09:46:27Z

This PR is currently a draft.

The objective is:

Allow everything to be built in a possibly self contained static library
Add mechanism for pluging (DNN modules, readers, interpreters) to register themselves statically

Should inference engine expose a function to register manually that is done in some static object constructor in the plugins ?
Or should inference engine be aware of the available plugins when static flag is enabled ?

Add enough CMakefile options to be able to choose included plugins in the final static library (we can use ENABLE_MKLDNN, ENABLE_CLDNN etc... but might be needed for interpreter and readers as well)

Right now i relied a lot on the already existing flag USE_STATIC_IE (that was used for tests) but maybe it's not a good idea ?

Issue reference: #2675
Mainteners that might be interested: @ilya-lavrenov @ilyachur

Feel free to comment that objectives

Removed dllimport when is not appropriate

ilya-lavrenov · 2020-11-19T11:11:49Z

inference-engine/src/mkldnn_plugin/CMakeLists.txt


+if (NOT BUILD_SHARED_LIBS)
+    target_compile_definitions(${TARGET_NAME}_obj PRIVATE NGRAPH_STATIC_LIBRARY)
+endif()


maybe we can add $<TARGET_PROPERTY:ngraph,INTERFACE_INCLUDE_DIRECTORIES> to target_include_directories above?

Will that add the compile definition ?
By the way i was thinking about renaming NGRAPH_STATIC_LIBRARY to USE_STATIC_NGRAPH

ilya-lavrenov · 2020-11-19T11:15:19Z

inference-engine/cmake/plugins/plugins.cmake

-    list(APPEND PLUGIN_FILES "${IE_PLUGIN_DEVICE_NAME}:${IE_PLUGIN_NAME}")
-    list(REMOVE_DUPLICATES PLUGIN_FILES)
-    set(PLUGIN_FILES "${PLUGIN_FILES}" CACHE INTERNAL "" FORCE)
+    if (BUILD_SHARED_LIBS)


maybe we call leave this as is, but don't call ie_register_plugins?

I think we can use plugins passed to ie_register_plugins to create a map of function pointers to plugin's CreatePluginEngine_xxx functions..

But it can be done in a separate PR, right? or you are going to add more changes to this PR?

I intend to add more change to this PR (see above description)

ngraph/core/include/ngraph/visibility.hpp

ilya-lavrenov · 2020-11-19T11:36:01Z

When I compiled with BUILD_SHARED_LIBS=OFF and run ieFuncTests, I see:

sandye51@sandye51-pc:~/Documents/Programming/git_repo/dldt/bin/intel64/Release$ ./ieFuncTests
[ERR] 2020-11-19T11:33:32z ngraph/core/src/util.cpp 163 malloc failed to allocate memory of size 2305843009213694016
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

I except different error..

emmanuelattia · 2020-11-20T10:08:03Z

When I compiled with BUILD_SHARED_LIBS=OFF and run ieFuncTests, I see:

sandye51@sandye51-pc:~/Documents/Programming/git_repo/dldt/bin/intel64/Release$ ./ieFuncTests
[ERR] 2020-11-19T11:33:32z ngraph/core/src/util.cpp 163 malloc failed to allocate memory of size 2305843009213694016
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

I except different error..

I guess it's because ngraph is linked statically and unit test are implemented as global variable initializers, the unit test relies on the fact that ngraph global variables (here ngraph::element::f32) being initialized before convolution_transformation.cpp:testValues which is not guaranteed since they are all part of the same executable in the end.

That could be easily fixed by replacing global testValues by a static getTestValues() that returns a static variable (and enforce its initialization just at the instantiation) but i'm afraid this kind of global initialization order issue might appear somewhere else

…se they might depend on other global variables and, if link statically, there is no guarantee in the initialization order.

emmanuelattia · 2020-11-20T16:04:06Z

Last commit should fix the unit test crash in ieFuncTests @ilya-lavrenov, it's not fully validated yet of course because some of them need the readers.

ilya-lavrenov · 2020-11-21T09:59:19Z

I guess it's because ngraph is linked statically and unit test are implemented as global variable initializers, the unit test relies on the fact that ngraph global variables (here ngraph::element::f32) being initialized before convolution_transformation.cpp:testValues which is not guaranteed since they are all part of the same executable in the end.

That could be easily fixed by replacing global testValues by a static getTestValues() that returns a static variable (and enforce its initialization just at the instantiation) but i'm afraid this kind of global initialization order issue might appear somewhere else

Looks like you right. @ilyachur it's not a good to have such global variables inside ngraph.. Don't know whether we can remove them or convert to enum..

ilyachur · 2020-11-23T05:53:03Z

I guess it's because ngraph is linked statically and unit test are implemented as global variable initializers, the unit test relies on the fact that ngraph global variables (here ngraph::element::f32) being initialized before convolution_transformation.cpp:testValues which is not guaranteed since they are all part of the same executable in the end.

That could be easily fixed by replacing global testValues by a static getTestValues() that returns a static variable (and enforce its initialization just at the instantiation) but i'm afraid this kind of global initialization order issue might appear somewhere else

Looks like you right. @ilyachur it's not a good to have such global variables inside ngraph.. Don't know whether we can remove them or convert to enum..

I agree that we should avoid global variables. I created a PR #3279 which should fix this behaviour.

ilyachur · 2020-12-02T18:28:26Z

PR with deprecation of nGraph global variables was merged to the master branch.

…into ie_static_lib

emmanuelattia · 2020-12-03T08:19:51Z

PR with deprecation of nGraph global variables was merged to the master branch.

Should I proceed by replacing test initializations by functions instead of global variable initializations or should i change all tests to avoid using global variable.

ilya-lavrenov · 2020-12-03T09:02:07Z

Should I proceed by replacing test initializations by functions instead of global variable initializations or should i change all tests to avoid using global variable.

I suppose no changes in tests are needed since the issue with global variables had to have gone.

emmanuelattia · 2020-12-03T09:33:11Z

Should I proceed by replacing test initializations by functions instead of global variable initializations or should i change all tests to avoid using global variable.

I suppose no changes in tests are needed since the issue with global variables had to have gone.

Unless i'm mistaken, tests in ieFuncTests still reference deprecated global variables.
For instance
inference-engine\tests\functional\inference_engine\lp_transformations\convolution_transformation.cpp:~352

            LayerTransformation::createParamsU8I8(),
            // ActualValues
            {
                ngraph::element::u8,
                {{element::f32}, {}, { {0.02f}, element::f32 }},
                op::Constant::create(ngraph::element::f32, ngraph::Shape{}, std::vector<float>{ 2.f }),
                { 255ul, Shape({ 1, 1, 1, 1 }), { 0.f }, { 254.f }, { -1.27f }, { 1.27f } }
            },

ilya-lavrenov · 2020-12-03T09:35:41Z

Unless i'm mistaken, tests in ieFuncTests still reference deprecated global variables

In this case we need to remove their usage with proposed alternatives.

CC @ilyachur

ilyachur · 2020-12-03T11:45:45Z

I created another PR #3463 where I tried to remove extern global element types.

…atically built

emmanuelattia · 2020-12-03T15:18:03Z

Another major issue is that when i try to link statically inference_engine_ir_reader, inference_engine_ir_v7_reader with inference_engine they are many duplicated symbols, for instance:

InferenceEngine::details::GetIRVersion that is defined in inference_engine_ir_reader:ie_ir_reader.cpp
InferenceEngine::details::GetIRVersion that is defined in inference_engine_ir_v7_reader:ie_ir_reader.cpp

They are also duplicated symbols between inference_engine_ir_reader and inference_engine_legacy

Should I move each ir_reader in its own unique namespace ? (instead of namespace InferenceEngine)

ilya-lavrenov · 2020-12-03T15:32:58Z

@emmanuelattia-philips will it help if we change:

size_t GetIRVersion(std::istream& model);

to

static size_t GetIRVersion(std::istream& model);

?

emmanuelattia · 2020-12-03T15:35:24Z

@emmanuelattia-philips will it help if we change:
size_t GetIRVersion(std::istream& model);
to
static size_t GetIRVersion(std::istream& model);
?

Basically it concerns every symbols of the reader, not just frunction but also class members

ilya-lavrenov · 2020-12-03T15:44:51Z

Ok, in this case we can compile using different namespaces as you suggested

…sion when statically linking everything together Readers can be linked statically

emmanuelattia · 2020-12-03T17:53:09Z

Ok, in this case we can compile using different namespaces as you suggested

Ok just pushed these (quite big) modifications, next step are the DNN plugins.

…into ie_static_lib

…ager

* Each plugin has a uniquely named factory * Each plugin is referenced by ie_core in static mode (even if no plugins.xml), there is still support for additional definitions in plugins.xml * New macro : - INTERENCE_PLUGIN_API now is extern "C" only when extensions are not statically linked - INFERENCE_PLUGIN_STATIC_API is always extern "C" - IE_DEFINE_PLUGIN_CREATE_FUNCTION_EX allow to specify a unique name for the factory activated only when USE_STATIC_IE_EXTENSIONS is defined otherwise behaves like IE_DEFINE_PLUGIN_CREATE_FUNCTION * Some refactoring of the code in ie_so_pointer.hpp to be able to create a shared_ptr directly from a factory function with the same code as creating a shared_ptr from a dynamic object/library symbol.

…be checked

emmanuelattia · 2020-12-05T15:12:05Z

Now i'm stuck with MKLDNNPlugin because registration of the node rely on static global variables (see mkldnn_node.h)

#define REG_MKLDNN_CONCAT2(X, Y) X ## Y
#define REG_MKLDNN_CONCAT(X, Y) REG_MKLDNN_CONCAT2(X, Y)
#define REG_MKLDNN_PRIM_FOR(__prim, __type) \
static MKLDNNNode::Registrar<__prim> REG_MKLDNN_CONCAT(_reg_, __LINE__)(__type)

And since these variables are not referenced, they are not linked statically and node are not registered.

Allows whole_archive cmake macros to specify visibility When linking statically, using whole_archive to include MKLDNNPlugin so global variables that does registration of the nodes are included in the final executable

…ic factory and whole archive flag to force initialization (like in mkldnn) - static inference plugin now take into account parameters in plugins.xml, static factory is used only if plugin is not present or library name is empty or invalid.

…ATIC/SHARED - fixed ie_core when SHARED mode

…into ie_static_lib

ilya-lavrenov · 2020-12-08T11:54:49Z

docs/template_extension/CMakeLists.txt

 file(GLOB_RECURSE SRC *.cpp)

-add_library(${TARGET_NAME} SHARED ${SRC})
+add_library(${TARGET_NAME} ${SRC})


if we remove SHARED from here and users take this code as a template for their standalone extensions, it will be built as a static library. Can we either keep SHARED here or add something like:

if((NOT DEFINED BUILD_SHARED_LIBS) OR BUILD_SHARED_LIBS) set(LIBRARY_TYPE SHARED) else() set(LIBRARY_TYPE STATIC) endif() add_library(${TARGET_NAME} ${LIBRARY_TYPE} ${SRC})

Should be fixed

ilya-lavrenov · 2020-12-08T11:55:33Z

inference-engine/cmake/check_features_ie.cmake


+if (NOT BUILD_SHARED_LIBS)
+    add_definitions(-DUSE_STATIC_IE)
+    add_definitions(-DUSE_STATIC_IE_EXTENSIONS)


can we rename USE_STATIC_IE_EXTENSIONS to USE_STATIC_IE_PLUGINS?

ilya-lavrenov · 2020-12-08T11:59:51Z

inference-engine/include/details/ie_so_pointer.hpp

+     * @brief Constructs an object with existing reference
+     * @param pointedObj Existing reference to wrap
+     */
+    explicit SOPointer(std::shared_ptr<T> pointedObj): _so_loader(), _pointedObj(pointedObj) {


we already have a ctor with existing reference, but with raw pointer. Do we need both of them? Can we keep only one? (maybe the one with shared_ptr)

ilya-lavrenov · 2020-12-08T12:02:23Z

inference-engine/include/ie_api.h

+# define INFERENCE_ENGINE_CDECL __cdecl
+#endif
+
+#if (!defined(__GNUC__) && defined(USE_STATIC_IE)) || (defined(__GNUC__) && (__GNUC__ < 4))


why do we need !defined(__GNUC__) here? Does not look consistent with other similar places.

Otherwise i have to add a #ifdef USE_STATIC_LIB inside the #if defined(_WIN32)

ilya-lavrenov · 2020-12-08T12:05:52Z

inference-engine/include/ie_api.h

 * @param type A plugin type
 */

+#if (__GNUC__ >= 4)  // NOLINT


I think we need #if (__GNUC__ >= 4) -> #if defined(__GNUC__ ) && (__GNUC__ >= 4)

ilya-lavrenov · 2020-12-08T12:13:51Z

inference-engine/src/CMakeLists.txt

+    endif()
+
+    ieTargetLinkWholeArchiveEx(inference_engine PRIVATE HeteroPlugin MultiDevicePlugin)
+


will we have a single inference_engine.a static library with all plugins integrated into it?

Yes, the objective is to have every plugin inside the static library
ENABLE_CLDNN, ENABLE_MKLDNN etc... allows to make a lighter static library.

I've tried to build this branch and inference_engine.a contains only IE object files. But plugin object files are not present here..

Right now i have successfully tested it under Windows (my main OS), looking at linux now...

ilya-lavrenov · 2020-12-08T12:19:03Z

inference-engine/src/readers/ir_reader/ie_ir_parser.hpp

-namespace InferenceEngine {
+#include "ie_ir_reader_namespace.hpp"
+
+namespace InferenceEngineIRReader {


do we need to change namespace for all the classes / functions inside reader plugin? I suppose we need only ie_ir_reader.cpp, ie_ir_reader.hpp files, is not it?

I think they are other colliding symbols between inference_engine_ir_reader and inference_engine_ir_v7_reader in the other files.

ilya-lavrenov · 2020-12-08T12:19:27Z

inference-engine/src/readers/ir_reader/ie_ir_reader.hpp

-namespace InferenceEngine {
+namespace InferenceEngineIRReader {
+
+INFERENCE_PLUGIN_API(StatusCode) CreateReader(IReader*& reader, ResponseDesc *resp) noexcept;


why do we need this declaration inside hpp?

No, i can remove it

ilya-lavrenov · 2020-12-08T12:20:30Z

inference-engine/src/readers/onnx_reader/ie_onnx_reader.cpp

 }

-INFERENCE_PLUGIN_API(StatusCode) InferenceEngine::CreateReader(IReader*& reader, ResponseDesc *resp) noexcept {
+INFERENCE_PLUGIN_API(StatusCode) InferenceEngineONNXReader::CreateReader(IReader*& reader, ResponseDesc *resp) noexcept {


@ilyachur looks strange that create function for reader plugins is located inside namespace.

For inference plugin we don't have it. Can we remove namespace for exported CreateReader functions?

That's a good question, since it's exported as "C", it does reserve the global name "CreateReader" anyways...

Ok, but not to confuse others I suggesting removing namespace :)

Do I remove the definition of CreateReader inside the namespace in ie_reader.hpp, it was here before this PR.

If I don't remove it, when i call it internally i have to use explicit global namespace otherwise there is ambiguity at compile time.

ilya-lavrenov · 2020-12-08T12:20:48Z

inference-engine/src/readers/onnx_reader/ie_onnx_reader.hpp

+
+using namespace InferenceEngine;
+
+INFERENCE_PLUGIN_API(StatusCode) CreateReader(IReader*& reader, ResponseDesc *resp) noexcept;


do we need this declaration?

No i will remove it

ilya-lavrenov · 2020-12-08T14:33:43Z

inference-engine/ie_bridges/c/samples/common/opencv_c_wraper/CMakeLists.txt

 file(GLOB HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/*.h)

 # create library
-add_library(${TARGET_NAME} SHARED ${HEADERS} ${SOURCES})


this code is a part of release OpenVINO package where BUILD_SHARED_LIBS is not defined. I think we need to add explicit BUILD_SHARED_LIBS=ON inside https://github.com/openvinotoolkit/openvino/blob/master/inference-engine/samples/CMakeLists.txt which works for both C and C++ samples.

ilya-lavrenov · 2020-12-08T14:59:00Z

I also have:

[100%] Linking CXX executable /home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/ieFuncTests
/home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/lib/libinference_engine_ir_v7_reader.a(unity_0_cxx.cxx.o): In function `InferenceEngine::details::BlobStream::BlobBuffer::~BlobBuffer()':
unity_0_cxx.cxx:(.text._ZN15InferenceEngine7details10BlobStream10BlobBufferD2Ev+0x0): multiple definition of `InferenceEngine::details::BlobStream::BlobBuffer::~BlobBuffer()'
/home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/lib/libinference_engine.a(unity_3_cxx.cxx.o):unity_3_cxx.cxx:(.text._ZN15InferenceEngine7details10BlobStream10BlobBufferD2Ev+0x0): first defined here
/home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/lib/libinference_engine_ir_v7_reader.a(unity_0_cxx.cxx.o): In function `InferenceEngine::details::BlobStream::BlobBuffer::~BlobBuffer()':
unity_0_cxx.cxx:(.text._ZN15InferenceEngine7details10BlobStream10BlobBufferD2Ev+0x0): multiple definition of `InferenceEngine::details::BlobStream::BlobBuffer::~BlobBuffer()'
/home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/lib/libinference_engine.a(unity_3_cxx.cxx.o):unity_3_cxx.cxx:(.text._ZN15InferenceEngine7details10BlobStream10BlobBufferD2Ev+0x0): first defined here
/home/sandye51/Documents/Programming/git_repo/dldt/bin/intel64/Release/lib/libinference_engine_ir_v7_reader.a(unity_0_cxx.cxx.o): In function `InferenceEngine::details::BlobStream::BlobBuffer::~BlobBuffer()':

when compiling the ieFuncTests target.

- template_extension now defaulted to shared - ie_so_pointer: removed raw pointer constructor (implicit cast to shared_ptr is enough) - CreateReader out of namespace

MrBlueSkyFox · 2020-12-22T17:29:46Z

This project (static version of OpenVINO) is very interesting for us. Can I help you with this PR?
But I have a question to you too. Can you tell me anything about current state? For example, I'm trying to use statics libraries of core parts. I compiled static libraries, but when I'm trying to compile source code with any class initialization I have an error: ngraph and other methods are undefined. Have you seen the same errors and have you any thoughts about this problem, how to solve it?

emmanuelattia · 2020-12-24T14:21:20Z

This project (static version of OpenVINO) is very interesting for us. Can I help you with this PR?
But I have a question to you too. Can you tell me anything about current state? For example, I'm trying to use statics libraries of core parts. I compiled static libraries, but when I'm trying to compile source code with any class initialization I have an error: ngraph and other methods are undefined. Have you seen the same errors and have you any thoughts about this problem, how to solve it?

Hello, this PR is a littler bit on hgold for holidays. A way to solve the problem of static initialization is to use whole-archive flag for linking, but i haven't tracked yet all the needed changes

MrBlueSkyFox · 2020-12-25T17:15:55Z

If

This project (static version of OpenVINO) is very interesting for us. Can I help you with this PR?
But I have a question to you too. Can you tell me anything about current state? For example, I'm trying to use statics libraries of core parts. I compiled static libraries, but when I'm trying to compile source code with any class initialization I have an error: ngraph and other methods are undefined. Have you seen the same errors and have you any thoughts about this problem, how to solve it?

Hello, this PR is a littler bit on hgold for holidays. A way to solve the problem of static initialization is to use whole-archive flag for linking, but i haven't tracked yet all the needed changes
Currently trying to link most of part and i

This project (static version of OpenVINO) is very interesting for us. Can I help you with this PR?
But I have a question to you too. Can you tell me anything about current state? For example, I'm trying to use statics libraries of core parts. I compiled static libraries, but when I'm trying to compile source code with any class initialization I have an error: ngraph and other methods are undefined. Have you seen the same errors and have you any thoughts about this problem, how to solve it?

Hello, this PR is a littler bit on hgold for holidays. A way to solve the problem of static initialization is to use whole-archive flag for linking, but i haven't tracked yet all the needed changes

If we use static version of libraries with --whole-archive and multiply definition flags, is it possible to run the most simple inference?

…into ie_static_lib

n-Guard · 2021-10-14T11:25:47Z

Hello everyone, thanks for your work so far, I think this will be a really useful feature!
I just wanted to ask if there is any progress on this or if it is still put on hold for the forseeable future?

emmanuelattia · 2021-10-14T12:34:13Z

Hello everyone, thanks for your work so far, I think this will be a really useful feature! I just wanted to ask if there is any progress on this or if it is still put on hold for the forseeable future?

Hello, I don't know if i'll be able to pursue this work

n-Guard · 2021-10-14T13:12:11Z

Hello, I don't know if i'll be able to pursue this work

That's unfortunate but thank you anyway for your previous effort!

ilya-lavrenov · 2022-01-11T10:11:32Z

Static libraries are now enabled in OpenVINO. See https://github.com/openvinotoolkit/openvino/wiki/StaticLibraries

All shared library now depend on BUILD_SHARED_LIBS flags

e7f6fdd

Removed dllimport when is not appropriate

openvino-pushbot added the ExternalPR External contributor label Nov 19, 2020

ilya-lavrenov reviewed Nov 19, 2020

View reviewed changes

emmanuelattia and others added 3 commits November 20, 2020 11:10

Merge branch 'master' into ie_static_lib

67d5239

Removing extra dllimport for fluid_preproc test

1ec1043

Changed global initializer for unit tests into static function, becau…

9c925e7

…se they might depend on other global variables and, if link statically, there is no guarantee in the initialization order.

Fixes line endings

28071f3

gcc + static library should still apply visibility attributes

4e5c57e

ilyachur mentioned this pull request Dec 2, 2020

Deprecate global element types #3444

Merged

emmanuelattia added 2 commits December 3, 2020 09:16

Merge branch 'master' of https://github.com/openvinotoolkit/openvino …

d96ae12

…into ie_static_lib

Updated test initialization for files changed by the last merge

f316769

Added auto registration of ie_backend and interpreter_backend when st…

c0d5168

…atically built

Readers are now defined in their own namespaces to avoid symbol colli…

895458b

…sion when statically linking everything together Readers can be linked statically

Merge branch 'master' of https://github.com/openvinotoolkit/openvino …

087ea2b

…into ie_static_lib

emmanuelattia added 5 commits December 3, 2020 20:43

Fixed ngraph backend registration now referenced from the backend_man…

65c0d29

…ager

Reverted tests to global initialization.

1a43c6f

Fix coding style

0ec9d38

so_loader_ can be nullptr (because 2 ctors allows it) so it must not …

afc8136

…be checked

emmanuelattia added 5 commits December 5, 2020 22:31

Removed vs_version in static mode to allow whole archive linking

b4721a7

Allows whole_archive cmake macros to specify visibility When linking statically, using whole_archive to include MKLDNNPlugin so global variables that does registration of the nodes are included in the final executable

Fixed cpplint failures.

f027d37

- fuild_preproc library now has its own #define for visibility wrt ST…

7db8026

…ATIC/SHARED - fixed ie_core when SHARED mode

Merge branch 'master' of https://github.com/openvinotoolkit/openvino …

eeb1384

…into ie_static_lib

ilya-lavrenov reviewed Dec 8, 2020

View reviewed changes

emmanuelattia added 2 commits December 8, 2020 16:10

- Renamed USE_STATIC_IE_EXTENSIONS => USE_STATIC_IE_PLUGINS

83f559f

- template_extension now defaulted to shared - ie_so_pointer: removed raw pointer constructor (implicit cast to shared_ptr is enough) - CreateReader out of namespace

- Fixed cpplinting - Fixed symbol collision in shared lib

8f6f1ba

Merge branch 'master' of https://github.com/openvinotoolkit/openvino …

b3b1c24

…into ie_static_lib

ilya-lavrenov closed this Jan 11, 2022

		endif()

		ieTargetLinkWholeArchiveEx(inference_engine PRIVATE HeteroPlugin MultiDevicePlugin)


		using namespace InferenceEngine;

		INFERENCE_PLUGIN_API(StatusCode) CreateReader(IReader& reader, ResponseDesc resp) noexcept;

Conversation

emmanuelattia commented Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilya-lavrenov Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ilya-lavrenov commented Nov 19, 2020

Uh oh!

emmanuelattia commented Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emmanuelattia commented Nov 20, 2020

Uh oh!

ilya-lavrenov commented Nov 21, 2020

Uh oh!

ilyachur commented Nov 23, 2020

Uh oh!

ilyachur commented Dec 2, 2020

Uh oh!

emmanuelattia commented Dec 3, 2020

Uh oh!

ilya-lavrenov commented Dec 3, 2020

Uh oh!

emmanuelattia commented Dec 3, 2020

Uh oh!

ilya-lavrenov commented Dec 3, 2020

Uh oh!

ilyachur commented Dec 3, 2020

Uh oh!

emmanuelattia commented Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilya-lavrenov commented Dec 3, 2020

Uh oh!

emmanuelattia commented Dec 3, 2020

Uh oh!

ilya-lavrenov commented Dec 3, 2020

Uh oh!

emmanuelattia commented Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emmanuelattia commented Dec 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

emmanuelattia commented Nov 19, 2020 •

edited

Loading

ilya-lavrenov Nov 19, 2020 •

edited

Loading

emmanuelattia commented Nov 20, 2020 •

edited

Loading

emmanuelattia commented Dec 3, 2020 •

edited

Loading

emmanuelattia commented Dec 3, 2020 •

edited

Loading

emmanuelattia commented Dec 5, 2020 •

edited

Loading

ilya-lavrenov Dec 8, 2020 •

edited

Loading