Python-implemented DNN custom layer Failed to call "forward" method #17514
Description
System information (version)
- OpenCV => 4.2.0 (installed through pip install opencv-contrib-python)
- Operating System / Platform => Ubuntu 18.04 64 Bit
- Compiler => omitted , precompiled binary used
Detailed description
I tried to re-implement a custom DNN layer in python , originally implemented in this C plus plus implementation on contrib module because current pip does not have 4.3.0 precompiled and i just wanted to try implementing a custom layer in python.
Keep in mind the target of this issue is to address the issue on why the error happens and less on using the dnn_superres module of opencv contrib.
After registering the layer and calling forward(), the code fails with the following error message :
cv2.error: OpenCV(4.2.0) /io/opencv/modules/dnn/misc/python/pyopencv_dnn.hpp:159: error: (-213:The function/feature is not implemented) Failed to call "forward" method in function 'forward'
upon reading similar issues in #15376 and since my code involves indexing a numpy array, i tried to make sure all indices are integers, but to no avail. Below is the implementation (included here to continue discussion):
class DepthToSpace:
def __init__(self,params,blob):
# print(params)
self.outshape = None
def getMemoryShapes(self,inputs):
# print(inputs)
if inputs[0][1] == 4 or inputs[0][1] == 9 or inputs[0][1] == 16:
scale = int(np.sqrt(inputs[0][1]))
else:
scale = int(np.sqrt(inputs[0][1]/3))
d0 = inputs[0][0]
d1 = int(inputs[0][1] / pow(scale,2))
d2 = int(scale * inputs[0][2])
d3 = int(scale * inputs[0][3])
self.outshape = (d0,d1,d2,d3)
return [[d0,d1,d2,d3]]
def forward(self,inputs):
modelinput = inputs[0].astype(np.float32)
_, nc, h, w = modelinput.shape
scale = int(self.outshape[2] / h)
output = np.zeros(self.outshape,dtype=np.float32)
count = 0
for c in range(nc):
for y in range(self.outshape[2]):
for x in range(self.outshape[3]):
x_coord = y // scale
y_coord = x // scale
c_coord = int(nc * scale * (y % scale) + nc * c * (x % scale))
ind = int((((c_coord * h) + x_coord) * w) + y_coord)
output.ravel()[(count)] = modelinput.ravel()[ind]
# output.ravel()[0] = modelinput.ravel()[0]
count += 1
return [output]if it helps, this is another implementation of forward that does not involve for loops (but still has the same error).
def forward(self,inputs):
modelinput = inputs[0].astype(np.float32)
print(modelinput.shape)
_, nc, h, w = modelinput.shape
scale = int(self.outshape[2] / h)
output = np.zeros(self.outshape,dtype=np.float32)
indices = np.indices(self.outshape[1:]).transpose(1,2,3,0).reshape(-1,3)
xs = np.floor(indices[:,1] / scale).astype(int)
ys = np.floor(indices[:,2] / scale).astype(int)
cs = (nc * scale * (indices[:,1] % scale) + nc * (indices[:,2] % scale) + indices[:,0]).astype(int)
assert len(xs) == len(ys) and len(xs) == len(cs)
indices = zip(*indices)
cyx = (cs,ys,xs)
output[0,indices] = modelinput[cyx]
# print(output)
return [output]and here is the full encompassing the whole model, if it helps
class Upscaler:
def __init__(self,model_path,modeltype,scale):
assert modeltype in ['espcn','lapsrn','fsrcnn'] , "modeltype not implemented"#,'edsr']
self.modeltype = modeltype
self.scale = scale
self.net = dnn.readNetFromTensorflow(model_path)
def _preprocess(self,image):
ycrcb = cv2.cvtColor(image,cv2.COLOR_RGB2YCrCb).astype(np.float32) / 255.0
return ycrcb
def _postprocess(self,y_output,source):
cr = source[:,:,1]
cb = source[:,:,2]
cr = cv2.resize(cr,None,fx=self.scale,fy=self.scale)
cb = cv2.resize(cb,None,fx=self.scale,fy=self.scale)
reconst_ycrcb = np.stack([y_output,cr,cb],axis=-1)
reconst_ycrcb *= 255
reconst_ycrcb = reconst_ycrcb.astype(np.uint8)
return cv2.cvtColor(reconst_ycrcb,cv2.COLOR_YCrCb2RGB)
def upscale(self,image):
img = self._preprocess(image)
y_ch = img[:,:,0]
blob = dnn.blobFromImage(y_ch)
self.net.setInput(blob)
output = self.net.forward()
output = dnn.imagesFromBlob(output)[0]
return self._postprocess(output,img)interestingly enough, when i altered the code that assigns values to the output variable of both implementations, replacing the calculated indices with a constant, say, zero, it works fine. Error happens only when the calculated indices is used to index modelinput or output to grab / assign values on them.
To avoid confusion,
this works
output.ravel()[0] = modelinput.ravel()[0]while all these does not
# count is defined outside the loop
output.ravel()[count] = modelinput.ravel()[0]# count is defined outside the loop
output.ravel()[int(count)] = modelinput.ravel()[0]# calculation of index can be seen above
output.ravel()[0] = modelinput.ravel()[index]# calculation of index can be seen above
output.ravel()[0] = modelinput.ravel()[int(index)]Steps to reproduce
The model used can be accessed through this link
Assuming the above code is present in the same sourcefile, and all libraries are imported :
cv2.dnn_registerLayer('DepthToSpace',DepthToSpace)
upscaler = Upscaler('/path/to/FSRCNN_x2.pb','fsrcnn',2)
frame = cv2.imread('/any/image/will/do')
frame = upscaler.upscale(frame) # this will error outIssue submission checklist
- [ V] I report the issue, it's not a question
- [V ] I checked the problem with documentation, FAQ, open issues,
answers.opencv.org, Stack Overflow, etc and have not found solution - [ X] I updated to latest OpenCV version and the issue is still there
- [ V] There is reproducer code and related data files: videos, images, onnx, etc