@alalek, is a unit test required to catch this?

@ganesh-k13 Thanks for the patch! Yes, unit test is very appreciated.

Sure @asmorkalov, I'll add it. I'm new to this, so I'll read a bit before adding.

Hi @mshabunin / @asmorkalov / @alalek , I have a basic doubt.
I tried the fix in a sample file:

int main() {
     Mat leftImg = imread("im2.png", 0);
     Mat rightImg = imread("im2.png", 0);
     Mat leftDisp;
     Ptr<StereoBM> bm = StereoBM::create(16,9);
     bm->setPreFilterType(StereoBM::PREFILTER_NORMALIZED_RESPONSE);
     bm->compute( leftImg, rightImg, leftDisp);
     return 0;
 }

I see no crash.
But if I do same in a Test(calib3d/test/test_stereomatching.cpp ):

TEST(Calib3d_StereoBM, regression17201)
 {
     String path = cvtest::TS::ptr()->get_data_path() + "cv/stereomatching/datasets/teddy/";
     Mat leftImg = imread(path + "im2.png", 0);
     ASSERT_FALSE(leftImg.empty());
     Mat rightImg = imread(path + "im2.png", 0);
     ASSERT_FALSE(rightImg.empty());
     Mat leftDisp;
     {
         Ptr<StereoBM> bm = StereoBM::create(16, 9);
         bm->setPreFilterType(StereoBM::PREFILTER_NORMALIZED_RESPONSE);
         bm->compute( leftImg, rightImg, leftDisp);
     }
 }

[1] 17033 segmentation fault (core dumped) bin/opencv_test_calib3d --gtest_filter=Calib3d_StereoBM.regression17201 almost as if change is not there.

Is there anything for this difference in behavior between normal code and gtest code?

@ganesh-k13 could you put your sample code to OpenCV samples and build OpenCV with -DBUILD_EXAMPLES=ON. All samples are built with the same compiler options and it should help to reproduce the issue.

Thanks @asmorkalov, I'll try that, I think it is some small config mistake from my end. I'll try some more stuff or post in answers.opencv.

@ganesh-k13 , prefilter pointers must be initialized with 0, add something like

prefilter[0] = 0;
prefilter[1] = 0;

at the beginning of the BufferBM constructor.

Thanks, @mshabunin, that fixed the issue.
I had a doubt regarding:

How was the value of 0.005+DBL_EPSILON arrived at?

I was planning to add bm->setPreFilterType(StereoBM::PREFILTER_NORMALIZED_RESPONSE); before compute there(or once again after PREFILTER_XSOBEL is used). Is this ok or I'll add a separate test?

Looks like this tests does not verify the output, it only checks that integer output does not differ from float output and this error margin is an arbitrary value.

Yes, I think you can add another filter to the same test, or you can make parameterized test (TEST_P(...)).

Thanks, @mshabunin , I have added cases, the aim of the cases is to trigger different mem allocs in BufferBM. So I added cases with combinations of preFilterCap, SADWindowSize and preFilterType. I also moved Calib3d_StereoBM, regression up because it was under Calib3d_StereoSBM

@alalek , @mshabunin , @asmorkalov any update on this?