Skip to content

autorotate in-memory jpegs#8492

Merged
alalek merged 6 commits intoopencv:masterfrom
brian-armstrong-discord:exif_inmemory
Apr 14, 2017
Merged

autorotate in-memory jpegs#8492
alalek merged 6 commits intoopencv:masterfrom
brian-armstrong-discord:exif_inmemory

Conversation

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor

resolves #8172

This pull request changes ExifReader to operate on istreams rather than filenames. It then builds a streambuf on top of Mat so that imdecode may use ApplyExifOrientation like imread does.

Copy link
Copy Markdown
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!
Could you please squash all commits into one?

Size bufSize = buf.size();
if( bufSize.width == 1 && buf.type() == CV_8UC1 )
{
ByteStreamBuffer bsb( reinterpret_cast<char*>(buf.data), bufSize.height );
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is about this? (do not assume exact Mat structure: column vector as 2D Mat)

if (buf.isContinuous())
{
    ByteStreamBuffer bsb( buf.ptr<char*>(), buf.total() * bufSize.elemSize() );
    ...
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh ok that makes sense 👍 i still want to check for CV_8UC1 though, right?

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

The test that failed here doesn't seem related to this patch

OpenCV: FFMPEG: tag 0x34363248/'H264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'

Could you help me find what I might have broke?

if( buf.isContinuous() )
{
ByteStreamBuffer bsb( reinterpret_cast<char*>(buf.data), buf.total() * buf.elemSize() );
std::istream stream( &bsb );
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the current ByteStreamBuffer implementation is buggy.

Please use this instead (with added #include <sstream> and removed ByteStreamBuffer):

std::istringstream stream(std::string(reinterpret_cast<char*>(buf.data), buf.total() * buf.elemSize()), std::ios_base::in);

(it does extra memcpy operation, but it is not critical for I/O tasks)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure this is what caused build to break - it passed with a nearly identical implementation

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

brian-armstrong-discord commented Apr 3, 2017

edit: nevermind, didnt see your reply

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

brian-armstrong-discord commented Apr 3, 2017

@alalek Do you feel pretty confident that's what caused it to break? The build did pass on a previous commit, though with the squash the changes have been hidden. IIRC the only things I changed were that I added an additional bounds check for INT_MIN and added your suggestion to use buf.total() * bufSize.elemSize(). That doesn't seem like enough to cause video files? to fail - in fact, after checking the test in question, I'm not even sure imread/imdecode are being invoked.

edit: anyway if you are confident then i'll change it, but i don't want to go on a goose chase here :)

@alalek
Copy link
Copy Markdown
Member

alalek commented Apr 3, 2017

Stack trace for ByteStreamBuffer ctor call:

#0  0x00000000005a3ef3 in (anonymous namespace)::ByteStreamBuffer::ByteStreamBuffer(char*, std::size_t) (this=0x7ffffffecf50, base=0x1908bf0 "\377\330\377", <incomplete sequence \340>, length=38608) at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/loadsave.cpp:66
#1  0x00000000005a5c97 in cv::ApplyExifOrientation(cv::Mat const&, cv::Mat&) (buf=..., img=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/loadsave.cpp:339
#2  0x00000000005a73c1 in cv::imdecode(cv::_InputArray const&, int) (_buf=..., flags=3)
    at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/loadsave.cpp:733
#3  0x000000000058f720 in cv::MotionJpegCapture::retrieveFrame(int, cv::_OutputArray const&) (this=0x1908840, output_frame=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap_mjpeg_decoder.cpp:824
#4  0x0000000000582bbb in cv::VideoCapture::retrieve(cv::_OutputArray const&, int) (this=0x7ffffffed4d0, image=..., channel=0)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:640
#5  0x0000000000582d96 in cv::VideoCapture::read(cv::_OutputArray const&) (this=0x7ffffffed4d0, image=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:663
#6  0x0000000000582dfb in cv::VideoCapture::operator>>(cv::Mat&) (this=0x7ffffffed4d0, image=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:691
#7  0x00000000004fc681 in CV_PositioningTest::run(int) (this=0x7fffffffd6f0)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/test/test_video_pos.cpp:142

It is called from "MotionJpegCapture" in this test.
BTW, Probably it is better to use IMREAD_IGNORE_ORIENTATION flag in MJPEG decoder.

Crash backtrace (with log messages):

ByteStreamBuffer(0x18f2310, 28272)
ByteStreamBuffer::seekoff(14, 1, ...)
ByteStreamBuffer::seekoff(65, 1, ...)
ByteStreamBuffer::seekoff(65, 1, ...)
ByteStreamBuffer::seekoff(29, 1, ...)
ByteStreamBuffer::seekoff(179, 1, ...)
ByteStreamBuffer::seekoff(29, 1, ...)
ByteStreamBuffer::seekoff(179, 1, ...)
ByteStreamBuffer::seekoff(15, 1, ...)
ByteStreamBuffer::seekoff(10, 1, ...)
ByteStreamBuffer::seekoff(35366, 1, ...)

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff102d970 in __memcpy_sse2_unaligned () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff102d970 in __memcpy_sse2_unaligned () at /lib64/libc.so.6
#1  0x00007ffff7b7833b in std::basic_streambuf<char, std::char_traits<char> >::xsgetn(char*, long) () at /lib64/libstdc++.so.6
#2  0x00007ffff7b51b9b in std::istream::read(char*, long) () at /lib64/libstdc++.so.6
#3  0x00000000005ca5c0 in cv::ExifReader::getExif() (this=0x7ffffffecf90)
    at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/exif.cpp:134
#4  0x00000000005ca1ba in cv::ExifReader::parse() (this=0x7ffffffecf90) at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/exif.cpp:83
#5  0x00000000005a5cd4 in cv::ApplyExifOrientation(cv::Mat const&, cv::Mat&) (buf=..., img=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/loadsave.cpp:342
#6  0x00000000005a73c1 in cv::imdecode(cv::_InputArray const&, int) (_buf=..., flags=3)
    at /home/alalek/projects/opencv/opencv_dev/modules/imgcodecs/src/loadsave.cpp:733
#7  0x000000000058f720 in cv::MotionJpegCapture::retrieveFrame(int, cv::_OutputArray const&) (this=0x1908840, output_frame=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap_mjpeg_decoder.cpp:824
#8  0x0000000000582bbb in cv::VideoCapture::retrieve(cv::_OutputArray const&, int) (this=0x7ffffffed4d0, image=..., channel=0)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:640
#9  0x0000000000582d96 in cv::VideoCapture::read(cv::_OutputArray const&) (this=0x7ffffffed4d0, image=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:663
#10 0x0000000000582dfb in cv::VideoCapture::operator>>(cv::Mat&) (this=0x7ffffffed4d0, image=...)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/src/cap.cpp:691
#11 0x00000000004fc681 in CV_PositioningTest::run(int) (this=0x7fffffffd6f0)
    at /home/alalek/projects/opencv/opencv_dev/modules/videoio/test/test_video_pos.cpp:14

The similar logs messages are on buildbot (search for "Failure" keyword).

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

brian-armstrong-discord commented Apr 3, 2017

Ah, that helps a lot.

I don't see stacktraces in the buildbot output. Maybe I don't have enough permissions? All I see is

Failed

	failure reason: Arithmetic exception
	test case #-1
	seed: ffffffffffffffff
-----------------------------------
	LOG:

File: /tmp/__opencv_temp.jNDKKz/__opencv_temp.YQfV7G.XVID.avi

File: /tmp/__opencv_temp.jNDKKz/__opencv_temp.UJvnn8.MPEG.avi

File: /tmp/__opencv_temp.jNDKKz/__opencv_temp.loKZRG.MJPG.avi
General failure:
	Arithmetic exception (-6)

-----------------------------------

which didn't really tell me anything :)

@brian-armstrong-discord brian-armstrong-discord force-pushed the exif_inmemory branch 2 times, most recently from 320eb96 to 703295f Compare April 7, 2017 21:15
brian-armstrong-discord added a commit to brian-armstrong-discord/opencv that referenced this pull request Apr 7, 2017
this builds on opencv#8492 and
opencv#8511 to add a new method,
.orientation(), to ImageDecoder. The JPEG subclass now overrides this to
do jpeg-specific exif tag reading

we also now export a method, OrientationTransform, which uses this
orientation value to do the actual matrix operations
@alalek
Copy link
Copy Markdown
Member

alalek commented Apr 11, 2017

ByteStreamBuffer has very limited implementation. Looks like it handles dir == std::ios_base::cur case only. Unfortunately there are no reliable checks for unsupported parameters (just several return -1 statements for some cases).
So I still prefer to avoid custom implementation of mapped byte stream and to use std::istringstream instead.

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

@alalek If you look at ExifReader.cpp, you can see that the only seeking it does (with seekg) is based on the current pointer, so in fact this is fine. If I changed ExifReader to inspect the return values from these seekg calls, would that work for you? It seems silly to me to allocate and copy a bunch of memory just for this purpose

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

@alalek Also, for what it's worth, seekoff's base behavior is to return -1, so doing so here merely conforms to that. http://www.cplusplus.com/reference/streambuf/streambuf/seekoff/

@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

@alalek I've pushed a new version which will check the result of seekg. I think this is in fact safer than the previous version which never checked the result of seek

{
return -1;
}
gbump( static_cast<int>(off) );
Copy link
Copy Markdown
Member

@alalek alalek Apr 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"int" type limitation can be dropped (with INT_MAX/INT_MIN checks above):

setg(eback(), gptr() + off, egptr());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can't be dropped because the windows build complains about silently converting a long to an int

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean this:

- gbump( static_cast<int>(off) );
+ setg(eback(), gptr() + off, egptr());

There is no conversion to "int".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@brian-armstrong-discord brian-armstrong-discord force-pushed the exif_inmemory branch 2 times, most recently from f6cd087 to d2fad10 Compare April 13, 2017 22:50
@brian-armstrong-discord
Copy link
Copy Markdown
Contributor Author

@alalek Thanks for all your help so far 👍 I know it's been kind of a struggle, heh

Let me know if there's anything else I can do to help this PR along

@alalek
Copy link
Copy Markdown
Member

alalek commented Apr 14, 2017

👍

@alalek alalek merged commit 9e054d3 into opencv:master Apr 14, 2017
brian-armstrong-discord added a commit to brian-armstrong-discord/opencv that referenced this pull request Apr 25, 2017
…mory

autorotate in-memory jpegs (opencv#8492)

* autorotate in-memory jpegs

* correct indentation (4 spaces)

* imgcodecs: don't apply EXIF rotation for unloaded images

* videoio: don't try to rotate MJPEG stream

* imgcodecs: ByteStreamBuffer::seekoff support all seek "dir"

* imgcodecs: fix condition: "off == egptr() - eback()" is valid offset
@caogang
Copy link
Copy Markdown

caogang commented Apr 22, 2019

This bug is fixed after which version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error in imdecode, acts differently than imread and doesn't things like orientation and more correctly

3 participants