Skip to content

ARROW-2023: [C++] Fix ASAN failure on malformed / empty stream input, enable ASAN builds, add more dev docs #1503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wesm wants to merge 4 commits into from
Closed

ARROW-2023: [C++] Fix ASAN failure on malformed / empty stream input, enable ASAN builds, add more dev docs #1503

wesm wants to merge 4 commits into from

Conversation

@wesm
Copy link
Member

@wesm wesm commented Jan 24, 2018

More to do here, see discussion in https://issues.apache.org/jira/browse/ARROW-1589

wesm added 2 commits February 26, 2018 14:17
@wesm
Start C++ stream reader tests for malformed inputs [ci skip]
69cb5b7 
Change-Id: I6af65bb1fe46f8aa815895570849fcef5b624bcc
@wesm
Add missing lsan-suppressions.txt to enable ASAN to run with unittest…
53d2c92 
…s [skip ci]

Change-Id: I007e567af807d2af4d1801a11cd34279d80fd740
wesm added 2 commits February 26, 2018 14:58
@wesm
Add ASAN instructions, llvm-symbolizer instructions for fuzzers. Remo…
90787ec 
…ve Apache Kudu cruft from run-test.sh

Change-Id: Ib97dbdb32e3f4da4919584e29f17db55bfb964a5
@wesm
Do not invoke memcpy unless number of bytes to read greater than 0. R…
e3a590e 
…eturn more informative error message

Change-Id: I335736571e56af36f4de909e35f8d667693db7fe
@wesm wesm changed the title WIP ARROW-2023: [C++] Start stream reader tests for malformed inputs ARROW-2023: [C++] Fix ASAN failure on malformed / empty stream input, enable ASAN builds, add more dev docs Feb 26, 2018
@wesm
Copy link
Member Author

wesm commented Feb 26, 2018

@xhochy @crepererum this is ready to go. note this only fixes one of the fuzzer failures, now it returns a Flatbuffers error:

==19598==The signal is caused by a READ memory access.
    #0 0x682c79 in int flatbuffers::ReadScalar<int>(void const*) /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:198:23
    #1 0x682b6f in flatbuffers::Table::GetVTable() const /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1583:20
    #2 0x682a3a in flatbuffers::Table::GetOptionalFieldOffset(unsigned short) const /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1590:19
    #3 0x6b0e8e in long flatbuffers::Table::GetField<long>(unsigned short, long) const /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1599:25
    #4 0x783934 in org::apache::arrow::flatbuf::Message::bodyLength() const /home/wesm/code/arrow/cpp/build/src/arrow/ipc/Message_generated.h:294:12
    #5 0x77ced7 in arrow::ipc::Message::ReadFrom(std::shared_ptr<arrow::Buffer> const&, arrow::io::InputStream*, std::unique_ptr<arrow::ipc::Message, std::default_delete<arrow::ipc::Message> >*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:140:37
    #6 0x7824d9 in arrow::ipc::ReadMessage(arrow::io::InputStream*, std::unique_ptr<arrow::ipc::Message, std::default_delete<arrow::ipc::Message> >*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:236:10
    #7 0x784036 in arrow::ipc::InputStreamMessageReader::ReadNextMessage(std::unique_ptr<arrow::ipc::Message, std::default_delete<arrow::ipc::Message> >*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:255:12
    #8 0x67abb8 in arrow::ipc::ReadMessageAndValidate(arrow::ipc::MessageReader*, arrow::ipc::Message::Type, bool, std::unique_ptr<arrow::ipc::Message, std::default_delete<arrow::ipc::Message> >*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:371:29
    #9 0x6b2e82 in arrow::ipc::RecordBatchStreamReader::RecordBatchStreamReaderImpl::ReadSchema() /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:426:23
    #10 0x67d40f in arrow::ipc::RecordBatchStreamReader::RecordBatchStreamReaderImpl::Open(std::unique_ptr<arrow::ipc::MessageReader, std::default_delete<arrow::ipc::MessageReader> >) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:407:12
    #11 0x6724d2 in arrow::ipc::RecordBatchStreamReader::Open(std::unique_ptr<arrow::ipc::MessageReader, std::default_delete<arrow::ipc::MessageReader> >, std::shared_ptr<arrow::RecordBatchReader>*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:477:36
    #12 0x672dc0 in arrow::ipc::RecordBatchStreamReader::Open(arrow::io::InputStream*, std::shared_ptr<arrow::RecordBatchReader>*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:484:10
    #13 0x651842 in LLVMFuzzerTestOneInput /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/ipc-fuzzing-test.cc:31:12
    #14 0x52a0ae in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52a0ae)
    #15 0x529a94 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*) (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x529a94)
    #16 0x52ad51 in fuzzer::Fuzzer::MutateAndTestOne() (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52ad51)
    #17 0x52af17 in fuzzer::Fuzzer::Loop() (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52af17)
    #18 0x523415 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x523415)
    #19 0x51e280 in main (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x51e280)
    #20 0x7fdeb9619f44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
    #21 0x51e185 in _start (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x51e185)

I created https://issues.apache.org/jira/browse/ARROW-2222 about adding an option to validate input messages

@wesm
Copy link
Member Author

wesm commented Feb 26, 2018

appveyor build: https://ci.appveyor.com/project/wesm/arrow/build/1.0.1724

xhochy
xhochy approved these changes Feb 27, 2018
View reviewed changes
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

@xhochy xhochy closed this in e0328b0 Feb 27, 2018
@asfimport asfimport mentioned this pull request Feb 27, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xhochy xhochy xhochy approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wesm @xhochy