Skip to content

See if reading JSON maps with simdjson is quicker#19

Closed
vadi2 wants to merge 46 commits intoSlySven:Enhance_importExportMapAsJSONfrom
vadi2:read-with-simd
Closed

See if reading JSON maps with simdjson is quicker#19
vadi2 wants to merge 46 commits intoSlySven:Enhance_importExportMapAsJSONfrom
vadi2:read-with-simd

Conversation

@vadi2
Copy link
Copy Markdown

@vadi2 vadi2 commented Jan 31, 2021

Brief overview of PR changes/additions

Use the quite popular simdjson library for parsing JSON

Motivation for adding to Mudlet

Mudlet needs to be fast, current Qt JSON parsing is not fast enough

Other info (issues closed, discussion etc)

Still a WIP.

@vadi2 vadi2 changed the title See if reasing JSON maps with simdjson is quicker See if reading JSON maps with simdjson is quicker Jan 31, 2021
vadi2 and others added 22 commits February 1, 2021 09:07
* add sysDropUrlEvent

* api to download package via drag and drop and url in general

* override instal package

* Update src/mudlet-lua/lua/Other.lua

Co-authored-by: Vadim Peretokin <vperetokin@gmail.com>

* strip profile home from file name in verbose package install

Co-authored-by: Piotr Wilczynski <piotr.wilczynski@bisnode.com>
Co-authored-by: Vadim Peretokin <vperetokin@gmail.com>
* Clarify download progress message

Co-authored-by: Kebap <kebap_spam@gmx.net>
This finally enables us to do away with the text under the profile icons
as it replicates the normal `QListWidget` behaviour to move the selection
to each item which begins with the first letter of the key pressed.

This also makes the non-matching profiles get disabled which I think makes
the action more obvious...

Revised to include a timeout on concatenating keypresses:
This is to provide the same functionality that the QListWidget does for
keypresses, if the keys are typed fast enough they are concatenated to form
the text to search for. However if the user types slower than this then
the individual keys are only considered on their own. This reflects what
both Vadim and I experienced - in that he was typing fast enough to spell
out a particular profile but I wasn't and so the profile that I was having
selected was changing on each keypress.

The time between keypresses allowed so that they can be concatenated I have
set to be 1 second. This is significantly slower that the Qt code but it
is long enough to permit the temporary disablement of non-matching profiles
to be a useful visual feedback - and besides, it makes it easier to specify
one of a number of profiles with a common initial text - the longer the
timeout the more keys the user can string together!

Add `std::chrono` so we can give a time delay as `1s` instead of `1000` mS.

Signed-off-by: Stephen Lyons <slysven@virginmedia.com>
This is a recent bug introduced in the big error message handling code
refactor.

Also spotted a couple of uses of `QStringList::operator[]` where
`QStringList::at(...)` would be better.

Signed-off-by: Stephen Lyons <slysven@virginmedia.com>
Co-authored-by: Stephen Lyons <slysven@virginmedia.com>
Bumps [3rdparty/qtkeychain](https://github.com/Mudlet/qtkeychain) from `999a0b4` to `f719ccd`.
- [Release notes](https://github.com/Mudlet/qtkeychain/releases)
- [Commits](Mudlet/qtkeychain@999a0b4...f719ccd)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mudlet-machine-account <mudlet-machine-account@users.noreply.github.com>
Co-authored-by: mudlet-machine-account <mudlet-machine-account@users.noreply.github.com>
@vadi2
Copy link
Copy Markdown
Author

vadi2 commented Feb 10, 2021

Answer: it is not.

@vadi2 vadi2 closed this Feb 10, 2021
SlySven pushed a commit that referenced this pull request Dec 4, 2025
…rocessing (Mudlet#8571)

<!-- Keep the title short & concise so anyone non-technical can
understand it,
     the title appears in PTB changelogs -->
#### Brief overview of PR changes/additions
Fix: heap-use-after-free when cleanup runs during alias/trigger/key
processing
#### Motivation for adding to Mudlet
Fixes crash when running
Mudlet#8559 (comment)
benchmark on Linux.
#### Other info (issues closed, discussion etc)
==617553==ERROR: AddressSanitizer: heap-use-after-free on address
0x51200086e6d0 at pc 0x589b650367f6 bp 0x7ffc44dbc700 sp 0x7ffc44dbc6f8
READ of size 8 at 0x51200086e6d0 thread T0
#0 0x589b650367f5 in Tree<TAlias>::isActive() const
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xe8a7f5) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#1 0x589b65d81408 in TAlias::match(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd5408) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#2 0x589b6560c156 in AliasUnit::processDataStream(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1460156) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#3 0x589b65c872b4 in Host::send(QString, bool, bool)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1adb2b4) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#4 0x589b65d96517 in TCommandLine::enterCommand(QKeyEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bea517) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#5 0x589b65d93095 in TCommandLine::event(QEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1be7095) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#6 0x7ac668391c8a in QApplicationPrivate::notify_helper(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:3307:31
#7 0x7ac66839b2f0 in QApplication::notify(QObject*, QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:2725:39
#8 0x7ac668b83f7f in QCoreApplication::notifyInternal2(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/corelib/kernel/qcoreapplication.cpp:1109:24
#9 0x7ac66840cc0b in QWidgetWindow::event(QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qwidgetwindow.cpp:285:23
#10 0x7ac668391c8a in QApplicationPrivate::notify_helper(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:3307:31
#11 0x7ac668b83f7f in QCoreApplication::notifyInternal2(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/corelib/kernel/qcoreapplication.cpp:1109:24
#12 0x7ac6677ee8e2 in
QGuiApplicationPrivate::processKeyEvent(QWindowSystemInterfacePrivate::KeyEvent*)
/home/qt/work/qt/qtbase/src/gui/kernel/qguiapplication.cpp:2609:46
#13 0x7ac655cf9a04 in
QIBusPlatformInputContext::filterEventFinished(QDBusPendingCallWatcher*)
/home/qt/work/qt/qtbase/src/plugins/platforminputcontexts/ibus/qibusplatforminputcontext.cpp:523:57
#14 0x7ac668be8b74 in QtPrivate::QSlotObjectBase::call(QObject*, void**)
/home/qt/work/qt/qtbase/src/corelib/kernel/qobjectdefs_impl.h:461:57
#15 0x7ac668be8b74 in void doActivate<false>(QObject*, int, void**)
/home/qt/work/qt/qtbase/src/corelib/kernel/qobject.cpp:4255:30
#16 0x7ac6671e5142 in void QMetaObject::activate<void,
QDBusPendingCallWatcher*>(QObject*, QMetaObject const*, int, void*,
QDBusPendingCallWatcher* const&)
/home/qt/work/qt/qtbase/src/corelib/kernel/qobjectdefs.h:319:17
#17 0x7ac6671e5142 in
QDBusPendingCallWatcher::finished(QDBusPendingCallWatcher*)
/home/qt/work/qt/qtbase_build/src/dbus/DBus_autogen/include/moc_qdbuspendingcall.cpp:137:32
#18 0x7ac668bdd56b in QObject::event(QEvent*)
/home/qt/work/qt/qtbase/src/corelib/kernel/qobject.cpp:1411:31
#19 0x7ac668391c8a in QApplicationPrivate::notify_helper(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:3307:31
#20 0x7ac668b83f7f in QCoreApplication::notifyInternal2(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/corelib/kernel/qcoreapplication.cpp:1109:24
#21 0x7ac668b879e4 in
QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*)
/home/qt/work/qt/qtbase/src/corelib/kernel/qcoreapplication.cpp:1904:36
#22 0x7ac668e7d416 in postEventSourceDispatch
/home/qt/work/qt/qtbase/src/corelib/kernel/qeventdispatcher_glib.cpp:246:39
#23 0x7ac6667145c4 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5d5c4)
(BuildId: 1eb6131419edb83b2178b682829a6913cf682d75)
#24 0x7ac666773736 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xbc736)
(BuildId: 1eb6131419edb83b2178b682829a6913cf682d75)
#25 0x7ac666713a62 in g_main_context_iteration
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5ca62) (BuildId:
1eb6131419edb83b2178b682829a6913cf682d75)
#26 0x7ac668e7caad in
QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)
/home/qt/work/qt/qtbase/src/corelib/kernel/qeventdispatcher_glib.cpp:399:43
#27 0x7ac668b9002a in
QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)
/home/qt/work/qt/qtbase/src/corelib/kernel/qeventloop.cpp:186:22
#28 0x7ac668b8ba59 in QCoreApplication::exec()
/home/qt/work/qt/qtbase/src/corelib/kernel/qcoreapplication.cpp:1452:36
#29 0x589b64ab0675 in main
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x904675) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#30 0x7ac66602a1c9 in __libc_start_call_main
csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#31 0x7ac66602a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#32 0x589b649c1d04 in _start
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x815d04) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)

0x51200086e6d0 is located 16 bytes inside of 296-byte region
[0x51200086e6c0,0x51200086e7e8)
freed by thread T0 here:
#0 0x589b64a9b9f1 in operator delete(void*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x8ef9f1) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#1 0x589b65d80711 in TAlias::~TAlias()
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd4711) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#2 0x589b6560eefc in AliasUnit::doCleanup()
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1462efc) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#3 0x589b65c8e88d in Host::incomingStreamProcessor(QString const&, int)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1ae288d) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#4 0x589b651b73b5 in TMainConsole::runTriggers(int)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x100b3b5) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#5 0x589b64e39c4b in TBuffer::commitLine(char, unsigned long&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xc8dc4b) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#6 0x589b64e28d4e in
TBuffer::translateToPlainText(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char>>&, bool)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xc7cd4e) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#7 0x589b651b638d in
TMainConsole::printOnDisplay(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char>>&, bool)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x100a38d) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#8 0x589b64f9ed5b in TLuaInterpreter::feedTriggers(lua_State*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xdf2d5b) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#9 0x7ac66942ffa0 in luaD_precall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:320:10
#10 0x7ac66943ad7a in luaV_execute
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/lvm.c:591:17
#11 0x7ac66942e96c in luaD_call
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:378:5
#12 0x7ac66942af70 in luaD_rawrunprotected
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:116:3
#13 0x7ac66942bb94 in luaD_pcall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:464:12
#14 0x7ac66942bce0 in lua_pcall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/lapi.c:821:12
#15 0x589b64fd65f1 in TLuaInterpreter::call(QString const&, QString
const&, bool) (/home/vadi/Programs/Mudlet/build/src/mudlet+0xe2a5f1)
(BuildId: c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#16 0x589b65d84d31 in TAlias::execute()
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd8d31) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#17 0x589b65d84577 in TAlias::match(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd8577) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#18 0x589b6560c156 in AliasUnit::processDataStream(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1460156) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#19 0x589b65c872b4 in Host::send(QString, bool, bool)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1adb2b4) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#20 0x589b65d96517 in TCommandLine::enterCommand(QKeyEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bea517) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#21 0x589b65d93095 in TCommandLine::event(QEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1be7095) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#22 0x7ac668391c8a in QApplicationPrivate::notify_helper(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:3307:31

previously allocated by thread T0 here:
#0 0x589b64a9b171 in operator new(unsigned long)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x8ef171) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#1 0x589b6500f595 in TLuaInterpreter::startTempAlias(QString const&,
QString const&) (/home/vadi/Programs/Mudlet/build/src/mudlet+0xe63595)
(BuildId: c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#2 0x589b6512c41d in TLuaInterpreter::tempAlias(lua_State*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xf8041d) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#3 0x7ac66942ffa0 in luaD_precall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:320:10
#4 0x7ac66943ad7a in luaV_execute
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/lvm.c:591:17
#5 0x7ac66942e96c in luaD_call
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:378:5
#6 0x7ac66942af70 in luaD_rawrunprotected
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:116:3
#7 0x7ac66942bb94 in luaD_pcall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/ldo.c:464:12
#8 0x7ac66942bce0 in lua_pcall
/build/lua5.1-rMDsVj/lua5.1-5.1.5/src/lapi.c:821:12
#9 0x589b64fd65f1 in TLuaInterpreter::call(QString const&, QString
const&, bool) (/home/vadi/Programs/Mudlet/build/src/mudlet+0xe2a5f1)
(BuildId: c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#10 0x589b65d84d31 in TAlias::execute()
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd8d31) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#11 0x589b65d84577 in TAlias::match(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bd8577) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#12 0x589b6560c156 in AliasUnit::processDataStream(QString const&)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1460156) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#13 0x589b65c872b4 in Host::send(QString, bool, bool)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1adb2b4) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#14 0x589b65d96517 in TCommandLine::enterCommand(QKeyEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1bea517) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#15 0x589b65d93095 in TCommandLine::event(QEvent*)
(/home/vadi/Programs/Mudlet/build/src/mudlet+0x1be7095) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4)
#16 0x7ac668391c8a in QApplicationPrivate::notify_helper(QObject*,
QEvent*)
/home/qt/work/qt/qtbase/src/widgets/kernel/qapplication.cpp:3307:31

SUMMARY: AddressSanitizer: heap-use-after-free
(/home/vadi/Programs/Mudlet/build/src/mudlet+0xe8a7f5) (BuildId:
c98a5e4208b6daa52aa1b083c4ee6c4ab4552cc4) in Tree<TAlias>::isActive()
const
Shadow bytes around the buggy address:
  0x51200086e400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51200086e480: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x51200086e500: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x51200086e580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51200086e600: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
=>0x51200086e680: fa fa fa fa fa fa fa fa fd fd[fd]fd fd fd fd fd
  0x51200086e700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51200086e780: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x51200086e800: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x51200086e880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51200086e900: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==617553==ABORTING

Co-authored-by: Vadim Peretokin <vadi2@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants