I found that running the latest MacOS binary release v0.20.7 on Big Sur (Version 11.1) causes a hang where the CPU usage is at 100%. Running in the terminal shows:

➜  /Applications ./RTABMap.app/Contents/MacOS/RTABMap --udebug
[DEBUG] (2020-12-19 22:44:57.075) MainWindow.cpp:182::MainWindow()

I believe the issue is the Qt version it's linked against (otool reports 5.14.2), which may have a bug with Bug Sur. The following workaround, inspired by this post, has worked for me:

➜  /Applications QT_MAC_WANTS_LAYER=1 ./RTABMap.app/Contents/MacOS/RTABMap --udebug
[DEBUG] (2020-12-19 22:51:09.559) MainWindow.cpp:182::MainWindow()
[DEBUG] (2020-12-19 22:51:09.804) MainWindow.cpp:209::MainWindow() Setup ui...
...

I think the solution will be to build against Qt version 5.15 or later to resolve the issue, but I have not built it myself yet.