TL;DR: The final solution can be found below.

Calling functions from another thread with signals

When I started to work with multithreading in Qt, I did all thread communication with signals and slots, because that was the only simple way I knew. I called functions from another thread using additional signals.

class Manager : public QObject
{
    Q_OBJECT
public:
    void action(const QString &parameter);
    Q_SIGNAL void actionRequested(const QString &parameter);
};

If Manager runs on thread A you can’t just call action() of course when you’re operating on thread B. However, you can emit manager->actionRequested(parameter) and Qt will call the connected slot (action()) on thread A using a Qt::QueuedConnection.

This approach has two big issues though:

1. You need to create weird signals for every function you want to call from another thread.
2. You can’t handle the results.

Now there are of course solutions to point 2, you can i.e. create another requested-signal on the calling object and call back, but your code will get much harder to understand.

It gets even worse if there are multiple places from where a function needs to be called. How do you know which callback to execute then?

I also knew about the old-variant of QMetaObject::invokeMethod which essentially has the same result handling problem, but additionally isn’t even checked at compile-time.

QMetaObject::invokeMethod(manager, "action", Qt::QueuedConnection,
                          Q_ARG(QString, parameter));

Executing a lambda on a QThread

I always wanted a solution that would allow to execute a lambda with captured variables as that would solve all of my issues. At some point I had the idea to create one signal/slot pair with a std::function<void()> parameter to do that, but it’s even easier. Since Qt 5.10 there’s a new version of QMetaObject::invokeMethod() that can do exactly what I needed.

auto parameter = QStringLiteral("Hello");
QMetaObject::invokeMethod(otherThreadsObject, [=] {
    qDebug() << "Hello from otherThreadsObject's thread" << parameter;
});

Note: You can’t use a QThread object as argument here, because the QThread object itself lives in the thread where it has been created.

I personally don’t like the name QMetaObject::invokeMethod, so I added an alias to my projects:

template<typename Function>
auto runOnThread(QObject *targetObject, Function function)
{
    QMetaObject::invokeMethod(targetObject, std::move(function));
}

The result handling as a caller also works pretty well:

runOnThread(object, [this, object] {
    // on object's thread
    auto value = object->action();

    runOnThread(this, [value]() {
        // on caller's thread again
        qDebug() << "Calculated result:" << value;
    });
});

One could only complain that all your code needs to be indented deeper each time you call runOnThread(). However, that could potentially be solved using C++20 coroutines.

However you should also be aware of the disadvantages of having multiple database threads:

1. Creating many database connections is going to cost you a lot of memory. This might be a point on weaker desktop computers or mobile devices.
2. If you’ve only got a modest amount of queries to be executed it might even be a bit slower since you can’t reuse prepared queries across threads. Also database connection creation doesn’t come without any costs.

In the end you should test whether the gained performance improvements are actually relevant and whether they outweigh the memory usage. You should also examine how well your database driver handles multithreading, e.g. sqlite can always only commit one transaction at a time. The good news is that at least the code doesn’t get complicated.

The solution

What we currently have got is a QThreadPool with only one thread. That is because the database connections aren’t able to handle queries from different threads, so we can’t just increase the maximum thread count.

So to fix this we need to create one database connection per thread. There’s just one problem: the thread pool doesn’t tell us when and which thread it’s creating or destructing, so we don’t know when we need to create the database connection and when to remove it again. Fortunately for us Qt has got a ready-made thread storage for exactly this kind of issue. It works like this:

• There’s a global static QThreadStorage
• On every thread you can check whether the storage contains a value for this thread and get or set it.
• When any QThread exits the thread storage automatically destructs the content for the thread. The thread storage takes ownership of the objects, so this also works with heap-allocated objects.

We’re going to use an extra class for the thread storage since we manually need to create and delete the database connection.

class DatabaseConnection
{
    Q_DISABLE_COPY(DatabaseConnection)
public:
    DatabaseConnection()
        : m_name(QString::number(QRandomGenerator::global()->generate(), 36))
    {
        auto database = QSqlDatabase::addDatabase(QStringLiteral("QSQLITE"), m_name);
        database.setDatabaseName("/home/me/data.sqlite");
        database.open();
    }

    ~DatabaseConnection()
    {
        QSqlDatabase::removeDatabase(m_name);
    }

    QSqlDatabase database()
    {
        return QSqlDatabase::database(m_name);
    }

private:
    QString m_name;
};

An object of this class will automatically create a new database connection and remove it as soon as it’s destroyed. Two things are important here:

• Each QSqlDatabase needs to have a unique name. In this case we generate a name consisting out of random alphanumerical characters, but there’re also other solutions. It’s just important that the name is unique.
• The QSqlDatabase isn’t stored as this would otherwise cause trouble when calling QSqlDatabase::removeDatabase().

Now we’ll see how to execute queries with the new code, but first let’s have a look at our current code:

QFuture<QList<User>> Database::fetchUsers()
{
    return QtConcurrent::run(m_pool, [this]() {
        QSqlQuery query(QSqlDatabase::database("main-connection"));
        query.exec("SELECT * FROM users");

        QList<User> users;
        while (query.next()) {
            // ...
        }
        return users;
    });
}

We now need to use the correct database connection on each thread and also need to create them if necessary.

// the storage for the database connection for each thread
static QThreadStorage<DatabaseConnection *> databaseConnections;

QSqlDatabase Database::currentDatabase()
{
    if (!databaseConnections.hasLocalData()) {
        databaseConnections.setLocalData(new DatabaseConnection());
    }
    return databaseConnections.localData()->database();
}

QFuture<QList<User>> Database::fetchUsers()
{
    return QtConcurrent::run(m_pool, [this]() {
        QSqlQuery query(currentDatabase());
        query.exec("SELECT * FROM users");

        QList<User> users;
        while (query.next()) {
            // ...
        }
        return users;
    });
}

The open() function from our previous version isn’t needed anymore and can be removed, because this is now done in our DatabaseConnection class. What’s still left to do is to adjust the thread pool settings in the constructor of our Database class:

// thread limit (default: number of system threads)
m_pool.setMaxThreadCount(4);
// unused threads are destroyed after 2 minutes (default: 30s)
m_pool.setExpiryTimeout(120000);

Thread creation and deletion is time-consuming, so you might want to choose an even higher expiry timeout here. On the other hand you might still want to save memory when the application is inactive. If you don’t care about the memory consumption, you can also disable the thread expiry (setting it to -1), that will avoid any recreation of database connections.

And basically this is it! The queries are distributed on multiple threads and are executed in parallel now!

There’re just some smaller things that don’t work like before anymore. We will take a look at two important issues and see how we can solve them.

Table creation & database migrations

In many cases you want your application to take care of table creations and database migrations. The database migrations you might already have written are fine and don’t need to be touched, but there’s one thing we need to pay attention to now: no other query must be started until all tables have been created correctly, so no errors are produced.

There are probably many ways of fixing this problem straightforwardly, here is one way:

class Database
{
    // ...
private:
    // creates a query and ensures that the tables are ready
    QSqlQuery createQuery();
    // executes database migrations to create all tables
    void createTables();

    QMutex m_tableCreationMutex;
    bool m_tablesCreated = false;
};

QSqlQuery Database::createQuery()
{
    if (!m_tablesCreated) {
        createTables();
    }
    return QSqlQuery(currentDatabase());
}

void Database::createTables()
{
    QMutexLocker locker(&m_tableCreationMutex);
    if (m_tablesCreated) {
        return;
    }
    
    QSqlQuery query(currentDatabase());
    query.exec("CREATE TABLE IF NOT EXISTS ...");

    m_tablesCreated = true;
}

If you’re using createQuery() everywhere to create queries, it’s now always safe that all database migrations have been done already.

Reusing queries

Queries can be reused in order to avoid that the SQL query string needs to be parsed every time. In a single-threaded environment we could use static QSqlQueries like this:

static auto query = []() {
    // this is only executed once
    QSqlQuery query = createQuery();
    query.prepare("SELECT * FROM users WHERE id = :1");
    return query;
}();

query.addBindValue(userId);
query.exec();

// ...

This way the query is reused and the query is only parsed once. If this query is executed very frequently this significantly increases the performance, but also (guess what) consumes more memory.

And again, this needs some adjustments to work with multithreading since the queries can’t be shared between different threads of course.

C++11 to the rescue! C++11 introduced a new keyword thread_local and this at least makes it possible to easily reuse the query for each thread. The code remained exactly the same, except there’s now a little new thread_local, which ensures that each thread is going to have its own query:

thread_local static auto query = []() {
    // this is only executed once on each thread
    QSqlQuery query = createQuery();
    query.prepare("SELECT * FROM users WHERE id = :1");
    return query;
}();

query.addBindValue(userId);
query.exec();

// ...

But of course keep in mind that this is only useful if a query is executed frequently, otherwise you just need more memory, especially since the query is cached for each thread now.

I hope I could help you with that. Feel free to comment if you’ve got any other solutions or ideas in mind or in production!

Multithreading in Qt

When working with Qt most of the time you do not need to care about threading. Most things already work asynchronously, they don’t block and there’s no reason to mess with additional threads. This is the case for network requests via the QNetworkAccessManager where signals are used (as pretty much everywhere). If you’ve got other tasks like hash calculation of large files or image scaling, then there’s QtConcurrent::run() for you which will execute a function on the application’s thread pool.

QtConcurrent uses QFutures to report the results. In Qt 5 QFutures are not very handy (you need a QFutureWatcher to get the results asynchronously and you manually need to create and delete it). With Qt 6 this has changed and now there’s a pretty nice QFuture::then() function where you can directly pass a lambda handling the result:

QtConcurrent::run([]() {
    // do heavy calculations ...
    return value;
}).then([](Result value) {
    qDebug() << "Calculation done:" << value;
});

This simple way to chain QFutures creates a nice flow in your code. Instead of having (maybe multiple) slots that all need to be connected, here you can just write your code directly behind the QtConcurrent::run() call.

Databases and threading

That’s all pretty nice, but it doesn’t work with databases. You can’t just open a QSqlDatabase and execute queries on it via QtConcurrent. Most (if not all) SQL database drivers are going to complain that you’re using the database from the wrong thread. So, what you need to do is creating the database on the same thread as where the queries are executed. This means we can’t use QtConcurrent (at least not without some adjustments) since we don’t know on which thread our job is going to be executed.

Solution #1: Thread-Worker model with signals

My first idea on how to solve this and how I also did it in the past was creating a normal QThread and a database worker class running on the thread. The database is opened on this new thread and all queries are also executed on it. Now with this approach we somehow need to trigger queries and receive their results. Qt offers signals and slots with queued connections for this. My approach was to create two signals and a slot on the database worker. Let’s look at an example here:

class DatabaseWorker : public QObject
{
    Q_OBJECT
public:
    DatabaseWorker(QObject *parent = nullptr);

signals:
    void fetchAllUsersRequested();
    void allUsersFetched(const QList<User> &users);

private slots:
    void fetchAllUsers();
};

DatabaseWorker::DatabaseWorker(QObject *parent)
    : QObject(parent)
{
    connect(this, &DatabaseWorker::fetchAllUsersRequested,
            this, &DatabaseWorker::fetchAllUsers);
}

DatabaseWorker::fetchAllUsers()
{
    // ... do query ..
    emit allUsersFetched(users);
}

If you want to fetch the users, you’d do the following:

emit database->fetchAllUsersRequested();

As soon as the database has executed the query, it will emit the allUsersFetched() signal, which you can handle.

However this approach has some problems:

1. You need to care about threading manually.
2. You need to create signals and connect them for every new function.
3. The caller (of fetchAllUsersRequested()) doesn’t know which request belonged to the results received from the signal (allUsersFetched()). This is not a problem in this case, but as soon as you’ve got multiple requests at the same time, this will get important.

The second point can be workarounded, but the code won’t be nice.

Solution #2: QtConcurrent with a one thread QThreadPool

Using QtConcurrent with QFutures would solve all three problems here, so we should have a deeper look at QtConcurrent. In the documentation we can see that QtConcurrent also provides the option to use a specific QThreadPool for the execution.

This helps us since with a custom thread pool we can set the maximum thread count to 1 and so this way can guarantee that everything is executed on the same thread. QThreadPool automatically deletes threads when they’re unused. We also need to prevent this, because the used thread of course must not change:

QThreadPool pool;
// limit to one thread
pool.setMaxThreadCount(1);
// prevent automatic deletion and recreation
pool.setExpiryTimeout(-1);

This basically already solved our problem. We now just need to do everything via QtConcurrent and our QThreadPool and we don’t need to care about threads at all anymore.

Our Database now could look like this:

class Database : public QObject
{
    Q_OBJECT
public:
    QFuture<void> open();

private:
    QThreadPool m_pool;
}

QFuture<void> Database::open()
{
    return QtConcurrent::run(m_pool, []() {
        auto database = QSqlDatabase::addDatabase("QSQLITE", "main-connection");
        database.setDatabaseName("/home/me/data.sqlite");
        database.open();
        // of course you should do some more checks here to see
        // whether everything went well :)
    });
}

Other queries can be done like this now:

QFuture<QList<User>> Database::fetchUsers()
{
    return QtConcurrent::run(m_pool, [this]() {
        QSqlQuery query(QSqlDatabase::database("main-connection"));
        query.exec("SELECT * FROM users");

        QList<User> users;
        while (query.next()) {
            // ...
        }
        return users;
    });
}

And with Qt 6 you can pretty easily handle the results now:

database->fetchUsers().then([](const QList<User> &users) {
    // do something
});

Unfortunately with Qt 5 it’s not so nice:

auto *watcher = new QFutureWatcher<QList<User>>();
connect(watcher, &QFutureWatcherBase::finished, [watcher]() {
    QList<User> users = watcher->result();
    // do something with the result
    watcher->deleteLater();
});
watcher->setFuture(database->fetchUsers());

This is not very nice and things can go wrong, you could forget to delete the QFutureWatcher for example. Thus, we use a template function to simplify this:

template<typename T, typename Handler>
void await(const QFuture<T> &future, QObject *context, Handler handler)
{
    auto *watcher = new QFutureWatcher<T>(context);
    QObject::connect(watcher, &QFutureWatcherBase::finished,
                     context, [watcher, handler { std::move(handler) }]() {
        handler(watcher->result());
        watcher->deleteLater();
    });
    watcher->setFuture(future);
}

await(database->fetchUsers(), this, [](QList<User> users) {
    // do something
});

And that already looks much better! :)

We now got a solution with the following features:

1. A database class running in the main thread. No manual thread management.
2. No signals and slots for each new function. Just QFuture::then / await.
3. Request and result handling can be easily ‘linked’ by using lambda captures.

This is a very nice solution for most of the use cases in our applications.

Do we want anything more? – Okay, maybe parallel queries with multiple threads, for building servers or a high-performing application, but that’s probably irrelevant in most of the cases. However, maybe we’ll see this in one of the next blog posts. :)

In April last year I setup the kaidan.im XMPP server with ejabberd. As you would expect I did that completely manually, so first running apt get install ejabberd, then editing the config file and so on. I read the ejabberd documentation in detail to enable all useful features and to get the nearly perfect server config. I also imported the database from a friend of mine, who hosted his server (jbb.ddns.net) on a Raspberry Pi before.

Then, in July I got a small job to install an XMPP server for the DBJR (Deutscher Bundesjugendring). It was already clear that we’ll first test the XMPP server on a small (Hetzner) cloud server and later move it to a larger one as the project grows. Besides the internal server chat.dbjr.org we also set up the yochat.eu domain (youth organization chat) to be opened for free registration later. Because we didn’t want to do all the setup multiple times (and researching everything after forgetting everything), I came up with the idea of using ansible to automate everything. The result was nice, but still very server specific. Later I based on the ansible playbook for moving my own server. Recently I also did some real abstraction, so now only some config options would need to be changed to get the chat.dbjr.org-server. On 9th January, I gave a talk about this at the XMPP Meetup.

How to setup your own server

My ansible role is publicly available on my GitLab instance. I’ll show you how to use it in this blog post, but we won’t go into details about ansible. The first step is to create a new ansible playbook for this project (as long as you don’t have one yet). Just create this folder structure (we’ll talk about the speific files later):

$ ls myplaybook/ myplaybook/env/ myplaybook/env/host_vars/
myplaybook/:
env/  roles/  mysetup.yml

myplaybook/env/:
myhosts.inventory  host_vars/

myplaybook/env/host_vars/:
myxmppserver.yml

Then clone the ejabberd and certbot roles into the roles folder:

git clone ssh://git@git.kaidan.im/lnj/ansible-role-ejabberd ejabberd
git clone ssh://git@git.kaidan.im/lnj/ansible-role-certbot certbot

After that you should add the target server to the inventory file:

# this is the group name:
[xmppservers]
# this is the server name (as you would use it when connecting via. ssh); you
# could also write `user@xmpp.server.example`, but it's better to add that info
# to your ~/.ssh/config
myxmppserver

Now copy the default configurations of the roles to your host configuration (remove the three dashes --- from the second file):

cat roles/ejabberd/defaults/main.yml roles/certbot/defaults/main.yml > env/host_vars/myxmppserver.yml

You can edit that file and adjust everything to your needs. The config options should all be self-explaining (contact me if they’re not). If you need some advanced options you can just edit the ejabberd.yml template file in the ejabberd role. So now what’s still missing is an ansible script (i.e. mysetup.yml) that will execute the ejabberd role on your server:

---

- name: Install ejabberd
  hosts: xmppservers # this is the group name, you could also specify a single host
  become: true
  roles:
    - certbot
    - ejabberd

And that’s basically it, you can now execute the playbook on your server using this command:

ansible-playbook -i env/myhosts.inventory mysetup.yml

If you’re using this, it’ll probably be much easier for you to setup the server (and especially when moving it to another machine!).

As you probably know, Qt provides many builds for different systems and architectures. They can be very useful, if you want to develop with the latest libraries, even if your distro doesn’t provide them (or if you’re using Windows/Mac and you got no Qt at all). In my I case I wanted to use the latest Qt on an LTS CI system. I know that there are PPAs for Ubuntu, however, if you want to use the Qt for Android builds, there’s really no other way: either you compile everything yourself or you need to use the official installer.

There’s just one problem: Qt only provides a graphical installer, which can’t be on a server. This has been a problem for a long time and there are some workarounds using the Qt scripting interface with a virtual display, but that workaround isn’t that easy to use and error-prone. You can find an example of this on stackoverflow.

Since the Qt Installer basically only lets you select the Qt version and then downloads & extracts everything, I thought that it couldn’t be too hard to rewrite that in a simple script. I browsed a bit through the Qt installer repository and started to write a python script that should output the URLs for a specific Qt version. Turns out: the URLs are not consistent in all Qt versions and they contain long build numbers. To solve this I needed to parse the Updates.xml file for each target/version. These files list all the packages and their dependencies. From these we only need the main package (it contains most modules as QtCore, QtSql, QtSvg and many more, only QtCharts and similar are missing) and can ignore its dependencies (it only depends on the qtcreator, documentation and examples, which we all don’t need for our purpose).

In the end I came up with a python script that can download and extract you every Qt version you want (even iOS builds can be downloaded on a Linux host system, which wasn’t possible before :D). The script (currently) just calls the p7zip commands for extracting the packages which isn’t optimal, but OK.

So if you would like to use the official Qt builds on a server and don’t want to argue with the graphical installer, you might want to try my script. It’s available on my GitLab instance sourcehut repo. Here is an example how to use it on debian-based systems:

sudo apt install python3-requests p7zip-full wget

wget https://git.kaidan.im/lnj/qli-installer/raw/master/qli-installer.py
chmod +x qli-installer.py

./qli-installer.py 5.12.0 linux desktop

This will download and extract Qt 5.12.0 to a folder called 5.12.0 in your current directory. You can find more details in the README. If you have any suggestions or you’ve found a bug (the installer repository format changes from time to time), please submit an issue.