REST API Development with Qt 6

January 14, 2026

This post describes an experiment using Qt 6.7’s REST APIs to explore Stripe’s payment model, and what I learned building a small desktop developer tool.

Recent Qt releases have included several conveniences for developing clients of remote REST APIs. I recently tried it out with the Stripe payments REST API to get to grips with the Qt REST API in the real world. The overloading of the term API is unhelpful, I find, but hopefully not too confusing here.

As with almost everything I try out, I created Qt desktop tooling as a developer aid to exploring the Stripe API and its behavior. Naming things is hard, but given that I want to put a “Q” in the name, googling “cute stripes” gives lots of hits about fashion, and the other too-obvious-to-say pun, I’ve pushed it to GitHub as “Qashmere“:

setAlternatingRowColors(true);

Developers using REST APIs will generally be familiar with existing tooling such as Postman and Bruno, for synthesizing calls to collections of REST APIs. Indeed, Qashmere uses the Stripe Postman JSON definition to present the collection of APIs and parameters. Such tools have scripting interfaces and state to create workflows that a client of the REST API needs to support, like “create a payment, get the id of the payment back from the REST API and then cancel the payment with the id”, or “create a payment, get the id of the payment back from the REST API and then confirm it by id with a given credit card”.

So why create Qashmere? In addition to REST APIs, Stripe maintains objects which change state over time. The objects remain at REST until acted on by an external force, and when such an action happens a notification is sent to clients about those state changes, giving them a chance to react. I wanted to be able to collect the REST requests/responses and the notified events and present them as they relate to the Stripe objects. Postman doesn’t know about events or about Stripe objects in particular, except that it is possible to write a script in Postman to extract the object which is part of a JSON payload. Postman also doesn’t know that if a Payment Intent is created, there are a subset of next steps which could be in a workflow, such as cancel, capture or confirm payment etc.

Something that I discovered in the course of trying this out is that when I confirm a Payment Intent, a new Charge object is created and sent to me with the event notification system. Experimental experiences like that help build intuition.

Stripe operates with real money, but it also provides for sandboxes where synthetic payments, customers etc can be created and processed with synthetic payment methods and cards. As Qashmere is only useful as a developer tool or learning aid, it only works with Stripe sandboxes.

Events from Stripe are sent to pre-configured web servers owned by the client. The web servers need to have a public IP address, which is obviously not appropriate for a desktop application. A WebSocket API would be more suitable and indeed the stripe cli tool uses a WebSocket to receive events, but the WebSocket protocol is not documented or stable. Luckily the stripe cli tool can be used to relay events to another HTTP server, so Qashmere runs a QHttpServer for that purpose.

Implementation with Qt REST API

The QRestReply wraps a QNetworkReply pointer and provides convenience API for accessing the HTTP return code and for creating a QJsonDocument from the body of the response. It must be created manually if using QNetworkAccessManager directly. However the new QRestAccessManager wraps a QNetworkAccessManager pointer, again to provide convenience APIs and overloads for making requests that are needed in REST APIs (though some less common verbs like OPTIONS and TRACE are not built-in). The QRestAccessManager has conveniences like overloads that provide a way to supply callbacks which already take the QRestReply wrapper object as a parameter. If using a QJsonDocument request overload, the “application/json” Content-Type is automatically set in the header.

One of the inconveniences of QRestAccessManager is that in Qashmere I use an external definition of the REST API from the Postman definition which includes the HTTP method. Because the QRestAccessManager provides strongly typed API for making requests I need to do something like:

if (method == "POST") {
    rest.post(request, requestData, this, replyHandler);
} else if (method == "GET") {
    rest.get(request, this, replyHandler);
} else if (method == "DELETE") {
    rest.deleteResource(request, this, replyHandler);
}

There is a sendCustomRequest class API which can be used with a string, but it does not have an overload for QJsonDocument, so the convenience of having the Content-Type header set is lost. This may be an oversight in the QRestAccessManager API.

Another missing feature is URL parameter interpolation. Many REST APIs are described as something like /v1/object/:object_id/cancel, and it would be convenient to have a safe way to interpolate the parameters into the URL, such as:

QUrl result = QRestAccessManager::interpolatePathParameters(
    "/v1/accounts/:account_id/object/:object_id/cancel", {
        {"account_id", "acc_1234"},
        {"object_id", "obj_5678"}
    }
);

This is needed to avoid bugs such as a user-supplied parameter containing a slash for example.

Coding Con Currency

In recent years I’ve been writing and reading more Typescript/Angular code which consumes REST services, and less C++. I’ve enjoyed the way Promises work in that environment, allowing sequences of REST requests, for example, to be easy to write and read. A test of a pseudo API could await on requests to complete and invoke the next one with something like:

requestFactory.setBaseURL("http://some_service.com");
async testWorkflow(username: string, password: string) {
    const loginRequest = requestFactory.makeRequest("/login");
    const loginRequestData = new Map();
    loginRequestData.setParam("username", username);
    loginRequestData.setParam("password", password);
    const loginResponse = await requestAPI.post(
        loginRequest, loginRequestData);
    const bearerToken = loginResponse.getData();
    requestAPI.setBearerToken(bearerToken);
    const listingRequest = requestFactory.makeRequest("/list_items");
    const listingResponse = await requestAPI.get(listingRequest);
    const listing = JSON.parse(listingResponse.getData());
    const firstItemRequest = requestFactory.makeRequest(
        "/retrieve_item/:item_id", 
        {
            item_id: listing[0].item_id
        }
    );
    const firstItem = await requestAPI.get(firstItemRequest);
}

The availability of async functions and the Promise to await on make a test like this quite easy to write, and the in-application use of the API uses the same Promises, so there is little friction between application code and test code.

I wanted to see if I can recreate something like that based on the Qt networking APIs. I briefly tried using C++20 coroutines because they would allow a style closer to async/await, but the integration friction with existing Qt types was higher than I wanted for an experiment.

Using the methods in QtFuture however, we already have a way to create objects representing the response from a REST API. The result is similar to the Typescript example, but with different ergonomics, using .then instead of the async and await keywords.

struct RestRequest
{
    QString method;
    QString requestUrl;
    QHttpHeaders headers;
    QHash<QString, QString> urlParams;
    QUrlQuery queryParams;
    std::variant<QUrlQuery, QJsonDocument> requestData;
};
struct RestResponse
{
    QJsonDocument jsonDoc;
    QHttpHeaders headers;
    QNetworkReply::NetworkError error;
    QUrl url;
    int statusCode;
};
QFuture<RestResponse> makeRequest(RestRequest restRequest)
{
    auto url = interpolatePathParameters(
        restRequest.requestUrl, 
        restRequest.urlParams);
    auto request = requestFactory.createRequest(url);
    auto requestBodyDoc = extractRequestContent(restRequest.requestData);
    auto requestBody = requestBodyDoc.toJson(QJsonDocument::Compact);
    auto reply = qRestManager.sendCustomRequest(request,
                                                restRequest.method.toUtf8(),
                                                requestBody,
                                                &qnam,
                                                [](QRestReply &) {});
    return QtFuture::connect(reply, &QNetworkReply::finished).then(
        [reply]() {
            QRestReply restReply(reply);
    
            auto responseDoc = restReply.readJson();
            if (!responseDoc) {
                throw std::runtime_error("Failed to read response");
            }
    
            RestResponse response;
            response.jsonDoc = *responseDoc;
            response.statusCode = restReply.httpStatus();
            response.error = restReply.error();
            response.headers = reply->headers();
            response.url = reply->url();
    
            return response;
        }
    );
}

The QRestAccessManager API requires the creation of a dummy response function when creating a custom request because it is not really designed to be used this way. The result is an API accepting a request and returning a QFuture with the QJsonDocument content. While it is possible for a REST endpoint to return something else, we can follow the Qt philosophy of making the most expected case as easy as possible, while leaving most of the rest possible another way. This utility makes writing unit tests relatively straightforward too:

RemoteAPI remoteApi;
remoteApi.setBaseUrl(QUrl("https://dog.ceo"));
auto responseFuture = remoteApi.makeRequest(
    {"GET",
     "api/breed/:breed/:sub_breed/images/random",
     {},
     {
         {"breed", "wolfhound"}, 
         {"sub_breed", "irish"}
     }});
QFutureWatcher<RestResponse> watcher;
QSignalSpy spy(&watcher, &QFutureWatcherBase::finished);
watcher.setFuture(responseFuture);
QVERIFY(spy.wait(10000));
auto jsonObject = responseFuture.result().jsonDoc.object();
QCOMPARE(jsonObject["status"], "success");
QRegularExpression regex(
    R"(https://images\.dog\.ceo/breeds/wolfhound-irish/[^.]+.jpg)");
QVERIFY(regex.match(jsonObject["message"].toString()).hasMatch());

The result is quite similar to the Typescript above, but only because we can use spy.wait. In application code, we still need to use .then with a callback, but we can additionally use .onFailed and .onCanceled instead of making multiple signal/slot connections.

With the addition of QtFuture::whenAll, it is easy to make multiple REST requests at once and react when they are all finished, so perhaps something else has been gained too, compared to a signal/slot model:

RemoteAPI remoteApi;
remoteApi.setBaseUrl(QUrl("https://dog.ceo"));
auto responseFuture = remoteApi.requestMultiple({
    {
        "GET",
        "api/breeds/list/all",
    },
    {"GET",
     "api/breed/:breed/:sub_breed/images/random",
     {},
     {{"breed", "german"}, {"sub_breed", "shepherd"}}},
    {"GET",
     "api/breed/:breed/:sub_breed/images/random/:num_results",
     {},
     {{"breed", "wolfhound"}, 
     {"sub_breed", "irish"}, 
     {"num_results", "3"}}},
    {"GET", "api/breed/:breed/list", {}, {{"breed", "hound"}}},
});
QFutureWatcher<QList<RestResponse>> watcher;
QSignalSpy spy(&watcher, &QFutureWatcherBase::finished);
watcher.setFuture(responseFuture);
QVERIFY(spy.wait(10000));
auto four_responses = responseFuture.result();
QCOMPARE(four_responses.size(), 4);
QCOMPARE(four_responses[0].jsonDoc.object()["status"], "success");
QVERIFY(four_responses[0].jsonDoc.object()["message"].
    toObject()["greyhound"].isArray());
QRegularExpression germanShepherdRegex(
    R"(https://images.dog.ceo/breeds/german-shepherd/[^.]+.jpg)");
QCOMPARE(four_responses[1].jsonDoc.object()["status"], "success");
QVERIFY(germanShepherdRegex.match(
    four_responses[1].jsonDoc.object()["message"].toString()).hasMatch());
QRegularExpression irishWolfhoundRegex(
    R"(https://images.dog.ceo/breeds/wolfhound-irish/[^.]+.jpg)");
QCOMPARE(four_responses[2].jsonDoc.object()["status"], "success");
auto irishWolfhoundList = 
    four_responses[2].jsonDoc.object()["message"].toArray();
QCOMPARE(irishWolfhoundList.size(), 3);
QVERIFY(irishWolfhoundRegex.match(irishWolfhoundList[0].toString()).
    hasMatch());
QVERIFY(irishWolfhoundRegex.match(irishWolfhoundList[1].toString()).
    hasMatch());
QVERIFY(irishWolfhoundRegex.match(irishWolfhoundList[2].toString()).
    hasMatch());
QCOMPARE(four_responses[3].jsonDoc.object()["status"], "success");
auto houndList = four_responses[3].jsonDoc.object()["message"].toArray();
QCOMPARE_GE(houndList.size(), 7);
QVERIFY(houndList.contains("afghan"));
QVERIFY(houndList.contains("basset"));
QVERIFY(houndList.contains("blood"));
QVERIFY(houndList.contains("english"));
QVERIFY(houndList.contains("ibizan"));
QVERIFY(houndList.contains("plott"));
QVERIFY(houndList.contains("walker"));

setAutoDeleteReplies(false);

I attempted to use new API additions in recent Qt 6 versions to interact with a few real-world REST services. The additions are valuable, but it seems that there are a few places where improvements might be possible. My attempt to make the API feel closer to what developers in other environments might be accustomed to had some success, but I’m not sure QFuture is really intended to be used this way.
Do readers have any feedback? Would using QCoro improve the coroutine experience? Is it very unusual to create an application with QWidgets instead of QML these days? Should I have used PyQt and the python networking APIs?

Tags: , , , ,
Posted in KDE, Qt | 1 Comment »

Grantlee version 5.3.1 now available

November 11, 2022

I’ve just made a new 5.3.1 release of Grantlee. The 5.3.0 release had some build issues with Qt 6 which should now be resolved with version 5.3.1.

Unlike previous releases, this release will not appear on http://www.grantlee.org/downloads/. I’ll be turning off grantlee.org soon. All previous releases have already been uploaded to https://github.com/steveire/grantlee/releases.

The continuation of Grantlee for Qt 6 is happening as KTextTemplate so as not to be constrained by my lack of availability. I’ll only make new Grantlee patch releases as needed to fix any issues that come up in the meantime.

Many thanks to the KDE community for taking up stewardship and ownership of this library!

Tags: ,
Posted in KDE, Qt | 3 Comments »

Grantlee version 5.3 now available

September 10, 2022

I previously announced the end of new Qt5-based Grantlee releases. The Grantlee template system is to find new life as part of KDE Frameworks 6 in the form of KTextTemplate. The Grantlee textdocument library will probably become part of another KDE library with similar scope.

Meanwhile, some changes have accumulated since the last Grantlee release, so I’ve made a new release to make them available to users. Many of the changes are small, but with a few new features which were cherry-picked from the Cutelee repo.

The other significant change is that Grantlee 5.3.0 can be built with Qt 6. This is not a change of plan regarding migration to KDE Frameworks, but is intended to assist with porting existing code to Qt 6.

Speaking of new releases, we welcomed our baby into the world almost a year ago. Years ago it was a common refrain within the KDE community to new parents to remind them, tongue in cheek, to never shake the baby. I was amused to find that the advise given as a printed book to all new Irish parents reminds the same :).

Tags: , ,
Posted in Qt, Uncategorized | 6 Comments »

Location, Location, Location

April 27, 2021

As of a few days ago, a new feature in clang-query allows introspecting the source locations for a given clang AST node. The feature is also available for experimentation in Compiler Explorer. I previously delivered a talk at EuroLLVM 2019 and blogged in 2018 about this feature and others to assist in discovery of AST matchers and source locations. This is a major step in getting the Tooling API discovery features upstream into LLVM/Clang.

Background

When creating clang-tidy checks to perform source to source transformation, there are generally two steps common to all checks:

  • Matching on the AST
  • Replacing particular source ranges in source files with new text

To complete the latter, you will need to become familiar with the source locations clang provides for the AST. A diagnostic is then issued with zero or more “fix it hints” which indicate changes to the code. Almost all clang-tidy checks are implemented in this way.

Some of the source locations which might be interesting for a FunctionDecl are illustrated here:

Pick Your Name

A common use case for this kind of tooling is to port a large codebase from a deprecated API to a new API.

A tool might replace a member call pushBack with push_back on a custom container, for the purpose of making the API more like standard containers. It might be the case that you have multiple classes with a pushBack method and you only want to change uses of it on a particular class, so you can not simply find and replace across the entire repository.

Given test code like 

    struct MyContainer
    {
        // deprected:
        void pushBack(int t);

        // new:
        void push_back(int t);    
    };

    void calls()
    {
        MyContainer mc;

        mc.pushBack(42);
    }

A matcher could look something like:

    match cxxMemberCallExpr(
    on(expr(hasType(cxxRecordDecl(hasName("MyContainer"))))),
    callee(cxxMethodDecl(hasName("pushBack")))
    )

Try experimenting with it on Compiler Explorer.

An explanation of how to discover how to write this AST matcher expression is out of scope for this blog post, but you can see blogs passim for that too.

Know Your Goal

Having matched a call to pushBack the next step is to replace the source text of the call with push_back. The call to mc.pushBack() is represented by an instance of CXXMemberCallExpr. Given the instance, we need to identify the location in the source of the first character after the “.” and the location of the opening paren. Given those locations, we create a diagnostic with a FixItHint to replace that source range with the new method name:

    diag(MethodCallLocation, "Use push_back instead of pushBack")
        << FixItHint::CreateReplacement(
            sourceRangeForCall, "push_back");

When we run our porting tool in clang-tidy, we get output similiar to:

warning: Use push_back instead of pushBack [misc-update-pushBack]
    mc.pushBack(42);
       ^~~~~~~~
       push_back

Running clang-tidy with -fix then causes the tooling to apply the suggested fix. Once we have tested it, we can run the tool to apply the change to all of our code at once.

Find Your Place

So, how do we identify the sourceRangeForCall?

One way is to study the documentation of the Clang AST to try to identify what API calls might be useful to access that particular source range. That is quite difficult to determine for newcomers to the Clang AST API.

The new clang-query feature allows users to introspect all available locations for a given AST node instance.

note: source locations here
    mc.pushBack(42);
       ^
 * "getExprLoc()"

note: source locations here
    mc.pushBack(42);
                  ^
 * "getEndLoc()"
 * "getRParenLoc()"

With this output, we can see that the location of the member call is retrievable by calling getExprLoc() on the CXXMemberCallExpr, which happens to be defined on the Expr base class. Because clang replacements can operate on token ranges, the location for the start of the member call is actually all we need to complete the replacement.

One of the design choices of the srcloc output of clang-query is that only locations on the “current” AST node are part of the output. That’s why for example, the arguments of a function call are not part of the locations output for a CXXMemberCallExpr. Instead it is necessary to traverse to the argument and introspect the locations of the node which represents the argument.

By traversing to the MemberExpr of the CXXMethodCallExpr we can see more locations. In particular, we can see that getOperatorLoc() can be used to get the location of the operator (a “.” in this case, but it could be a “->” for example) and getMemberNameInfo().getSourceRange() can be used to get a source range for the name of the member being called.

The Best Location

Given the choice of using getExprLoc() or getMemberNameInfo().getSourceRange(), the latter is preferable because it is more semantically related to what we want to replace. Aside from the hint that we want the “source range” of the “member name”, the getExprLoc() should be disfavored as that API is usually only used to choose a position to indicate in a diagnostic. That is not specifically what we wish to use the location for.

Additionally, by experimenting with slightly more complex code, we can see that getExprLoc() on a template-dependent call expression does not give the desired source location (At time of publishing! – This is likely undesirable in this case). At any rate, getMemberNameInfo().getSourceRange() gives the correct source range in all cases.

In the end, our diagnostic can look something like: 

    diag(MethodCallLocation, "Use push_back instead of pushBack")
        << FixItHint::CreateReplacement(
            theMember->getMemberNameInfo().getSourceRange(), "push_back");

This feature is a powerful way to discover source locations and source ranges while creating and maintaining clang-tidy checks. Let me know if you find it useful!

Tags: , ,
Posted in clang, Events, KDE, Qt | 1 Comment »

AST Matchmaking made easy

February 14, 2021

The upcoming version of Clang 12 includes a new traversal mode which can be used for easier matching of AST nodes.

I presented this mode at EuroLLVM and ACCU 2019, but at the time I was calling it “ignoring invisible” mode. The primary aim is to make AST Matchers easier to write by requiring less “activation learning” of the newcomer to the AST Matcher API. I’m analogizing to “activation energy” here – this mode reduces the amount of learning of new concepts must be done before starting to use AST Matchers.

The new mode is a mouthful – IgnoreUnlessSpelledInSource – but it makes AST Matchers easier to use correctly and harder to use incorrectly. Some examples of the mode are available in the AST Matchers reference documentation.

In clang-query, the mode affects both matching and dumping of AST nodes and it is enabled with:

set traversal IgnoreUnlessSpelledInSource

while in the C++ API of AST Matchers, it is enabled by wrapping a matcher in:

traverse(TK_IgnoreUnlessSpelledInSource, ...)

The result is that matching of AST nodes corresponds closely to what is written syntactically in the source, rather than corresponding to the somewhat arbitrary structure implicit in the clang::RecursiveASTVisitor class.

Using this new mode makes it possible to “add features by removing code” in clang-tidy, making the checks more maintainable and making it possible to run checks in all language modes.

Clang does not use this new mode by default.

Implicit nodes in expressions

One of the issues identified is that the Clang AST contains many nodes which must exist in order to satisfy the requirements of the language. For example, a simple function relying on an implicit conversion might look like.

struct A {
    A(int);
    ~A();
};

A f()
{
    return 42;
}

In the new IgnoreUnlessSpelledInSource mode, this is represented as 

ReturnStmt
`-IntegerLiteral '42'
and the integer literal can be matched with 
returnStmt(hasReturnValue(integerLiteral().bind("returnVal")))

In the default mode, the AST might be (depending on C++ language dialect) represented by something like:

ReturnStmt
`-ExprWithCleanups
  `-CXXConstructExpr
    `-MaterializeTemporaryExpr
      `-ImplicitCastExpr
        `-CXXBindTemporaryExpr
          `-ImplicitCastExpr
            `-CXXConstructExpr
              `-IntegerLiteral '42'

To newcomers to the Clang AST, and to me, it is not obvious what all of the nodes there are for. I can reason that an instance of A must be constructed. However, there are two CXXConstructExprs in this AST and many other nodes, some of which are due to the presence of a user-provided destructor, others due to the temporary object. These kinds of extra nodes appear in most expressions, such as when processing arguments to a function call or constructor, declaring or assigning a variable, converting something to bool in an if condition etc.

There are already AST Matchers such as ignoringImplicit() which skip over some of the implicit nodes in AST Matchers. Still though, a complete matcher for the return value of this return statement looks something like

returnStmt(hasReturnValue(
    ignoringImplicit(
        ignoringElidableConstructorCall(
            ignoringImplicit(
                cxxConstructExpr(hasArgument(0,
                    ignoringImplicit(
                        integerLiteral().bind("returnVal")
                        )
                    ))
                )
            )
        )
    ))

Another mouthful.

There are several problems with this.

  • Typical clang-tidy checks which deal with expressions tend to require extensive use of such ignoring...() matchers. This makes the matcher expressions in such clang-tidy checks quite noisy
  • Different language dialects represent the same C++ code with different AST structures/extra nodes, necessitating testing and implementing the check in multiple language dialects
  • The requirement or possibility to use these intermediate matchers at all is not easily discoverable, nor are the required matchers to saitsfy all language modes easily discoverable
  • If an AST Matcher is written without explicitly ignoring implicit nodes, Clang produces lots of surprising results and incorrect transformations

Implicit declarations nodes

Aside from implicit expression nodes, Clang AST Matchers also match on implicit declaration nodes in the AST. That means that if we wish to make copy constructors in our codebase explicit we might use a matcher such as

cxxConstructorDecl(
    isCopyConstructor()
    ).bind("prepend_explicit")

This will work fine in the new IgnoreUnlessSpelledInSource mode.

However, in the default mode, if we have a struct with a compiler-provided copy constructor such as:

struct Copyable {
    OtherStruct m_o;
    Copyable();
};

we will match the compiler provided copy constructor. When our check inserts explicit at the copy constructor location it will result in:

struct explicit Copyable {
    OtherStruct m_o;
    Copyable();
};

Clearly this is an incorrect transformation despite the transformation code “looking” correct. This AST Matcher API is hard to use correctly and easy to use incorrectly. Because of this, the isImplicit() matcher is typically used in clang-tidy checks to attempt to exclude such transformations, making the matcher expression more complicated.

Implicit template instantiations

Another surpise in the behavior of AST Matchers is that template instantiations are matched by default. That means that if we wish to change class members of type int to type safe_int for example, we might write a matcher something like

fieldDecl(
    hasType(asString("int"))
    ).bind("use_safe_int")

This works fine for non-template code.

If we have a template like

template  
struct TemplStruct {
    TemplStruct() {}
    ~TemplStruct() {}

private:
    T m_t;
};

then clang internally creates an instantiation of the template with a substituted type for each template instantation in our translation unit.

The new IgnoreUnlessSpelledInSource mode ignores those internal instantiations and matches only on the template declaration (ie, with the T un-substituted).

However, in the default mode, our template will be transformed to use safe_int too:

template  
struct TemplStruct {
    TemplStruct() {}
    ~TemplStruct() {}

private:
    safe_int m_t;
};

This is clearly an incorrect transformation. Because of this, isTemplateInstantiation() and similar matchers are often used in clang-tidy to exclude AST matches which produce such transformations.

Matching metaphorical code

C++ has multiple features which are designed to be simple expressions which the compiler expands to something less-convenient to write. Range-based for loops are a good example as they are a metaphor for an explicit loop with calls to begin and end among other things. Lambdas are another good example as they are a metaphor for a callable object. C++20 adds several more, including rewriting use of operator!=(...) to use !operator==(...) and operator<(...) to use the spaceship operator.

[I admit that in writing this blog post I searched for a metaphor for “a device which aids understanding by replacing the thing it describes with something more familiar” before realizing the recursion. I haven’t heard these features described as metaphorical before though…]

All of these metaphorical replacements can be explored in the Clang AST or on CPP Insights.

Matching these internal representations is confusing and can cause incorrect transformations. None of these internal representations are matchable in the new IgnoreUnlessSpelledInSource mode.

In the default matching mode, the CallExprs for begin and end are matched, as are the CXXRecordDecl implicit in the lambda and hidden comparisons within rewritten binary operators such as spaceship (causing bugs in clang-tidy checks).

Easy Mode

This new mode of AST Matching is designed to be easier for users, especially newcomers to the Clang AST, to use and discover while offering protection from typical transformation traps. It will likely be used in my Qt-based Gui Quaplah, but it must be enabled explicitly in existing clang tools.

As usual, feedback is very welcome!

Tags: , ,
Posted in clang, Events, KDE, Qt | 3 Comments »

Grantlee version 5.2 (Alles hat ein Ende, nur die Wurst hat zwei) now available

December 18, 2019

The Grantlee community is pleased to announce the release of Grantlee version 5.2.0.

For the benefit of the uninitiated, Grantlee is a set of Qt based libraries including an advanced string template system in the style of the Django template system.

{# This is a simple template #}
{% for item in list %}
    {% if item.quantity == 0 %}
    We're out of {{ item.name }}!
    {% endif %}
{% endfor %}

Included in this release contains a major update to the script bindings used in Grantlee to provide Javascript implementations of custom features. Allan Jensen provided a port from the old QtScript bindings to new bindings based on the QtQml engine. This is a significant future-proofing of the library. Another feature which keeps pace with Qt is the ability to introspect classes decorated with Q_GADGET provided by Volker Krause. Various cleanups and bug fixes make up the rest of the release. I made some effort to modernize it as this is the last release I intend to make of Grantlee.

This release comes over 3 and a half years after the previous release, because I have difficulty coming up with new codenames for releases. Just joking of course, but I haven’t been making releases as frequently as I should have, and it is having an impact on the users of Grantlee, largely in KDE applications. To remedy that, I am submitting Grantlee for inclusion in KDE Frameworks. This will mean releases will happen monthly and in an automated fashion. There is some infrastructure to complete in order to complete that transition, so hopefully it will be done early in the new year.

Posted in KDE, Qt | 3 Comments »

The Future of AST-Matching refactoring tools (EuroLLVM and ACCU)

April 30, 2019

I recently made a trip to LLVM in Brussels and ACCU in Bristol. It was a busy week. I gave a talk at both conferences on the topic of the future of AST Matchers-based refactoring.

As usual, the ‘hallway track’ also proved useful at both conferences, leading to round-table discussions at the LLVM conference with other interested contributors and getting to talk to other developers interested in refactoring tooling at ACCU.

Presentations

The learning curve for AST-Matcher-based refactoring is currently too steep. Most C++ developers who are not already familiar with the internal Clang APIs need to invest a lot in order to learn how to make such bespoke tooling to improve and maintain their codebase.

The presentations both include demos of steps I’ve been taking to try to address these problems.

The first demo is of clang-query discovery features which aim to reduce the need to infer AST Matcher code by examining the Clang AST itself. I also showed the debugging features I am preparing to upstream to clang-query. Finally – in terms of demo content – I showed a Qt tool which can eliminate some of the additional difficulties and problems of long-developer-iteration-time.

The debugging features and the Qt tool were world exclusives at the LLVM conference (and at the ACCU conference because most people didn’t attend both 🙂 ). I hadn’t shown them to anyone else before, so I was quite happy the demos went well.

Videos

My 25 minute presentation to the LLVM developers tried to show that these changes can make mechanical refactoring more easily available to C++ developers.

The aim was to show the features to the LLVM community to

  1. illustrate the issues as I see them
  2. get some feedback about whether this is a good direction
  3. introduce myself for the sake of further code reviews (and collaborators). As this was my first LLVM conference, I am not already familiar with most of the attendees.

My 1.5 hour ACCU presentation is a far-less-rushed presentation of the same tools and a repetition of some of the content at code::dive 2018. In the ACCU presentation, the new demo content starts about 30 minutes in. This talk is the one to watch if you are interested in using mechanical refactoring on your own code.

Feedback was very positive from both talks, so I’m happy with that.

Qt Tooling

Earlier this year I refactored the clang AST dump functionality. It was previously implemented in one class, ASTDumper, which had the dual responsibilities of traversing the clang AST and creating a textual representation of it. I separated the textual output from the generic output independent traversal, which introduces the possibility of alternative output formats such as JSON.

Of course, given my KDE and Qt contribution history, I would only create a generic tree traversal class in order to implement QAbstractItemModel for it.

The demos show all of the features you would expect from a point-and-click refactoring tool including exploring, feature discovery, debugging with in-source feedback, live source updates, experimental refactoring etc.

Of course, all of this requires changes to Clang upstream (for example to add the debugging interface) which was the point of my visit to EuroLLVM. Hopefully, once enough of the changes are upstream, I’ll be able to open source the tool.

The idea as always is to hopefully have enough functionality in Clang itself that IDEs such as Qt-Creator, KDevelop and Visual Studio would be able to integrate these features using their own GUI APIs, making the simple tool I made obsolete anyway. I only made it for demo purposes.

This will take the mechanical refactoring workflow which is currently

and turn it into

You will still do the same things, but with much faster development iteration to achieve the same result.

There is even more that can be done to make the process of mechanical refactoring with clang easier and faster. We discussed some of that at EuroLLVM, and hopefully all the pieces can come together soon. Meanwhile I’ll be upstreaming some of this work, talking at NDC Oslo, and at my local meetup group on this topic.

Posted in clang, Events, KDE, Qt, Slides | 4 Comments »

Debugging Clang AST Matchers

April 16, 2019

Last week I flew to Brussels for EuroLLVM followed by Bristol for ACCU.

At both conferences I presented the work I’ve been doing to make it easier for regular C++ programmers to perform ‘mechanical’ bespoke refactoring using the clang ASTMatchers tooling. Each talk was prepared specifically for the particular audience at that conference, but both were very well received. The features I am working on require changes to the upstream Clang APIs in order to enable modern tooling, so I was traveling to EuroLLVM to try to build some buy-in and desire for those features.

I previously delivered a talk on the same topic about AST Matchers at code::dive 2018. This week I presented updates to the tools and features that I have worked on during the 6 months since.

One of the new features I presented is a method of debugging AST Matchers.

Part of the workflow of using AST Matchers is an iterative development process. For example, the developer wishes to find functions of a particular pattern, and creates and ever-more-complex matcher to find all desired cases without false-positives. As the matcher becomes more complex, it becomes difficult to determine why a particular function is not found as desired.

The debugger features I wrote for AST Matchers intend to solve that problem. It is now possible to create, remove and list breakpoints, and then enable debugger output to visualize the result of attempting to match at each location. A simple example of that is shown here.

When using a larger matcher it becomes obvious that the process of matching is short-circuited, meaning that the vertically-last negative match result is the cause of the overall failure to match the desired location. The typical workflow with the debugger is to insert break points on particular lines, and then remove surplus breakpoints which do not contribute useful output.

This feature is enabled by a new interface in the Clang AST Matchers, but the interface is also rich enough to implement some profiling of AST Matchers in the form of a hit counter.

Some matchers (and matcher sub-trees) are slower/more expensive to run than others. For example, running a matcher like `matchesName` on every AST node in a translation unit requires creation of a regular expression object, and comparing the name of each AST node with the regular expression. That may result in slower runtime than trimming the search tree by checking a parameter count first, for example.

Of course, the hit counter does not include timing output, but can give an indication of what might be relevant to change. Comparison of different trees of matchers can then be completed with a full clang-tidy check.

There is much more to say about both conferences and the tools that I demoed there, but that will be for a future log post. I hope this tool is useful and helps discover and debug AST Matchers!

Posted in clang, Events, KDE, Qt, Slides | 4 Comments »

Refactor with Clang Tooling at code::dive 2018

January 2, 2019

I delivered a talk about writing a refactoring tool with Clang Tooling at code::dive in November. It was uploaded to YouTube today:

The slides are available here and the code samples are here.

This was a fun talk to deliver as I got to demo some features which had never been seen by anyone before. For people who are already familiar with clang-tidy and clang-query, the interesting content starts about 15 minutes in. There I start to show new features in the clang-query interpreter command line.

The existing clang-query interpreter lacks many features which the replxx library provides, such as syntax highlighting and portable code completion:

It also allows scrolling through results to make a selection:

A really nice feature is value-based code-completion instead of type-based code completion. Existing code completion only completes candidates based on type-compatibility. It recognizes that a parameterCountIs() matcher can be used with a functionDecl() matcher for example. If the code completion already on the command line is sufficiently constrained so that there is only one result already, the code completion is able to complete candidates based on that one result node:

Another major problem with clang-query currently is that it is hard to know which parenthesis represents the closing of which matcher. The syntax highlighting of replxx help with this, along with a brace matching feature I wrote for it:

I’m working on upstreaming those features to replxx and Clang to make them available for everyone, but for now it is possible to experiment with some of the features on my Compiler Explorer instance on ce.steveire.com.

I wrote about the AST-Matcher and Source Location/Source Range discovery features on my blog here since delivering the talk. I also wrote about Composing AST Matchers, which was part of the tips and tricks section of the talk. Over on the Visual C++ blog, I wrote about distributing the refactoring task among computers on the network using Icecream. My blogs on that platform can be seen in the Clang category.

All of that blog content is repeated in the code::dive presentation, but some people prefer to learn from conference videos instead of blogs, so this might help the content reach a larger audience. Let me know if there is more you would like to see about clang-query!

Posted in clang, Events, KDE, Qt, Slides | 8 Comments »

Composing AST Matchers in clang-tidy

November 20, 2018

When creating clang-tidy checks, it is common to extract parts of AST Matcher expressions to local variables. I expanded on this in a previous blog.

auto nonAwesomeFunction = functionDecl(
  unless(matchesName("^::awesome_"))
  );

Finder->addMatcher(
  nonAwesomeFunction.bind("addAwesomePrefix")
  , this);

Finder->addMatcher(
  callExpr(callee(nonAwesomeFunction)).bind("addAwesomePrefix")
  , this);

Use of such variables establishes an emergent extension API for re-use in the checks, or in multiple checks you create which share matcher requirements.

When attempting to match items inside a ForStmt for example, we might encounter the difference in the AST depending on whether braces are used or not.

#include <vector>

void foo()
{
    std::vector<int> vec;
    int c = 0;
    for (int i = 0; i < 100; ++i)
        vec.push_back(i);

    for (int i = 0; i < 100; ++i) {
        vec.push_back(i);
    }
}

In this case, we wish to match the push_back method inside a ForStmt body. The body item might be a CompoundStmt or the CallExpr we wish to match. We can match both cases with the anyOf matcher.

auto pushbackcall = callExpr(callee(functionDecl(hasName("push_back"))));

Finder->addMatcher(
    forStmt(
        hasBody(anyOf(
            pushbackcall.bind("port_call"), 
            compoundStmt(has(pushbackcall.bind("port_call")))
            ))
        )
    , this);

Having to list the pushbackcall twice in the matcher is suboptimal. We can do better by defining a new API function which we can use in AST Matcher expressions:

auto hasIgnoringBraces = [](auto const& Matcher)
{
    return anyOf(
        Matcher, 
        compoundStmt(has(Matcher))
        );
};

With this in hand, we can simplify the original expression:

auto pushbackcall = callExpr(callee(functionDecl(hasName("push_back"))));

Finder->addMatcher(
    forStmt(
        hasBody(hasIgnoringBraces(
            pushbackcall.bind("port_call")
            ))
        ) 
    , this);

This pattern of defining AST Matcher API using a lambda function finds use in other contexts. For example, sometimes we want to find and bind to an AST node if it is present, ignoring its absense if is not present.

For example, consider wishing to match struct declarations and match a copy constructor if present:

struct A
{
};

struct B
{
    B(B const&);
};

We can match the AST with the anyOf() and anything() matchers.

Finder->addMatcher(
    cxxRecordDecl(anyOf(
        hasMethod(cxxConstructorDecl(isCopyConstructor()).bind("port_method")), 
        anything()
        )).bind("port_record")
    , this);

This can be generalized into an optional() matcher:

auto optional = [](auto const& Matcher)
{
    return anyOf(
        Matcher,
        anything()
        );
};

The anything() matcher matches, well, anything. It can also match nothing because of the fact that a matcher written inside another matcher matches itself.

That is, matchers such as

functionDecl(decl())
functionDecl(namedDecl())
functionDecl(functionDecl())

match ‘trivially’.

If a functionDecl() in fact binds to a method, then the derived type can be used in the matcher:

functionDecl(cxxMethodDecl())

The optional matcher can be used as expected:

Finder->addMatcher(
    cxxRecordDecl(
        optional(
            hasMethod(cxxConstructorDecl(isCopyConstructor()).bind("port_method"))
            )
        ).bind("port_record")
    , this);

Yet another problem writers of clang-tidy checks will find is that AST nodes CallExpr and CXXConstructExpr do not share a common base representing the ability to take expressions as arguments. This means that separate matchers are required for calls and constructions.

Again, we can solve this problem generically by creating a composition function:

auto callOrConstruct = [](auto const& Matcher)
{
    return expr(anyOf(
        callExpr(Matcher),
        cxxConstructExpr(Matcher)
        ));
};

which reads as ‘an Expression which is any of a call expression or a construct expression’.

It can be used in place of either in matcher expressions:

Finder->addMatcher(
    callOrConstruct(
        hasArgument(0, integerLiteral().bind("port_literal"))
        )
    , this);

Creating composition functions like this is a very convenient way to simplify and create maintainable matchers in your clang-tidy checks. A recently published RFC on the topic of making clang-tidy checks easier to write proposes some other conveniences which can be implemented in this manner.

Posted in clang, KDE, Qt, Slides | 4 Comments »

« Older Entries
Design a site like this with WordPress.com
Get started