Dmitry's Blog

Mcp Sse The Hard Way

Sun, 14 Sep 2025 00:00:00 +0000

Based on https://deadprogrammersociety.com/2025/03/calling-mcp-servers-the-hard-way.html.

In one terminal:

➜ curl http://127.0.0.1:64342/sse                                                            
data: /message?sessionId=41d13160-1698-420c-b5d3-3f925ac5c019
event: endpoint

In another terminal (copying session id from above):

➜ MCP_ENDPOINT=http://127.0.0.1:64342/message?sessionId=41d13160-1698-420c-b5d3-3f925ac5c019
➜ curl -X POST "${MCP_ENDPOINT}" -H "Content-Type: application/json" -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
        "name": "get_file_problems",
        "arguments": {
            "errorsOnly": false,
            "filePath": "pro/ai/mcp/client/src/main/kotlin/org/http4k/ai/mcp/client/McpClient.kt"
        }
    }
}'
Accepted%

The first terminal shows the response:

data: {"id":1,"jsonrpc":"2.0","result":{"content":[{"text":"{\"filePath\":\"pro/ai/mcp/client/src/main/kotlin/org/http4k/ai/mcp/client/McpClient.kt\",\"errors\":[{\"severity\":\"WARNING\",\"description\":\"Function \\\"staaop\\\" is never used\",\"lineContent\":\"    fun staaop() = close()\",\"line\":29,\"column\":9}]}","type":"text"}],"isError":false,"_meta":{}}}
event: message

Tidy Kotlin

Tue, 26 Dec 2023 00:00:00 +0000

This blog post is a catalog of Kotlin tidyings based on my experience of writing server-side Kotlin. The term “tidying” is inspired by the “Tidy First?” book and essentially means a small refactoring. Some of them are specific to Kotlin, while others are applicable to any programming language. I plan to keep this catalog updated so it might naturally evolve over time.

Be aware these are not rules but suggestions on how to improve the code. It’s also not a comprehensive list, so there are other forces affecting the design not mentioned here. These forces can be technical, for example, objects’ lifetime or DRY, and non-technical, for example, code style preferences of other people or priorities of the project. Depending on the context it might be better to avoid or delay tidying.

Many of these tidyings are about code style and formatting, which are sometimes regarded as cursory issues. While it’s true that projects always have more significant issues, and it’s possible to read and navigate the code presented in any sensible way, the main premise of the tidyings is that fixing small things does matter. One reason is that small fixes are more likely to happen than bigger ones because they normally require a limited amount of effort and it’s easier to see the end state. Instead of planning a large-scale improvement at some point later, which might never happen, we can benefit from small improvements of code ergonomics now, which will accumulate over time. Another reason to fix small and seemingly insignificant code issues is that the interaction with the code via tydings gives us more of a tactile experience of the code, which can help us discover better design, and really understand the code and its changeability constraints. Finally, some of the issues are bigger than they seem to be or they act as gatekeepers for bigger refactorings. By tidying we can unlock a bigger area of improvement or get an insight to ask the right question.

==== This is work-in-progress. Feel free to share the link but be aware that the content will change. Feedback is welcome via email or social media. ====

High-level declarations first
Maximum privacy
Keep variables close to their usages
Inline variables with single usage
Remove argument names when their types are distinct
Add argument names when types are the same or generic
Pass named arguments in the order of parameter declaration
Put parameters on one line
Put arguments on one line
Put parameters on separate lines
Stop the CONSTANT SHOUTING
Consider using tiny types
…

High-level declarations first

Declare high-level interfaces, classes, functions, or properties before implementation details. A more formal way to think about it is by presenting code as a directed graph, where nodes are declarations and edges point from the declaration to its usages. The code should be ordered so that all edges point to the beginning of the file (unless there are circular dependencies). With the edges pointing up, declarations at the top of the file will be visually higher, matching the “high/low-level” metaphor.

The main motivation for this tidying is to improve the navigability of the codebase.

Let’s assume we’re mostly familiar with the codebase and would like to remind ourselves what FruitStore interface looks like. We open FruitStore.kt containing the following code. We have to skim the file from points 1️⃣ to 4️⃣ until we finally get to the FruitStore interface at points 5️⃣ and 6️⃣.

sealed interface Fruit {...} 1️⃣ 👀 data class Apple(...) : Fruit {...} 2️⃣ 👀 data class Banana(...) : Fruit {...} 3️⃣ 👀 data class StorageId(...) {...} 4️⃣ 👀 interface FruitStore { fun store(apples: List<Apple>): Result<StorageId> 5️⃣ 👀 fun store(bananas: List<Banana>): Result<StorageId> 6️⃣ 👀 } class FruitStoreInTheCloud(...) : FruitStore { override fun store(apples: List<Apple>): Result<StorageId> {...} override fun store(bananas: List<Banana>): Result<StorageId> {...} }

Real-world APIs can be much bigger and might have intermediate interfaces, so we can’t just stop skimming at the first interface. We also can’t start scrolling bottom-up (as if reading bottom-up was normal) in case there are implementation(s) of the interface.

The best solution is to put the most important thing in the FruitStore.kt, i.e. FruitStore interface at the top of the file. After opening the file, we can go straight to FruitStore at points 1️⃣ and 2️⃣.

interface FruitStore { fun store(apples: List<Apple>): Result<StorageId> 1️⃣ 👀 fun store(bananas: List<Banana>): Result<StorageId> 2️⃣ 👀 } sealed interface Fruit {...} data class Apple(...) : Fruit {...} data class Banana(...) : Fruit {...} data class StorageId(...) {...} class FruitStoreInTheCloud : FruitStore { override fun store(apples: List<Apple>): Result<StorageId> {...} override fun store(bananas: List<Banana>): Result<StorageId> {...} }

In this example, putting higher-level abstraction first saved us from skimming four other classes, which you might argue is not that much. This is a fair point. One problem is that the cost can be higher in a real code and it tends to grow as the codebase grows. Another is that small issues (as well as improvements) can have a cumulative effect. Ignoring the navigation issue in one place is ok, ignoring it across the whole codebase is much worse.

Maximum privacy

Make values, functions and classes as private as possible. There are a few reasons for that:

Private declaration is easier to understand because the context in which it’s used is limited and it’s likely to have fewer usages to analyse.
Because it’s easier to understand, it’s easier to modify (or delete). With fewer usages, there are fewer things that can go wrong when we change the function/class.
It’s also easier for the compiler/IDE/editor to type check or analyse the code because there are fewer places to check.
Reduced coupling. As the codebase evolves private code is less likely to be used and pull the design of the function/class in various directions, so there is less design tension.
Namespace pollution. Unnecessarily public functions/classes show up in auto-completion making it harder to choose the right function/class.
Specific to Kotlin support in IntelliJ, when searching for short public names and fields of data classes can be really (unusably) slow, so you’ll have to resort to pure text search.

Imagine we open a file with the code below and try to understand what it’s doing. The FruitStoreInTheCloud class somewhat makes sense. It stores apples/bananas and returns an ID on success. The handle() function is a bit of a mystery though. The name is too generic, its type signature is not particularly revealing and its implementation (collapsed in the code below) makes us wonder if it’s mostly redundant. We try Find Usages, but the name is too generic and the IDE search doesn’t finish in a reasonable time. We get bored wondering if the search will ever finish. Text search finishes quickly with too many results to be sure they all match the same handle() function. We try the final trick of leaning on the compiler. We make the function private and compile the project. Luckily, it just works meaning that the handle() function is not used anywhere else, so running FruitStoreInTheCloudTests should be enough to validate our tidying if we choose to go ahead with it. We could’ve saved some time and a few WTFs if only the function was private in the first place.

class FruitStoreInTheCloud : FruitStore { override fun store(apples: List<Apple>): Result<StorageId> {...} override fun store(bananas: List<Banana>): Result<StorageId> {...} } fun handle(fruit: Fruit, f: (Fruit) -> Unit): Fruit {...} 👀 😕

There is a question if fields in test classes need to be as private as possible. All the reasons above still apply and it’s good to have fewer special cases by using the same rules for production and test code, so this would be my preference. The main argument against it is that test classes rarely share their fields so adding the private keyword for each field and function is unnecessarily verbose. This is a valid reason and interestingly earlier versions of Kotlin had internal as a default visibility (see this blog post). If you go down this path, you might want to reconfigure IntelliJ to apply inspection which suggests making the field/function private only to tests.

Looking at this tidying from the code as data point of view, this is a form of encapsulation. This might be a reasonable analogy considering the code base and people involved in software development as a sociotechnical system.

Keep variables close to their usages

Keep variables (both vals and vars) close to where you use them. This is a simple way of saying to declare variables in the innermost scope and within the minimum amount of lines/statements from usages. The main goal of this tidying is higher code cohesion or more specifically:

Reduced cognitive load while reading code. With fewer things to remember, you can focus on other aspects of the code.
Discover/communicate that variables are only related to a specific scope. Since a smaller scope can provide a higher privacy level, you also get the benefits of Maximum privacy.
Discover/communicate dependencies. As soon as you try moving a variable, it will “pull” on all the variables/functions that are using it. You might find that they can all be moved into a smaller scope. This is a way to organise code into “clusters”.
Cohesive code can lead to other tidyings like inline single usage, or extract function.

Scopes in Kotlin have a specific well-defined meaning but the for purpose of this tidying it might be useful to think of “scope” in a more vague way as “an area of code which makes sense as a whole”. For example, paragraphs can be cohesive enough to make sense on their own (sure, there is a question if they should be extracted into functions, but we don’t know yet if it’s possible before doing the tidying).

To illustrate the distance between variable declaration and its usage(s) imagine we come across a function with the following code. There are two lines between point 1️⃣ and the only usage of apple, and four lines between point 2️⃣ and the only usage of banana. This creates coupling between the paragraphs of code, or “tension” which we could overall quantify as 2 + 4 = 6.

val apple = Apple(...) 1️⃣ 👀 val banana = Banana(...) │ 2️⃣ 👀 │ │ process(apple) ◄─┘ │ doSomething() │ │ process(banana) ◄────┘ doSomethingElse()

After the tidying, the code might look like in the snippet below.

val apple = Apple(...) 1️⃣ 👀 process(apple) ◄─┘ doSomething() val banana = Banana(...) 2️⃣ 👀 process(banana) ◄─┘ doSomethingElse()

It now has only two paragraphs and the tension between apple/banana and their usages is zero (this makes us wonder if they should be inlined). Sure declaraing variable in the paragraph does not guarantee it can’t be used somewhere else, but it still makes it easier to skim the code especially once we have done a few of these tidyings and can trust the codebase. What the compiler does enforce though is that banana cannot be used above point 2️⃣ (we did reduce its scope after all).

To illustrate reduced variable scope imagine we look at the following function. The type is declared in the scope of the whole function but is only used within forEach.

val type = Blended ─┐ 👀 fruits.forEach { fruit -> │ process(fruit, type) │ log("Processed $fruit as $type") │ } │ doSomething() │ doSomethingElse() ─┘

After the tidy-up the code looks like this:

fruits.forEach { fruit -> val type = Blended ─┐ 👀 process(fruit, type) │ log("Processed $fruit as $type") ─┘ } doSomething() doSomethingElse()

The amount of lines where type can be used has reduced from 6 to 2 lines. Now when type usages are all within the same paragraph, we might wonder if type can be inlined. It’s not ideal to duplicate Blended in two places, so maybe we don’t need to log the type? Or maybe we do, and then the question might be why the process() function won’t log it for us. These questions are examples of how simple tidying can help us explore/discover software design.

Interestingly, this tidying implies that, for example, a single usage of a magic value in a function might be better off extracted into a local variable rather than a constant.

Moving variables close to their usages is often a good idea but it’s not a fixed rule. Sometimes it might be worth moving all/most declarations to some outer scope and see if it reveals a better structure of the code. It is an exploration.

Similar to a few other tidyings this one is fractal. While the examples above discuss low-level abstractions, it’s possible to apply the same principles at the class, file, module, service, application, or architectural level.

Inline variables with single usage

Variables (both vals and vars) with single usage are often better off inlined. You might think of it as an extreme version of keeping variables close to their usages, so some of the motivations overlap. The reasons to inline single usage are:

Inlined variables are obviously not used elsewhere in the code, so we save time and effort by not checking if there are other usages.
To communicate the structure of nested objects or function calls.
The order of variable initialisation doesn’t dictate how variables are ordered in the code, so it’s easier to have high-level declarations first.
Extracting a single usage variable made sense in Java to give the argument a name. Kotlin has named arguments that have the same expressiveness (unless you want to have a name different from the parameter).

To illustrate how inlining variables can help, imagine we are trying to figure out how exactly FruitStoreInTheCloud() is constructed. We search for the usages of the constructor (by invoking “Show Usages” before the first “(“) and end up in the following code at point 1️⃣. We would like to see how the constructor arguments are created, so we navigate to uri declaration which takes us to point 2️⃣. URI construction looks ok, but it’s not clear if uri might be used elsewhere in the function. We search for uri usages. There is only one usage, so we end up back at point 1️⃣. We repeat the same process for credentials navigating to point 3️⃣ and back to 1️⃣ (it’s the only usage). Navigation is a bit more tricky with config because we first go to point 4️⃣, then to connectionTimeout 5️⃣ and back, then to retryAttempts 6️⃣ and back. This is a lot of non-linear navigation for the construction of two nested objects!

To be fair, we could’ve been more efficient by using the “Highlight Usages in File” action on all variables and visually checking the editor scrollbar area if there are any matches outside of visible code. It’s still not worth it though considering that all usages can be inlined.

val uri = Uri("https://fruit.cloud") 2️⃣ val credentials = Credentials(...) 3️⃣ val connectionTimeout = 10.seconds 5️⃣ val retryAttempts = 3 6️⃣ val config = Config(connectionTimeout, retryAttempts) 4️⃣ val store = FruitStoreInTheCloud(uri, credentials, config) 1️⃣ 👀 😵‍💫 doSomething() doSomethingElse() ...

After inlining variable usages, we can follow the constructor arguments points 1️⃣ to 6️⃣ in a linear way. We don’t need to guess if the variables are used elsewhere. And it’s nice that indentation at points 5️⃣ and 6️⃣ communicates the nested structure of objects.

val store = FruitStoreInTheCloud( 1️⃣ 👀 uri = Uri("https://fruit.cloud"), 2️⃣ credentials = Credentials(...), 3️⃣ config = Config( 4️⃣ connectionTimeout = 10.seconds, 5️⃣ retryAttempts = 3 6️⃣ ) ) doSomething() doSomethingElse() ...

As a side note, the FruitStoreInTheCloud() constructor in the example above has arguments with distinct types, so it might be ok to remove argument names.

What if some of the arguments have multiple usages (for example, if uri was passed to a function), is it still worth inlining variables with single usage? I would argue that it’s almost always worth trying. Reduced scope is likely to pay off the inconsistency of declaring arguments both in place and as variables.

Remove argument names when their types are distinct

When all arguments have distinct incompatible types, named arguments might be redundant and can be removed. Named arguments can help with accidentally passing value to the wrong parameter when types are the same. But if all types are different, this is not a problem as it will be checked by the compiler. Assuming that argument values have descriptive names, named arguments don’t bring any benefits to justify verbosity. This is more likely to be the case when using tiny types.

For example, given that uri, credentials, and config all have different types, argument names can be removed in the code below. Once removed, we end up with one argument per line and can put them on one line.

val store = FruitStoreInTheCloud( uri = uri, credentials = credentials, config = config ) ...

The code after tidying:

val store = FruitStoreInTheCloud(uri, credentials, config) ...

Note that in IntelliJ there is a “Remove all argument names” intention, which can be invoked via the Alt+Enter popup menu or assigned its own shortcut.

Add argument names when types are the same or generic

When arguments have the same or generic types, add argument names to make them distinct and clarify their meaning. One reason is that it can take some effort to tell the difference between the arguments of the same type. This is error-prone because we can pass values in the wrong order or introduce a bug while reordering parameters. Another reason is that generic types don’t communicate the meaning of the value, so it’s harder to understand the code without IDE/editor help, which is also an invitation for subtle bugs.

IntelliJ can help us by showing names as inlay hints or parameter information at the cursor. The inlay hints are useful, but they don’t provide the guarantees of the compiler. And because you can’t know if the reader of the code will have hints enabled, it’s safer to assume nothing. Parameter information is displayed in a popup window, so we might end up checking every argument while the popup is visible or try remembering the order. Either way, it’s a bit too much effort.

For example, given the following constructor, the meaning of empty strings and numbers 10 and 3 is not very clear without looking up the FruitStoreInTheCloud declaration or using IDE support.

val store = FruitStoreInTheCloud("https://fruit.cloud", "", "", 10, 3) ...

The code after tidying makes the meaning of the arguments more obvious.

val store = FruitStoreInTheCloud( url = "https://fruit.cloud", user = "", password = "", connectionTimeout = 10, retryAttempts = 3 ) ...

Argument names have increased the length of the line to the point that we had to put parameters on separate line. So while the argument names made the code less error-prone and more understandable, they did it at the cost of verbosity. It’s up to us to decide if it was worth it or if there is another avenue for tidying, e.g. introducing tiny types.

Note that in IntelliJ there is an “Add names to call arguments” intention, which can be invoked via the Alt+Enter popup menu or assigned its own shortcut.

Pass named arguments in the order of parameter declaration

Pass named arguments in function/constructor invocations in the same order as parameter declaration. The reason is that if parameters are declared in a meaningful order, then arguments can only benefit from following it. Consistent ordering can also be useful when comparing multiple function/constructor invocations by doing a visual comparison or textual diff. Finally, because the “Change signature” refactoring automatically reorders arguments to be in the order of parameters even on an unrelated change, it is beneficial to have arguments in this order so that the refactoring is not mixed with the argument reordering.

For example, given the following invocations of the Config constructor, it’s not obvious what the differences are between them.

val config = Config( retryAttempts = 10, callTimeout = INFINITE, connectionTimeout = 3.seconds ) ... val anotherConfig = Config( connectionTimeout = 10.seconds, retryAttempts = 3, callTimeout = 5.minutes )

With the arguments arranged in the same order, it’s easier to notice the differences (and maybe ask ourselves why these differences exist).

val config = Config( connectionTimeout = 3.seconds, callTimeout = INFINITE, retryAttempts = 10 ) ... val anotherConfig = Config( connectionTimeout = 10.seconds, callTimeout = 5.minutes, retryAttempts = 3 )

Put parameters on one line

When a function or class constructor declaration has only a few parameters, put them on one line. How few is enough to justify the tidying is subjective and depends on the length of parameter names, the length and complexity of parameter types, the length of the default values, the surrounding code, readers’ attention span, etc. The motivation is to have “optimal” information density on the screen.

With one or two parameters per line, the information density per line is a bit low, so we might end up scrolling the source code up and down a lot, reading it almost as a single column.

data class Password( val value: String ) data class Credentials( val user: String, val password: Password )

Putting class or function parameters on a single line will increase the information density and can make it easier to skim.

data class Password(val value: String) data class Credentials(val user: String, val password: Password)

Note that in IntelliJ there is a “Put parameters on one line” intention, which can be invoked via the Alt+Enter popup menu or assigned its own shortcut.

Put arguments on one line

When a function or constructor invocation has only a few arguments, put them on one line. How few is enough to justify the tidying is subjective and depends on the length of argument names (especially with named arguments), the surrounding code, readers’ attention span, etc. The motivation is to have “optimal” information density on the screen.

One or two arguments on separate lines are often a good opportunity for the tidy-up. You might notice in the example below that arguments are vertically misaligned with constructors, so we have to read the code from right to left. Sometimes this is inevitable, but in this case, it’s easy to fix by putting arguments on one line.

val password = Password( ↙️ 👀 "********" ) val credentials = Credentials( ↙️ 👀 user = "Bob", password = password )

The code after tidying:

val password = Password("********") val credentials = Credentials(user = "Bob", password = password)

The next tidying might be to inline single usage of password or to remove argument names.

Note that in IntelliJ there is a “Put arguments on one line” intention, which can be invoked via the Alt+Enter popup menu or assigned its own shortcut.

Put parameters on separate lines

When a function or class constructor declaration has too many parameters, put them on separate lines. How many is too many is subjective and depends on the length of parameter names, the length and complexity of parameter types, the length of the default values, the surrounding code, etc. Too many parameters can also be a good point to ask ourselves if some of them should be extracted into a separate class.

For example, having five constructor parameters might be too much for a single line.

class FruitStoreInTheCloud(val uri: Uri, val user: String, val password: Password, val connectionTimeout: Duration, val retryAttempts: Int = 5)

So we can try putting them on separate lines.

class FruitStoreInTheCloud( val uri: Uri, val user: String, val password: Password, val connectionTimeout: Duration, val retryAttempts: Int = 5 )

Since some of the parameters are related, we might be tempted to group them.

class FruitStoreInTheCloud( val uri: Uri, val user: String, val password: Password, val connectionTimeout: Duration, val retryAttempts: Int = 5 )

The problem with this layout is that it is more irregular than one parameter per line. What we really mean by grouping the parameters is that there is cohesion and each group could be expressed as a separate class.

class FruitStoreInTheCloud( val uri: Uri, val credentials: Credentials, val config: Config )

And with only three parameters in the constructor, we might consider putting them on one line (never mind the irony).

Note that in IntelliJ there is a “Put parameters on separate lines” intention, which can be invoked via the Alt+Enter popup menu or assigned its own shortcut.

Stop the CONSTANT SHOUTING

Constants should be lowercase following the same convention as vals and vars. I realise this tidying contradicts Kotlin coding conventions but there are NO GOOD REASONS for constant names to be uppercase other than history.

As a short summary of the Stop the Constant Shouting article by Jonathan Wakely (which heavily inspired this tidying), in the C programming language it’s common to use macros in cases when we would use a constant in Kotlin. This is because initially constants were not part of the language and even now they still have limitations. Using uppercase for macros made sense because they’re not normal code so it wasn’t a bad idea for macros to STAND_OUT_IN_THE_CODE, especially at the time when smart code editors and IDEs didn’t exist. The coding style for constants (actually macros) was copied from C to C++, to Java, and then to Kotlin.

As you might have noticed, uppercase text REALLY DRAWS OUR ATTENTION. At the same time constants are one of the most boring parts of the code. They don’t change and don’t have any important side-effects (unlike, for example, System.exit(1)). Yet we use the most expressive text style for constants. There is an argument that the uppercase convention is too widespread to ignore. I’m not convinced though that familiarity outweighs the harm done by unnecessary screaming uppercase. The accidental code style for constants needs to be fixed and the sooner the easier it will be.

Instead of SHOUTING CONSTANTS:

companion object { private const val INITIAL_BUFFER_SIZE = 8192 } private val buffer = ByteArray(INITIAL_BUFFER_SIZE)

you can use lowercase:

companion object { private const val initialBufferSize = 8192 } private val buffer = ByteArray(initialBufferSize)

As a side note, in the example above it would be good to inline single usage constant and explain the reasons for choosing number 8192 (no particular reason is still useful information). Often extracting a magic number into a constant doesn’t make the number less magic.

Once constants follow the same naming convention as variables, it’s easier to change constants to variables and the other way round because we don’t need to update all usages.

Consider using tiny types

…

Sane Intellij Plugin Development With Live Plugin

Thu, 30 Dec 2021 00:00:00 +0000

Pretty much all modern web browsers have developer tools console which lets you type some JavaScript code and run it in the browser. In theory, this allows you to do any kind of automation and extend browser functionality at runtime. In practice, it’s not suitable for the purpose, and, to be fair, most users are probably not even aware of the developer console, so for a browser development team spending extra effort to make it possible to extend the browser at runtime via console might not be justified.

This is different for code editors and IDEs with their target audience being power users, so it seems like a sensible idea to make extending editor/IDE functionality and task automation as easy as possible. Surprisingly, this is not the case, and most code editors and IDEs don’t have the functionality to develop extensions at runtime. It doesn’t have to be this way though.

This blog post is about LivePlugin - a plugin for adding/modifying IntelliJ IDEs functionality at runtime without IDE restarts using Kotlin (or Groovy). Even though this blog has details specific to IntelliJ platform and the JVM, some concepts and ideas are universal enough to be applied in other environments.

Why?
What does it look like?
How does it work?
Notifications, logging, basic input
Actions
Action groups
Editor and Document
Intentions and Inspections
IntelliJ APIs Mini Cheat Sheet
Final thoughts

Why?

The main reason is faster and simpler workflow for plugin development. From the IDE user point of view, this means that you can write plugins for things you wouldn’t bother writing plugins otherwise, e.g. project-specific scripts and workflows that can be added to IDE. In a way it’s like creating IDE macros using a programming language. It also makes it easier to add “missing” IDE features and try these features before creating a “proper” plugin.

Minimal setup — no need to create a separate project for plugin development.
Fast feedback loop — plugins are executed without IDE restarts.
Simpler API — smaller surface area API to make extending IDE functionality easier.

What does it look like?

Once you install LivePlugin, you can open any project and notice that there is a panel called “Plugins”. It contains a list of folders with plugins source code. LivePlugin comes with a few example plugins, some of which, including “hello-world”, are installed by default. By convention plugin.kts or plugin.groovy files are plugin entry points. To get started, you can select the “hello-world” plugin and press the green “Run Plugin” button to compile and run it. The IDE should display some notification messages which should also be available in the “Event Log” tool window.

The next thing to try is modifying the plugin code. You can, for example, leave only one show() function and change the message. If you press the Rerun Plugin button, the IDE will display a notification with an updated message. (You can also try writing syntactically invalid code or throw an exception from the code to see how it fails.)

If you’re using IntelliJ IDEA or Android Studio, and have Kotlin plugin installed, all standard IDE features like auto-completion and navigation should work out-of-the-box when editing plugin code. LivePlugin is bundled with its own source code, so you should be able to navigate, for example, to the definition of show() function. Other IntelliJ IDEs, which are not compatible with Kotlin plugin, will only have basic syntax highlighting.

There are few more details, which are described in the bundled examples starting from “hello-world”, but overall this is it. LivePlugin compiles and runs code in the same JVM instance as IDE with access to all IDE classes and objects.

How does it work?

Overall, the whole workflow can be summarized in three steps:

dispose resources from previous plugin execution
compile plugin source code into .class file(s)
load plugin classes into newly created classloader and execute the code

This sounds simple but there are more details at each step. In the first place, why do you need to clean up resources from the previous execution? The reason for the cleanup is that if a plugin registers, for example, an IDE event listener, there is no way for the garbage collector to know when the listener is no longer used. So there is a common pattern in IntelliJ IDEs to pass an instance of com.intellij.openapi.Disposable class which represents a lifetime of the listener (or some other resource). Each time LivePlugin executes a script, it creates a new pluginDisposable object and passes it to the script. The script then will need to use it with IDE APIs or add an explicit cleanup callback (e.g. using .whenDisposed() function). When LivePlugin unloads the script, or before script is executed again, it disposes pluginDisposable from the previous execution.

At the next step LivePlugin compiles plugin source code. This obviously requires a compiler, so the question is where do you get one? IntelliJ IDEs used to come with an embedded Kotlin compiler jar, but it’s not the case anymore, and it was hard to know when Kotlin compiler would be updated (with potentially breaking changes), so LivePlugin is bundled with its own Kotlin compiler. Another question is how to compile code which uses IDE APIs. Luckily, since IDE is running on the JVM, it’s possible to look up the location of IDE jar files and use them for compilation classpath. This is limiting in a sense that you cannot write code for an older or newer version of IDE, but most of the time it’s not a problem and compiling again current version of IDE just makes setup much easier. LivePlugin can also compile code which uses external libraries or other plugins.

It might seem a bit strange why LivePlugin creates a new classloader for each execution of a script. The answer is that a new classloader is needed to make sure that classes from the previous script executions can be unloaded. Classloaders are special classes which can bring other classes into existence, e.g. by reading bytecode from disk, network or generating bytecode. Classloaders form a hierarchy in which child classloaders can see classes from their parents, but not the other way round. Classloaders can only load classes with unique names, so plugin classloader won’t be able to load another java.lang.String (which is already loaded by bootstrap classloader). On the other hand, sibling classloaders don’t “see” each other’s classes, so two leaf classloaders can both load my.project.Foo class at the same time. From that point of view, classloaders are like namespaces. When there are no more references to the classloader and the objects created from its classes, the classloader and the classes become eligible for garbage collection. This is the main reason for creating a new classloader for each script execution.

The final step is running plugin code. At the JVM level, plugin script is really just a class which can be reflectively instantiated with some parameters (including pluginDisposable). The construction basically runs all the code in plugin.kts (or plugin.groovy) file. This happens on the event dispatch thread (EDT) so it is safe to modify IDE state at the top level of the plugin script (see IntelliJ threading rules). On the other hand, this means that blocking operations (like CPU intense calculations or long-running IO) should be done on a background thread, to avoid freezing IDE UI. Just like any other plugin it has access to all objects in the IDE JVM, so it is possible to use any functionality from other plugins and IDE itself.

Here is a diagram with an overview of the components described above, where filled lines represent control flow and dotted lines represent dependencies. It’s not a very detailed or precise one, but hopefully it is still useful as a visual guide:

This is all there is to it. If you want to extend IDE functionality by, for example, adding some actions, it’s up to you to invoke specific functions, i.e. there is no special syntax or some XML/JSON/YAML file with configuration. It’s all just Kotlin code.

Notifications, logging, basic input

Being able to show notifications or print messages is one of the most fundamental ways to get feedback when using LivePlugin. There is no way to debug plugins with breakpoints because there is only one instance of the JVM. In theory, you could test-drive all plugin functionality, but in practice, many APIs are not very TDD-friendly. So notifications are the most useful way to know why something is (not) working.

To show a standard IDE notification you can use the show() function (the name is inspired by the Show typeclass in Haskell and the fact that “notify” would clash with method on java.lang.Object). The function takes few optional parameters and in its most complex form can look like this:

show( message = "Hello <a href=''>world</a>", title = "Foo", groupDisplayId = "MyDisplayGroup", notificationListener = { notification, hyperlinkEvent -> show("Hi ${notification.content} ${hyperlinkEvent.sourceElement}") } )

Where message is an object which will be converted to a string; title, as you can guess, is a title of the message; groupDisplayId is a string id which can be configured in “Event Log” settings so that for example IDE plays a sound on events with a particular group id; notificationListener is callback invoked when user clicks on hyperlinks in the HTML formatted message. (Note that the support for basic HTML formatting and hyperlink callbacks are really just part of IntelliJ notification API which is pretty cool 😎)

If the notification message is too long, IDE will not show the whole content of the message. In this case, you can use the showInConsole() function which creates and opens a tool window with the specified text. It’s an extension function on the Project object which you can get as a part of the script context or via standard IDE API, e.g. from AnActionEvent in an action callback:

project?.showInConsole( message = "a really long message, or maybe even a stacktrace", consoleTitle = "Some title", contentType = ERROR_OUTPUT )

Using notifications and tool windows for what is essentially logging is ok on a small scale to get fast feedback, but not ideal for a larger volume of data. There is a logging API based on com.intellij.openapi.diagnostic.Logger which works pretty much as expected appending info/warn/error messages to the IDE log file. And if you want something really simple, you can always use println() which ends up in the IDE log anyway:

val logger = Logger.getInstance("MyPlugin") logger.info("info message") logger.warn("warn message") logger.error("error message") // Will log and show error notification in IDE. println("info message")

Finally, there are couple Kotlin functions in the com.intellij.openapi.ui package and few static functions in the com.intellij.openapi.ui.Messages Java class for basic UI dialogs:

val isYes = com.intellij.openapi.ui.showYesNoDialog( "Dialog Title", "And this is a question?", project ) show(if (isYes) "Yes!" else "No 😿") val userInput = com.intellij.openapi.ui.Messages.showInputDialog( project, "Please enter something useful", "Dialog Title", AllIcons.Ide.Gift, "initial value", null ) show("userInput = $userInput")

Actions

Actions are one of the most fundamental ways of user interaction with IDE. All menu items, text editing activities (e.g. moving cursor) and all items in Settings -> Keymap are actions. Actions are essentially stateless functions which can be presented in UI and produce side effects when invoked. Once registered, action becomes available in all projects and will be invoked when you click on a menu item, use corresponding shortcut or, for example, choose an action in the “Find Action…” action. Actions are executed on the UI thread, so long-running tasks must be done in the background to avoid freezing UI.

All actions need to implement com.intellij.openapi.actionSystem.AnAction abstract class which is not that difficult, but to make it a bit easier, LivePlugin has registerAction() function with lambda as a callback. In the simplest form it looks like this (the new action will appear in the “Find Action…” list, where it can be invoked):

registerAction("Show Project Name") { show(it.project?.name) }

Since there is a registerAction(), it would be reasonable to expect the unregisterAction() function. There isn’t one though. The reason is that unregistering actions and other callbacks is essentially resource cleanup and done by passing an instance of com.intellij.openapi.Disposable. In the example above, disposable is passed as part of the LivePluginScript object which is an extension function receiver of registerAction(). In a more explicit form, each plugin execution has its own instance of pluginDisposable which can be passed to various APIs, e.g. this is a more verbose way of registering action:

registerAction("Show Project Name", disposable = pluginDisposable) { show(it.project?.name) }

When a plugin is rerun or unloaded (via the “Unload Plugin” button in the “Plugins” tool window), pluginDisposable is disposed unregistering actions and freeing other resources. You can test it by unloading the plugin and checking that the action disappeared from the “Find Action…” list.

There are few more useful details about actions which can be illustrated by expanding on the example above:

registerAction( id = "Show Project Name", keyStroke = "alt shift .", actionGroupId = "ToolsMenu" ) { actionEvent: AnActionEvent -> show(actionEvent.project?.name) }

Here id is the action id which must be unique within the IDE. For convenience, it’s also used by registerAction() as a textual representation of the action.
Keyboard keyStroke to invoke the action, where modification keys are lowercase (e.g. “ctrl”, “alt”, “shift”, “cmd”), letters are uppercase, and other keys are uppercase based on the constant names in java.awt.event.KeyEvent (e.g. “ENTER”, “ESCAPE”, “SPACE”, “LEFT”, “UP”, “F1”, “F12”).
actionGroupId which is a string identifying a menu/popup/toolbar to which action will be added. Unfortunately, there is no single place to find all these ids, but there are common menu ids listed in liveplugin.ActionGroupIds object.
Instance of AnActionEvent with invocation “context” which passed into the lambda. This is usually how actions know about the project, editor or UI component in which they were invoked.

The registerAction() function is part of LivePlugin and uses lambda just for convenience. It’s entirely legal to implement AnAction abstract class from IntelliJ API. Creating a subclass of AnAction is a bit more effort, but it also has more configurability. For example, you can specify an action icon or configure if the action is enabled/disabled in the current context.

registerAction(id = "Show Project Name", action = ShowProjectName()) class ShowProjectName : AnAction(AllIcons.Ide.Gift) { override fun actionPerformed(event: AnActionEvent) { show(event.project?.name) } override fun update(event: AnActionEvent) { event.presentation.isEnabled = true } }

Even though LivePlugin comes with some helper functions, you don’t have to use any of them. Instead of registerAction(), you could use IntelliJ API to register actions. The problem is that in this particular case, API is used by IDE to add actions from XML config files, and unlike newer APIs doesn’t support Disposables.

val actionManager = com.intellij.openapi.actionSystem.ActionManager.getInstance() actionManager.registerAction("action id", action)

Finally, since actions are just like functions, it’s natural to ask if actions can invoke other actions. The answer is “yes” with the caveat that you need to look up or create AnAction and AnActionEvent objects. Some actions have simple constructors, so it might be easy to just create a new instance. Some might not be public or hard to construct. In this case, they can be looked up via ActionManager.getAction(actionId). The easiest way to get an instance of AnActionEvent is to pass on event argument from already invoked action (or if you really need to construct one, see liveplugin.implementation.Actions#anActionEvent() for ideas):

registerAction("Invoke Another Action") { event -> val action = ActionManager.getInstance().getAction("ToggleBookmark1") action.actionPerformed(event) // Bookmark current line }

Action groups

Actions can be grouped into action groups which are actions themselves. The main reason for creating an action group is that it’s a good way to present a bunch of actions in the UI as a submenu or a popup window.

All action groups implement the com.intellij.openapi.actionSystem.ActionGroup class. The simplest way to create one is by instantiating DefaultActionGroup, however, when used as a menu item it will add all actions without sub-menu, unless the isPopup property is set to true. For this reason there is liveplugin.PopupActionGroup() which makes the group a popup.

val action1 = registerAction("Action 1") { show("1") } val action2 = registerAction("Action 2") { show("2") } registerAction( id = "Some Actions", actionGroupId = ActionGroupIds.EditorPopupMenu, action = DefaultActionGroup(action1, action2).also { it.isPopup = true } )

Another option is to display an action group as a popup window. LivePlugin comes with createPopup() extension function which has few useful default parameters and uses JBPopupFactory under the hood. Note that there is a special Separator action in IntelliJ API which represents a separator in the menu.

registerAction("Show Action Group") { PopupActionGroup(name = "Some Actions", action1, action2, Separator.getInstance(), PopupActionGroup("Sub Menu", action3, action4, ) ).createPopup() .showCenteredInCurrentWindow(it.project!!) }

All of the above is straightforward but there are few subtle things worth mentioning. One is that actions should not share AnActionEvents between each other. For example, in the code snippet below, “action 1” is using event from the outer “Show Action Group” action. This is wrong. The reason is that events capture the content in which an action was invoked, and this context might change by the time another action is invoked. Even if you’re sure that sharing an event will be fine, in some cases IDE checks if there was a shared event and will display an error. Another caveat is that when creating a popup, we do want to pass some of the context to actions in the popup window via event.dataContext, otherwise, in the example below “action 2” will not know what the current file is.

registerAction("Show Action Group") { event: AnActionEvent -> PopupActionGroup(name = "Some Actions", AnAction("action 1") { show(event.virtualFile?.path) }, AnAction("action 2") { show(it.virtualFile?.path) } ).createPopup(event.dataContext) .showCenteredInCurrentWindow(event.project!!) }

Actions and action groups are the lowest hanging fruit in terms of the plugin development. Even with this minimal IDE integration there are few useful things that can be done, e.g. by sharing bookmarks and doing various automation via shell scripts and HTTP requests. I will not explain the examples below in detail because they are roughly based on the examples bundled with LivePlugin. It’s worth noting though that it’s possible to use third-party libraries (e.g. http4k) and execute long-running tasks in the background without blocking UI thread.

// add-to-classpath $PLUGIN_PATH/libs/* PopupActionGroup(name = "Some Actions", AnAction("LivePlugin on GitHub") { openInBrowser("https://github.com/dkandalov/live-plugin") }, AnAction("LivePlugin on GitHub 2") { it.project?.openInIdeBrowser("https://github.com/dkandalov/live-plugin") }, AnAction("Open plugin.kts") { it.project?.openInEditor("$pluginPath/plugin.kts") }, AnAction("Http Request") { val client = org.http4k.client.OkHttp() val response = client(Request(GET, "https://duckduckgo.com")) show(response.status) }, AnAction("Run a Script") { runBackgroundTask(taskTitle = "Running shell script", task = { show(runShellScript(""" '$pluginPath/hello.sh' sleep 5 """.trimIndent() ).stdout) }) }, )

Editor and Document

Writing software is still mostly based on textual representation of code and even though IDEs are good at manipulating syntax tree, being able to programmatically control text editor can be quite useful. This can be achieved using com.intellij.openapi.editor.Editor and Document APIs.

If you feel like experimenting, one of the simplest ways might be to use the current editor without any actions directly in the plugin script. The code below uses caretModel to move cursor to the beginning of the file and selectionModel to select all text:

val editor = project?.currentEditor ?: error("No editor") editor.caretModel.moveToOffset(0) editor.selectionModel.setSelection(0, editor.document.textLength)

However, as a motivating example, let’s write “Random Case” action which will change letters of the current word to be randomly upper or lower case. I would normally write the code incrementally, manually testing each new piece of functionality. The first thing to check in the action is that document and editor are available. The following action will show a message when invoked in an editor but will not show notification if the focus is, for example, in the “Project” tool window.

registerAction("Random Case", "alt shift .") { event -> val document = event.document val editor = event.editor if (document != null && editor != null) { show("Document and editor are available") } }

The next step is to get the word under the caret. The trick here is to use offset from editor.caretModel which represents a shift in characters from the beginning of the current file. There is always a potential for off-by-one errors on the edges around words and beginning/end of file but the code below seems to be good enough for the job.

Randomising letters is an easy task in Kotlin (I have to mention though that Random object has side effects and should be ideally passed in from outside so that it’s possible to give it a seed and write an automated test for the action):

val text = (textBefore + textAfter) .map { if (Random.nextBoolean()) it.toUpperCase() else it.toLowerCase() } .joinToString(separator = "") show(text)

The final step is to update text in the document. It is a trivial thing to do using the Document.replaceString() function which takes from/to offsets and a replacement string. However, if you run the code below, it doesn’t modify the document but instead fails with “Assertion failed: Write access is allowed inside write-action only (see com.intellij.openapi.application.Application.runWriteAction())”.

document.replaceString( offset - textBefore.length, offset + textAfter.length, text )

This happens because modifying the state of a document without a write lock violates IntelliJ Threading Rules. I recommend reading the rules to understand the details, but overall the rules can be summarized by the following table, where ReadAction and WriteAction are classes with static run(ThrowableRunnable) method and in spite of the name are not related to AnAction class described in the section above (yes, naming is hard 🙈️).

	Read	Write
UI thread	✅	WriteAction
Other threads	ReadAction	❌

Since “Random Case” action code is executed on the UI thread, the document modification code can be wrapped with WriteAction:

WriteAction.run(ThrowableRunnable { document.replaceString( offset - textBefore.length, offset + textAfter.length, text ) })

Unfortunately, it still fails but this time with a different message “IncorrectOperationException: Must not change document outside command or undo-transparent action. See com.intellij.openapi.command.WriteCommandAction or CommandProcessor”. The reason is that there is an additional rule that document modification can only happen within an undoable command, meaning that the modification can be undone/redone and will be available in the “Main Menu -> Edit -> Undo/Redo”. To make document modification code a bit less nested, LivePlugin comes with an extension function executeCommand, so the final code for the “Random Case” action looks like this:

registerAction("Random Case", "alt shift .") { event -> val document = event.document val offset = event.editor?.caretModel?.offset if (document != null && offset != null) { val textAfter = document.text.drop(offset).takeWhile { it.isLetterOrDigit() } val textBefore = document.text.take(offset).reversed().takeWhile { it.isLetterOrDigit() }.reversed() val text = (textBefore + textAfter) .map { if (Random.nextBoolean()) it.toUpperCase() else it.toLowerCase() } .joinToString("") document.executeCommand(event.project!!, description = "Random Case") { replaceString( offset - textBefore.length, offset + textAfter.length, text ) } } }

Intentions and Inspections

There are two similar concepts in IntelliJ based IDEs: intentions and inspections. Both intentions and inspections run in the background analysing code in the current editor. If a particular intention/inspection can be applied, it shows up in the light bulb popup menu available on alt+enter, where you can select it to perform a code transformation. The main conceptual difference is that inspections provide a way to transform code (e.g. invert if condition or replace && with ||) and inspections are looking for problems in the code (e.g. unreachable code or potential NullPointerException).

Both intentions and inspections work with PSI elements (short for Program Structure Interface) which are basically nodes in an abstract syntax tree. PSI elements are represented by different classes in different programming languages, so usually intentions/inspections are written for a specific language. There is UAST (Unified Abstract Syntax Tree) API for JVM languages (Java, Kotlin, Scala, Groovy) but it’s still somewhat experimental at the moment of writing.

The best way to learn about PSI is to see what it looks like for particular code examples. This is what println("OMG!!") looks like using PsiViewer plugin:

There are only two things you need to know to write intentions/inspections in LivePlugin. One is registerIntention() and registerInspection() functions which will register intention/inspection on execution and unregister when pluginDisposable is disposed. And another is // depends-on-plugin plugin-id instruction which can be added after imports and tells LivePlugin to add dependency on a plugin which is not part of core IDE API. The rest is just standard IntelliJ API for working with PSI.

The example below shows a simplistic Kotlin inspection which replaces “OMG” with “🙀” inside string literals. Its isAvailable() function will be called on the current PSI element in the editor and if it returns true, the intention will be displayed in the light bulb popup menu. When invoked, the intention uses KtPsiFactory to construct a string literal PSI node from newText and replaces the current element with it. Note that unlike the “Random Case” action example above, PSI transformation doesn’t need WriteAction or undoable command because it’s already done in its superclass PsiElementBaseIntentionAction.

// depends-on-plugin org.jetbrains.kotlin registerIntention(object : PsiElementBaseIntentionAction() { override fun isAvailable(project: Project, editor: Editor?, element: PsiElement) = element.parentOfType<KtStringTemplateExpression>() != null && element.text.contains("OMG") override fun invoke(project: Project, editor: Editor?, element: PsiElement) { val newText = element.text.replace("OMG", "🙀") val newElement = KtPsiFactory(element).createLiteralStringTemplateEntry(newText) element.replace(newElement) } override fun startInWriteAction() = true override fun getText() = "Replace with 🙀" override fun getFamilyName() = "Hello" })

Similarly, we can define an inspection which finds “!!” in string literals and replaces it with “💥”. Inspections have a different API which instead of returning a boolean, registers a “problem” with a suggested quick fix, but conceptually it’s quite similar to intention.

// depends-on-plugin org.jetbrains.kotlin registerInspection(object : AbstractKotlinInspection() { override fun buildVisitor(holder: ProblemsHolder, isOnTheFly: Boolean): PsiElementVisitor { return expressionVisitor { expression: KtExpression -> if (expression is KtStringTemplateExpression && expression.text.contains("!!")) { holder.registerProblem(expression, "Found !!", MyQuickFix()) } } } override fun getShortName() = "Usage of !!" override fun getDisplayName() = "Usage of !!" override fun getGroupDisplayName() = "Hello" override fun isEnabledByDefault() = true }) inner class MyQuickFix: LocalQuickFix { override fun applyFix(project: Project, descriptor: ProblemDescriptor) { val element = descriptor.psiElement val newText = element.text.replace("!!", "💥") val newElement = KtPsiFactory(element).createExpression(newText) element.replace(newElement) } override fun getName() = "Replace with 💥" override fun getFamilyName() = "Hello" }

IntelliJ APIs mini cheat sheet

There is an obvious and a very reasonable question of how you can find which functions or classes to use. Or, in other words, given a blank screen, how do you figure out what to type? Just like with any other large code base there is no simple answer.

LivePlugin wraps some of IDE APIs, so you can check what is available in the liveplugin package and liveplugin.PluginUtil class (this is a Groovy class, but it can be used from Kotlin). The simplest way is to type liveplugin. and use auto-complete.

You can explore IntelliJ platform source code on GitHub, Upsourse or clone and explore it locally: git clone https://github.com/JetBrains/intellij-community.git. One useful code exploration strategy is to find existing IDE functionality which does almost what you want to do or is likely to use the API you’re interested in. You can then find source code and see how it actually works. If some functionality is presented in the user interface, you can search for the text you see on the screen as a way to find an entry point into the code.

Finally, it is useful to read IntelliJ Platform SDK docs and be familiar with common IDE APIs. Below I listed APIs which I found useful in the form of a mini cheat sheet, but it is by no means exhaustive or complete.

“Core” classes:

Application - core application-wide functionality and methods for working with threads
MessageBus - subscribe/publish messaging infrastructure
Project - represents project in IDE, used in many APIs for project-specific functionality
ProjectManager - open/close/get list of projects (see also ProjectManagerListener)
DumbAware - marker interface for actions and tool windows that can work while indices are being updated

Actions:

AnAction - all user interactions are performed through actions (see also EditorAction)
ActionManager - register/unregister/find actions
IntentionAction - interface for intention actions
IntentionManager - register/unregister intentions

Editing files:

Editor - represents an instance of text editor (see also FileEditorManager)
CaretModel - caret(s) position in editor
SelectionModel - text selection(s) in editor
MarkupModel - custom text range highlighting, markers on the gutter, etc
FoldingModel - custom text range folding
Document - represents the contents of a text file loaded into memory and possibly opened in a text editor (see also FileDocumentManager and IntelliJ Platform SDK docs)
VirtualFile - represent a file on disk, in archive, HTTP server, etc. (see also VirtualFileSystem, VirtualFileListener, VirtualFileManager and IntelliJ Platform SDK docs)

Syntax tree:

PsiElement - common base interface for all PSI elements (see also IntelliJ Platform SDK docs)
PsiFile - PSI element representing a file
PsiManager - gets PsiFile for VirtualFile
“PsiUtil” classes - misc methods for PSI manipulation

UI components:

Messages - various yes/no/ok/cancel and basic input dialogs
DialogBuilder - builder for custom dialogs
ToolWindowManager - get registered tool windows, balloon notification for tool windows
ToolWindowFactory - creates tool windows
See also User Interface Components IntelliJ Platform SDK page

Other interesting APIs:

com.intellij.codeInsight.completion.CompletionContributor
com.intellij.lang.LanguageExtension
com.intellij.patterns.PsiElementPattern (and com.intellij.patterns.PlatformPatterns, com.intellij.patterns.StandardPatterns)
com.intellij.psi.util.PsiUtilCore

Final thoughts

If writing plugins/extensions at runtime is such a good idea, why is it not more widespread? This is the question I ask myself and one of the answers I have is that this might be a chicken and egg problem. There is a stereotype that creating an editor or IDE plugin/extension is really hard and advanced, so very few people actually try it. Editor/IDE developers correctly conclude that there isn’t a lot of demand, so they don’t see the need to make plugin/extension development easier. The development experience remains difficult, and reinforces the stereotype of plugin development being hard, so nothing really happens.

There are some inherently difficult problems related to plugin development at runtime which I didn’t mention above. One is the state migration between plugin reloads. In dynamic languages like Groovy the new “version” of plugin can use objects from the previous “version” as long as they have the same shape (old objects will prevent old classloader and classes from being garbage collected, which is a “small” memory leak, but at least it will work). In strongly typed languages like Kotlin, it’s even harder because classes from different classloaders are incompatible even if their API and implementation are identical. So the solution is to use serialisation/deserialisation which is a problem of its own.

Another difficulty is creating a good API for extending IDE functionality. Ideally, it should cover the areas which are most likely to be extended by users. It should be discoverable, reusable and designed for reloadability. On top of that, just like with any other published API used by many users, once it’s public, it might be difficult to evolve. This is a really hard problem, and I don’t claim that LivePlugin solves it.

There are a lot of benefits though. One is “outsourcing” IDE features to users, so instead of creating feature requests and waiting for them to be completed (if ever), users could add the functionality themselves. Some user interfaces naturally evolve into a complex conditional configuration anyway. For example, Run Configuration has options to run other tasks before/after the current task and many other options which could be expressed via some API instead of a complex user interface. Another benefit is customisation specific to a particular project or library. The simplest automation is to run project-specific scripts in the context of the currently open file or selected text. More advanced uses might include navigation or code completion specific to a library/framework. This is in the territory of traditional plugins and the point here is that if plugin development is easy enough, then it might reach a point when it’s a cultural norm to create a plugin with library, framework or project support. To be fair, it might be too much to ask for example, for a Python project, but it seems more reachable for Kotlin. There is one major platform targeting Kotlin development, so having a standard practice of creating tailored plugins is achievable within the Kotlin community. To be precise, there is IntelliJ IDEA and there is now Fleet editor by Jetbrains which supports Kotlin. But Fleet backend is still based on IntelliJ and is likely to be compatible with existing plugins. And the Fleet frontend might be a good chance to explore the minimal most useful user-facing API for both Fleet and IntelliJ. Very exciting times for the ultimate power of custom plugins 🎉

Limited WIP — test && commit || revert

Tue, 04 May 2021 00:00:00 +0000

Limited work-in-progress doesn’t seem to be a very well-known idea in the software development world but there are few practices which we use without realising that they’re limited work-in-progress (WIP) techniques. One example is pomodoro which suggests to focus on a task for 25 minutes and then take a break (without checking email or social networks), i.e. pomodoro limits the amount of tasks that can be done in parallel. A less obvious example is TDD which makes us focus on the problem taking one step at a time, i.e. TDD limits what we can do at each stage of the red-green-refactor cycle (for example, we can’t add production code when tests are green). Another not-so-obvious example is continuous integration (CI) which is about regularly combining all changes into a working piece of software, i.e. CI is about limiting merge debt.

Unfortunately, there is no common practice to quantify merge debt and doing CI can be a bit more subtle than most CI tools want us to think. In particular, it might seem ok to stay on a branch as long as it can be merged into trunk (aka main/master branch). The problem is that there is no guarantee that pending changes on the branch can be integrated with the pending changes on all other active branches. You can optionally make the problem more interesting by adding code reviews via pull-requests and multiple “main” branches (e.g. dev/uat/prod).

Without suggesting that it’s the best, easy or even practical solution, one extreme way of doing CI might be developing on trunk using “test && commit || revert” (TCR). TCR is a workflow inspired by TDD and intended to encourage small changes which are safe to integrate.

The idea

TCR can be summarised with the following three rules:

You can’t commit without running tests.
If the tests pass, all changes are automatically committed.
If the tests fail, all changes are automatically reverted.

There is an analogy with some arcade computer games where if you reach a safe point in the game, then the next time the character dies, it respawns at the last safe point. Similarly, with TCR if the code is broken (i.e. has any failing tests), you’re taken to the last safe point when all the tests were green. Unlike arcade games it’s up to you to decide where the safe points are and there are no lives or score in TCR (it might be an interesting idea to explore though).

TCR might seem hard because of the constant pressure to get the code and the tests right on the first attempt. If anything goes wrong (even a typo in assertion), all uncommitted work is deleted. The trick is to find ways of working in tiny increments so that at each step it’s nearly impossible to fail and if you do fail, there isn’t much to lose anyway (this is quite similar to auto-revert constraint).

The most continuous integration

Because TCR makes us work in small increments, merging code becomes much easier. Small frequent changes are less likely to have conflicts and even when a conflict occurs, it’s easier to resolve it or to just drop local changes and do them again. Since TCR only allows to commit on passing tests, all commits should be safe to share with other people. Obviously, frequent pushes will also ensure that you receive updates from everyone else, so it might be somewhat like using a collaborative code editor except that it’s done via version control. From this point of view, “test && push || revert” (TPR) could be a better name to promote continuous integration at the smallest possible level.

Given that the commits are done automatically, there is a question of when/how to specify commit messages. One straightforward solution might be to edit the commit message just before it’s done automatically. Or you can try specifying a commit message in advance before making changes, e.g. in some environment variable. Although these options are likely to get in the way of TCR flow and with lots of small changes you might discover that writing a commit message after each modification is too tedious. So another approach is to use a “default” commit message for a series of commits and squash them later. Or use a placeholder message and later squash commits into more meaningful chunks with better commit messages.

It’s also interesting to challenge the whole idea of writing a commit message for each code modification. Having an explanation of why code has changed is good, but should it really be part of the commit? Maybe there is more value in decoupling incremental changes from the task description and decisions made on a project which often span many commits anyway. And as a nice bonus we’ll get a detailed history of code modifications.

TCR with TDD?

Following the basic TCR rules, there is no way to have a failing test without code being reverted. This means that you can’t really use TCR with TDD but there are few workarounds. One approach is to bend TCR rules a bit and only revert production code. Another approach is to write negated assertion(s) before implementation and fix it once the implementation is done. Or you can try negating/modifying assertions after a successful commit to confirm that they fail (somewhat like a manual mutation testing).

TCR is perfect for refactoring though. It encourages the refactoring strategy to run tests after each change and go back if anything went wrong without wasting time trying to figure out why refactoring didn’t work. After a failed attempt you can do a simpler, smaller or just different refactoring. Trying to analyse failure from a stacktrace or debugging code with TCR is problematic anyway, because the stacktrace will be pointing to the code which doesn’t exist anymore, and you can’t rerun a failed test with a debugger (although I suppose you could run tests in debugger in the first place).

There is a question of which tests should be run when using TCR. The ideal answer is all tests. In practice, running all tests might be too slow, so a subset of tests which are the most relevant for modification is a pragmatic choice. Unfortunately, on many code bases even running a single test might be too slow because of build/compilation time. It’s not a great experience to spend 2 minutes writing a bit of code and then wait for a minute to see it all reverted. There are two possible conclusions. One is that TCR is not usable on some projects and with certain toolsets. Another is that some projects and toolsets are not very usable themselves. The second conclusion is particularly interesting because it’s hard to say that many mainstream toolsets are really focusing on performance. This is especially sad because as software developers we created all these tools ourselves (editors, compilers and testing frameworks). What stops us from focusing on a really good tool performance in the range of milliseconds so that it’s instant from a human perception point of view? Maybe a hanging code editor or one minute compilation time should be socially unacceptable similar to a major security breach or personal data leak. Maybe testing frameworks should be more opinionated and fail unit tests which run for longer than a second.

Try it yourself!

The easiest way to try TCR is by setting up a toy project or a code kata and manually following the rules. Writing a small script (or finding one on the Internet) might be a bit more effort but is worth it to make TCR more interactive and a bit like playing a mini-arcade game. I ended up adding TCR-mode to the Limited WIP plugin for IntelliJ IDEs which already had similar functionality. Either way don’t take it too seriously, experiment with the rules and make sure to have fun.

Limited WIP — Auto-revert

Wed, 28 Apr 2021 00:00:00 +0000

(Updated 17th October 2024)

As software developers, we seem to be mostly interested in technology and tools rather than the human side of the process such as software development workflows. For example, consider TDD — one of the few workflows that made it into the mainstream. These days pretty much every project has some kind of unit testing library imported and everyone has TDD on their CV. However, in practice even when people do write tests, they often do it after writing production code (which contradicts the definition of TDD) and instead of the red-green-refactor cycle do something like red-code-debug-refactor-code-green.

An easy way to fail at red-green-refactor is by making too many changes. We focus on the solution, start exploring how it can be done, find a couple of problems in the code that need fixing, and end up writing/changing way more code than is necessary to pass existing tests. When we finally add more tests for the implemented functionality, the new code might not work as expected, so we start debugging it. It might also feel convenient and efficient to do the refactoring while we change everything anyway, so it’s often done with failing tests. These are common enough patterns, that Adrian Bolboaca came up with the baby steps constraint for code retreats which essentially adds a time limit to red-green-refactor steps, so you can’t go off on a tangent and have to concentrate on tiny steps or your code is reverted.

The idea of time-limited work-in-progress (WIP) is not specific to TDD. Similar to other limited WIP techniques, the benefits are about doing more focused work with fewer mistakes while constantly keeping the project in a working state. In this blog post, I’ll describe a generic version of the baby steps constraint which I call “auto-revert”.

What is auto-revert?

As the name suggests, auto-revert is about automatically reverting uncommitted changes after a time interval:

Start a countdown timer, e.g. 5 minutes.
On each commit, reset the timer.
On a timeout, revert all changes.
You can only commit when all tests pass.

If you never heard about auto-revert, it probably sounds like an absolute waste of time. It is ridiculous to give up control of when the code is reverted and use a timer instead. Imposing time pressure on yourself to finish the task (as if doing work isn’t hard enough) to achieve what, more stress? With a short timeout duration, it will be impossible to get anything done.

All of the above is valid, but… The trick is to learn how to change your ways of working to make small incremental code modifications (aka baby steps) so that the code is never too far away from being in a working state that can be committed. Once you get into this flow, auto-revert fades into the background, and reverts become rare. Even when some changes are reverted, it’s not a big deal anymore because it’s only a few minutes of work. You can also think about redoing reverted changes as a micro-codekata. The code rewrite after a revert might not follow the same steps or end up in the same place as before. And that’s ok. The process of making the change again often gives a good insight into a better way of doing it.

Another positive effect of auto-revert is time awareness. We often underestimate even the most basic tasks (perhaps, because we think about the size of the change in terms of text editing and downplay the conceptual complexity and the probability of us not fully understanding the problem). Using an explicit timeout makes the duration of tasks very visible. It also helps to avoid distractions because you know you only need to concentrate only for a limited time. This is similar to pomodoro technique which suggests working in 25-minute blocks with regular breaks.

Thought experiments

To explore auto-revert, let’s perform a couple of thought experiments:

Auto-revert with a 10-year timeout is unlikely to be noticed by anyone. Uncommitted code from years ago will be long forgotten and will be out-of-date anyway. Considering the probability of hardware failures, we are already working with this constraint.
Similarly, auto-revert with a timeout of months, weeks, or several days will make no difference for most people because changes are done and committed within shorter time intervals. From this point of view, there is nothing radical about auto-revert.
It becomes more interesting with a timeout measured in hours and minutes. This is the range where the learning opportunities are and auto-revert will be uncomfortable until you change your ways of working.
Finally, if the auto-revert timeout is just a couple of seconds (or less!), making any change becomes impossible. One interesting solution could be to come up with code transformation tools that can do safe code modifications and immediately commit them. There is a parallel here with using the smallest maximum change size threshold.

What about real-world projects?

Using auto-revert on real-world projects comes with its own challenges. The main issue I’ve seen is complex code which takes a long time to explore and understand, let alone change it within a few minutes. An obvious solution might be to not make any changes during the exploration phase (except for small improvements which can be done and committed within timeout). If it looks like the complexity makes it impossible to implement a feature in a series of small steps, it might be worth asking “Why”, “How did we get here”, “Can the code be refactored to make it possible”, etc.

Another problem is slow build and tests. To be usable, the auto-revert timeout needs to be noticeably longer than the duration of an average feedback loop. For example, if you modify code for 2 minutes, compile/run all relevant tests for 2 minutes, then it’s not practical to revert changes every 5 minutes because if the tests fail, there will be only 1 minute left to fix them. Long feedback loops require longer auto-revert timeout which requires a larger change size. And nobody wants to lose large code changes. What’s worse is that if the feedback loop is too long, we naturally get distracted and forget about the auto-revert timeout (until all the changes are gone). This can be interpreted as auto-revert doesn’t work on projects with long build times. Or you can use this as an opportunity to ask yourself “why”. In particular, if running tests takes too long, was it a conscious decision based on some trade-offs, or did the tests happen to slow down over time because nobody cared enough? Slow compilers, build tools, and editors/IDEs also get in the way of fast feedback. It’s not very common to continuously measure the speed of a compiler or a build tool on the project — it feels that it’s outside our circle of influence. But maybe we should! Looking at modern programming environments, many of them don’t provide particularly fast feedback, so unless we collectively make a big deal out of toolchain performance, it’s unlikely to improve.

Finally, when using auto-revert, you might start committing often. This brings up the question about the meaning of commits and commit messages. If there are changes every few minutes, commits can become too fine-grained to write a separate message for each of them. One option is to squash tiny commits into more chunky ones with meaningful commit messages. Another idea is to decouple integration of code changes with other people from explaining the changes and documenting project history (so commits on the trunk / main branch can be essentially a “Save” button for distributed work).

Try it yourself!

The simplest way to try auto-revert is to use a timer (on the command line or on your phone). Be aware that if you’re not pair-programming, it’s easy to negotiate with yourself and skip the revert “just this one time”. A bit more advanced approach might be to write a script that will perform a git reset every few minutes and restart the timer if there are no local changes. Going down this path I ended up creating Limited WIP plugin for IntelliJ IDEs.

Auto-revert is an essential constraint to try with a codekata to challenge your current ways of working and to observe your reaction to code being reverted. It can be more tricky on a large-scale project, but it’s worth trying to find the bottlenecks in your working environment. Similar to other workflows, nothing is set in stone with auto-revert. Feel free to experiment with it and have fun.

Limited WIP — Maximum change size

Thu, 22 Apr 2021 00:00:00 +0000

There is a lot of focus on the technical side of software development: programming languages, libraries/frameworks, architecture, etc. But not that much focus on what humans are supposed to do while using the tools. There seems to be an implicit assumption that developers will do things in the most optimal way. Unfortunately, this assumption is often false. In the same way that people don’t always follow the lifestyle which is best for them, we don’t always take the best path when creating software.

To give a specific example, let’s say while working on a feature, you spot some code which can be improved. So you go ahead with the improvement and go back to the feature. This happens a few more times, and before you know it there are twenty modified files with changes including both code improvements and the feature itself. The feature is not complete yet but the change feels big enough that it needs to be committed. The bad news is that it’s now hard to disentangle code improvements from the feature, and you have to choose between (1) figuring out what to include into a partial commit, (2) committing mixed changes with an incomplete feature or (3) continuing with an already oversize change. Not ideal. It would be good to not end up in situations like this in the first place.

This is where techniques to limit work-in-progress (WIP) come to the rescue by stopping us from doing too many things in parallel. The benefits of limiting work-in-progress are not just about better commits (which is often a symptom), they are about more focused work, fewer mistakes and improved workflow. In this blog I’ll describe the technique of setting the maximum change size.

The idea

Using maximum change size to limit work-in-progress is quite simple — if the size of changes is bigger than some threshold, you’re not allowed to commit. Instead, you have to find a way to split the work into smaller committable chunks. It also means that as soon as you reach the threshold, there is no point making more changes because you won’t be able to commit them anyway. Reaching the threshold should be a rare event though. You should keep yourself aware of the current change size, for example, by monitoring diff size in version control or IDE, and dealing with it before it gets too big.

Many developers already have some kind of implicit change size limit. It manifests itself as the feeling that it’s time to commit during a long coding session. So you might be already using the maximum change size constraint without being fully aware of it. The advantage of an explicit change size limit is that you can quantify it and keep yourself honest by comparing change size to some threshold. You can also use a tool or script which will watch uncommitted changes and notify you as soon as the change size gets close to or exceeds the threshold instead of relying on the gut feeling to know when there are too many changes. It’s also usually harder to “negotiate” with a script (e.g. a Git hook) compared to convincing yourself that the commit still has a reasonable size “just this one time”.

After a while this technique should naturally nudge you into working in a more focused way (unless you’re doing it already) along the lines of making one change at a time until it’s finished and committed. And if something else comes up during the change, you can add the new task to a TODO list for later or shelve the current modifications, finish the new task and unshelve. Sounds simple but it’s surprising how often we divert into more complex workflows.

Calculating change size

Calculating change size is a bit more tricky than it seems. The most obvious question is what to count: lines, words, characters or even the amount of actions/keystrokes in the editor since last commit? Should empty lines and whitespace characters be excluded? Regardless of the choice, there is a problem that modifications of the same size might have different real-world weight. For example, in Java adding an import statement is usually less work compared to writing a line of code that will be executed. Another problem is that conceptually simple changes can produce large diffs. For example, renaming a function which is used across the project will create a large diff even though it’s a single rename. There are other problems of this kind and none of them have simple solutions just because it’s extremely hard to measure complexity and impact of code. In practice though, I found that using a simple metric like non-empty lines of code works quite well.

Another question is how to calculate modifications on diffs. If we are measuring change size in lines, then adding a brand-new file with 10 lines of code should count as 10. But what if the change is replacing 10 lines of code with completely different 5 lines? Should the change size include only the new code, both old and new 10 + 5 = 15 or maybe the average (10 + 5) / 2 = 7.5?

Finally, there is a question about how to quantify file system changes such as moving/renaming files or creating directories and how well the diff algorithm can determine if a file was moved or it’s a new file with similar content.

Given the above, the maximum change size is obviously a heuristic which you will need to experiment with to make it useful. A good strategy might be to start with a threshold which is quite low and gradually increase it. Note that you can also track your progress in version control by analysing how commit sizes changed over time.

Thought experiments

To explore the idea, let’s perform a couple of thought experiments:

With the maximum change of 1_000_000 lines of code you will never notice that there is a change size limit because pretty much nobody changes a million lines of code on daily basis.
If we go down to the maximum change size of 10_000, once again it’s unlikely to be noticed, except for large-scale code migration and restructuring.
It becomes more interesting with the change size limit around 100 and below. This is where the learning opportunities are.
Finally, with the limit of 1 or 2 lines, it becomes impossible to make any significant changes. Unless we cheat and redefine the constraint to be 1 or 2 actions/keystrokes in the editor. Then automated code transformations are allowed (for example, inline refactoring or create new class). The interesting aspect of it is that if transformations don’t break existing functionality, then they are perfectly fine to commit and share the with other people after every single change. There is probably not enough functionality in your editor/IDE to actually do this but it was just a thought experiment.

Try it yourself!

Maximum change size is an easy constraint to try. You can start by watching change size manually on the command line (e.g. with git diff). Or you can roll out a proper script to regularly check the change size and prohibit commits above threshold. I went down a different rabbit hole of writing a Limited WIP plugin for IntelliJ IDEs. Whatever path you choose, please remember that just like with other techniques nothing is set in stone, so feel free to experiment with it and have fun.

Hello Http4k

Thu, 30 Jul 2020 00:00:00 +0000

This is a quick intro into http4k — a Kotlin library for writing HTTP servers and clients. Unlike many other libraries and frameworks with (over)complicated core abstractions and workflows, http4k uses a few simple concepts which encourage good design and testability.

The examples below are based on the “Live coding: Server as a function with http4k” talk I’ve been doing earlier this year.

Hello handler

The main concept in http4k is HttpHandler, which is literally a function that takes immutable Request and returns immutable Response (where Request and Response support all the usual HTTP things like headers and queries):

typealias HttpHandler = (Request) -> Response

So the simplest server in http4k is just a lambda which returns an instance of Response with OK status and (optional) body content. Of course, lambdas can’t send/receive HTTP requests over the network — for that we need an actual HTTP server. Http4k doesn’t attempt to reinvent the wheel and instead of implementing its own server/client, it has adapters to integrate with existing HTTP Java servers/clients. In this case, let’s use ApacheServer which is an adapter for Apache HttpComponents:

import org.http4k.core.* import org.http4k.server.* import org.http4k.core.Status.Companion.OK fun main() { val httpHandler: HttpHandler = { request: Request -> Response(OK).body("Hello 🌍") } httpHandler.asServer(ApacheServer(port = 1234)).start() }

We can run the code above in IDE and test it from command line with curl:

$ curl -D - http://localhost:1234 HTTP/1.1 200 OK ... Hello 🌍%

The output looks good but it would be nice to use server code directly from Kotlin. To do this we need an HTTP client. In particular, we can use OkHttp which is an adapter for OkHttp library. Note that HTTP client is also HttpHandler, i.e. it has the same type as the server!

fun main() { val httpHandler: HttpHandler = { request: Request -> Response(OK).body("Hello 🌍") } httpHandler.asServer(ApacheServer(port = 1234)).start() val httpClient: HttpHandler = OkHttp() val response: Response = httpClient(Request(GET, "http://localhost:1234")) println(response) }

If we run the code above, the output will be almost identical to the curl output:

HTTP/1.1 200 OK ... Hello 🌍

Since both server and client have the same signature, we can refactor the code to remove ApacheServer and OkHttp adapters so that the server handler is called directly:

fun main() { val httpServer: HttpHandler = { request: Request -> Response(OK).body("Hello 🌍") } val httpClient = httpServer val response = httpClient(Request(GET, "http://localhost:1234")) println(response) }

And after a couple more refactorings:

fun main() { val httpServer: HttpHandler = { _: Request -> Response(OK).body("Hello 🌍") } println(httpServer(Request(GET, "http://localhost:1234"))) }

In general, this outlines the http4k approach for making HTTP servers testable: start the servers in the same process and use in-memory communication, or if it’s an external server, replace it with a fake server (supported by contract tests) or with record/replay client.

Hello routes

The server above works just fine except that it replies to any request regardless of the URI. For example, it will reply with “Hello 🌍” even if we POST to http://localhost:1234/foo.

In order to fix this, we can use the routes() function which takes one or more RoutingHttpHandlers and aggregates them into a composite RoutingHttpHandler.

The simplest way to create a RoutingHttpHandler is by using the following mini-DSL. First, "/hello" bind GET bundles path and HTTP method into the PathMethod data class, and then, ... to { request: Request -> ... } creates RoutingHttpHandler.

import org.http4k.core.* import org.http4k.server.* import org.http4k.core.Method.GET import org.http4k.core.Status.Companion.OK fun main() { val httpHandler: HttpHandler = routes( "/hello" bind GET to { request: Request -> Response(OK).body("Hello 🌍") } ) httpHandler.asServer(ApacheServer(port = 1234)).start() val httpClient: HttpHandler = OkHttp() val response: Response = httpClient(Request(GET, "http://localhost:1234")) println(response) }

If we run the code above, it will print 404 response from RoutingHttpHandler because httpClient still requests “/” while httpHandler only responds to the “/hello” path. Obviously, to fix this we should modify the request:

val response: Response = httpClient(Request(GET, "http://localhost:1234/hello"))

Hello filter

Another important concept in http4k is Filter, which is essentially an HttpHandler wrapper:

interface Filter : (HttpHandler) -> HttpHandler

To illustrate this, let’s create a Filter from scratch. The simplest possible Filter is the one that just returns its argument (note in the code below the usage of the Filter { ... } function which saves us the effort of creating an anonymous object):

fun main() { // Equivalent to `Filter.NoOp` val filter = Filter { nextHandler -> nextHandler } }

Let’s wrap nextHandler into another HttpHandler called wrapperHandler. Functionally, the code still works in the same way (and doesn’t do anything useful):

fun main() { val filter = Filter { nextHandler -> val wrapperHandler: HttpHandler = { request -> nextHandler(request) } wrapperHandler } }

However, we can now add more functionality to the wrapperHandler. For example, to create “catch all” filter we can surround nextHandler(request) with try/catch expression and return I_M_A_TEAPOT status in case of Exception:

fun main() { val catchAll = Filter { nextHandler -> val wrapperHandler: HttpHandler = { request -> try { nextHandler(request) } catch (e: Exception) { Response(I_M_A_TEAPOT) // not the best error handling strategy } } wrapperHandler } }

To try out the filter, let’s make the server extract the “name” query parameter from the request and throw an exception if the “name” parameter is missing: request.query("name") ?: error("No name"). We can apply the filter using .withFilter(catchAll):

fun main() { val catchAll = Filter { nextHandler -> val wrapperHandler: HttpHandler = { request -> try { nextHandler(request) } catch (e: Exception) { Response(I_M_A_TEAPOT) } } wrapperHandler } val httpHandler: HttpHandler = routes( "/hello" bind GET to { request: Request -> val name = request.query("name") ?: error("No name") Response(OK).body("Hello $name") } ).withFilter(catchAll) httpHandler.asServer(ApacheServer(port = 1234)).start() }

As expected, it’s a 200 response on “hello?name=world” GET request:

$ curl -D - http://localhost:1234/hello?name=world HTTP/1.1 200 OK ... Hello world%

And it’s a 418 “I’m a teapot” response if we forget to specify the “name”:

$ curl -D - 'http://localhost:1234/hello' HTTP/1.1 418 I'm a teapot ...

You can find more filters bundled with http4k in org.http4k.filter.ServerFilters and org.http4k.filter.ClientFilters objects.

HTTP as a function

Just like any other library or framework http4k is only an abstraction on top of the HTTP protocol. Even the simplest GET request can end up talking to multiple servers (think DNS, SSL) and with containerised clouds (using kubernetes, etc.) there is even more stuff going on in the background. So the library design is by no means a representation of reality but rather it’s an abstraction for the library users. Arguably, for the majority of users plain old function (POF?) with an immutable request/response is the best abstraction over HTTP.

The design of http4k was inspired by “Your Server as a Function” paper which defines HttpHandler in a bit more complicated way and roughly translates to Kotlin like this:

interface HttpHandler<Request, Response>: (Request) -> Future<Response>

The main difference compared to http4k is the Future in the return type which highlights the asynchronous nature of HTTP requests. The advantage is that requests can be treated as tasks to be scheduled for execution asynchronously. The disadvantage is that Future complicates every request and response. There are potential performance benefits, and no doubt some large-scale companies can benefit from this, but the chances are you’re not one of them. Another argument against using Future is that HTTP abstraction should only be dealing with HTTP and stay away from threading strategy, similar to how you would avoid mixing HTTP and domain logic.

From the simple design point of view, it’s also not easy to justify the conceptual complexity of an average HTTP framework with its own lifecycle and special testing suite. That’s not to mention the use of annotations, which are pretty much custom keywords and should be used sparingly.

Summary

One thing to remember from this blog, is that HTTP servers and clients are uniform, i.e. both server and client have the same type (in Kotlin terms (Request) -> Response or variation of it). This is exactly what makes it possible to write “your server as a function” and http4k takes this approach to its simplest form. So if you’re thinking about writing a new web-framework or library, please really consider using this as its core concept!

It’s worth mentioning that http4k has been successfully used in large-scale enterprise systems. There are quite a few http4k modules and this blog just scratches the surface of the core module. You can find out more about http4k on its website. For a bit more in-depth overview, here is a video by http4k authors:

Given the amount of web-frameworks and technical marketing (things that come to mind are Spring and, obviously, this blogpost), it can be genuinely hard to know what is the right tool for the problem at hand. The important thing is to avoid Resume Driven Development by actually solving the problem instead of focusing on a particular technology.

Squasher

Sat, 18 Jul 2020 00:00:00 +0000

Recently Peter Kofler wrote a blog post in which he looks at the Design of a Separable Transition-Diagram Compiler paper by Melvin E. Conway (which was published in 1963 and considered to be the first paper mentioning coroutines) and reimplements a basic coroutine in the modern assembly for Windows. Since I don’t have access to any Windows machines, Peter kindly agreed to pair up with me on porting the code from his blog post to macOS and 64-bit assembly. While doing this I ended up reading the paper and then cloning and refactoring the code to make it a bit simpler at the expense of the code not matching the paper anymore.

Below you will find my interpretation of the problem as described in the first two pages of the paper and solutions with/without coroutines in Kotlin and NASM (see this github repo for full source code and tests). It’s worth mentioning that NASM solutions and implementation of basic coroutines are based on Peter’s code.

The problem

Imagine a simple task of replacing all occurrences of ** (two asterisks) with ^ in a stream of data. For example, given ***abc, the output should be ^*abc. In the paper this process is called “squashing” and can be implemented in Kotlin as a one-liner:

fun String.squash() = this.replace("**", "^")

However, there is an additional constraint which makes the problem a bit more interesting. In particular, squasher can only consume or output one character at a time. In Kotlin this can be expressed as an Iterator<Char> which wraps another Iterator<Char>. (In the paper the input is being read infinitely from punch cards but it’s safe to skip these details.)

Kotlin implementation

Given the constraint, a straightforward squasher implementation might look like this:

fun squasher(input: Iterator<Char>) = object : Iterator<Char> { var hasLastChar = false var lastChar = 0.toChar() override fun next() = if (hasLastChar) { hasLastChar = false lastChar } else { var char = input.next() if (char == '*') { lastChar = input.next() if (lastChar == '*') char = '^' else hasLastChar = true } char } override fun hasNext() = input.hasNext() || hasLastChar }

Because of the requirement to output only one char at a time, the code has additional non-essential complexity of keeping the state of the iterator between invocations in hasLastChar and lastChar properties. In a way, we can think about the code block guarded by hasLastChar as a callback or even as a state machine with only two states.

This is exactly the situation when coroutines can help by “returning” values in the middle of a function and resuming from that point later. The squasher implementation below doesn’t need the hasLastChar flag since this logic is already embedded in the control flow of the code.

fun squasher(input: Iterator<Char>): Iterator<Char> = sequence { while (input.hasNext()) { val char = input.next() if (char != '*') yield(char) else { val lastChar = input.next() if (lastChar == '*') { yield('^') } else { yield(char) yield(lastChar) } } } }.iterator()

I don’t think there is anything particularly revealing in the examples above but it feels nice to be able to translate such an old problem to a modern language and it, hopefully, explains the problem domain before diving into assembly code.

NASM implementation (with functions)

From this point, I’ll assume that you are familiar with x86 assembler enough to be able to get the gist of the following code snippets (which are written using NASM and can be found in this github repo). If not, you can try this intro, this guide or this article on 64-bit assembly.

For simplicity, the NASM implementation doesn’t attempt to mimic an Iterator but rather reads a fixed size input from stdin. The main function is just a loop which for each input char calls squasher and prints to stdout one character at a time:

main: _call read_input mov qword [hasLastChar], FALSE .loop: _call squasher ; squash char from input and put result into rax _call write ; print char from rax mov rax, [i] cmp rax, INPUT_SIZE jne .loop ; jump back if i != INPUT_SIZE

The squasher function uses next_char to get the next input character and puts the output character into the rax register before returning to main. Similar to the Kotlin version without coroutines, squasher has to keep global state and includes additional logic to return lastChar.

squasher: mov rax, [hasLastChar] cmp rax, FALSE je .no_lastChar mov qword [hasLastChar], FALSE mov rax, [lastChar] _ret .no_lastChar: _call next_char ; puts return value into rax cmp rax, '*' je .check_second_asterisk _ret .check_second_asterisk: mov rbx, rax ; temporary save first char to rbx _call next_char ; puts return value into rax cmp rax, '*' je .do_squashing mov [lastChar], rax mov qword [hasLastChar], TRUE ; remember to write lastChar next time mov rax, rbx ; load first char from rbx _ret .do_squashing: mov rax, '^' _ret

As you may have noticed, the code above uses _call and _ret macros (instead of built-in call and ret). This is to show implementation details of simplistic function calling convention.

%macro _call 1 mov rdx, %%_end push rdx ; save returning point on stack jmp %1 ; call sub-function %%_end: nop ; returning point %endmacro %macro _ret 0 pop rdx ; load returning point from stack jmp rdx ; jump back into caller function %endmacro

NASM implementation (with coroutines)

Since x86 assembler doesn’t have coroutines, let’s implement them. The idea is to reserve additional memory for each coroutine to store the address of its resuming point (initially set to the first instruction of coroutine) so when coroutine is called again, it continues from where it left off.

section .data instruction_at_main: dq main ; stores resuming point of main instruction_at_squasher: dq squasher ; stores resuming point of squasher

To pass control to a coroutine we can use the following co_call macro. The macro expects to be called from a coroutine so it updates the resuming point of the current coroutine (in case the coroutine gets control back some time later). Then the macro looks up the resuming point of the coroutine it’s about to call and jumps to that address. No doubt, this is a toy implementation which doesn’t save/restore registers or stack when switching between coroutines but, hopefully, it’s simple enough to illustrate the concept.

%macro co_call 1 ; update resuming point of the current coroutine pop rdx ; load address of the store mov rcx, %%_end ; mov [rdx], rcx ; update resuming point in the store ; pass control to the next coroutine mov rdx, instruction_at_%1 ; get address of the store push rdx ; save address of the store for later jmp [rdx] ; resume the next coroutine %%_end: nop ; resuming point %endmacro

This main function is the same as the version without coroutines except for the use of co_call macro and having to prepare the stack so the macro thinks it’s called from a coroutine.

main: call read_input mov rax, instruction_at_main push rax ; prepare stack for coroutine call .loop: co_call squasher ; squash char from input and put result into rax call write ; print char from rax mov rax, [i] cmp rax, INPUT_SIZE jne .loop ; jump back if i != INPUT_SIZE

There are more changes in squasher though. The most interesting change is that there are no “returns” because conceptually squasher is not any different from the main coroutine and there is nowhere to return to, so squasher can only call sub-functions or pass control to another coroutine. As a consequence the squasher now is an infinite loop (see jmp squasher instructions). Similar to the Kotlin version with coroutines, squasher doesn’t need the hasLastChar variable anymore because the branching is stored as a resuming point.

squasher: call next_char cmp rax, '*' je .check_second_asterisk co_call main jmp squasher .check_second_asterisk: mov rbx, rax ; temporary save first char to rbx call next_char cmp rax, '*' je .do_squashing mov [lastChar], rax ; save rax so its not erased by main mov rax, rbx ; load first char from rbx co_call main mov rax, [lastChar] co_call main jmp squasher .do_squashing: mov rax, '^' co_call main jmp squasher

You can find the full source code for 64-bit Linux and macOS in this github repo.

How is that different from threads?

Fundamentally, OS kernel switching between threads has no magic and is based on the instruction pointer manipulation similar to the macro above (for example, see this stackoverflow question). The main difference compared to coroutines is that conceptually switching between threads is preemptive, i.e. the function which is being executed doesn’t decide when to switch to another function, instead, thread scheduler does it. Another difference is that thread scheduler normally doesn’t provide any guarantees about which CPU core thread will run on and, therefore, the need for “thread safety” which really is all about “multi-core safety”. (As an interesting side note there are no special assembly instructions for switching between CPU cores. Instead, several CPUs just run instructions independently with access to common shared memory. See The Unofficial SMP FAQ). Overall, even though the underlying mechanism might be similar, coroutines give you more control over context switches compared to threads.

Summary

Kotlin being a high-level language has a more sophisticated coroutine implementation, than the one shown above, with a lot of details related to other language features and type system but at the core it’s the same trick of storing the latest entry point of a coroutine and, when the coroutine is resumed, jumping to the right part of the code (you can find the implementation in CoroutineTransformerMethodVisitor which uses tableswitch instruction to do the jump).

Hopefully, this blog helps with understanding basic coroutine implementation details. For a description of coroutine flavours from the point of view of programming language user, you can try reading the following posts:

Intellij Settings

Tue, 26 May 2020 00:00:00 +0000

For the benefit of anyone interested and as a reference for myself, here are some of the IntelliJ and related macOS settings I normally use. This page will be occasionally updated as IntelliJ and my preferences evolve.

Last updated on 20th May 2025 for IntelliJ IDEA 2025.2 EAP.

Settings

In Main Menu > View > Appearance
- Enable Navigation Bar > Don't Show
- Enable Compact Mode
- Disable Toolbar
In Editor > General
- Disable Appearance > Show intention bulb
- Disable Breadcrumbs > Show breadcrumbs
- Enable Smart Keys > Use "CamelHumps" words
- Disable Smart Keys > Honor "CamelHumps" words on double click
- Disable Smart Keys > Surround selection on typing quote or brace
- Disable Rich-Text Copy > Copy as rich text
- Disable Sticky Lines > Show sticky lines when scrolling
- Set Editor Tab > Tab placement to None
In Editor > Code Editing
- Disable Usages of element at caret
- Disable Show quick documentation on hover
- Set The 'Next Error' action goes through to All problems
In Editor > Inlay Hints
- Enable all, and then disable with Toggle Inlay Hints Globally action
  (optionally assign a shortcut to the action for a better to way show hints on-demand)
In Editor > Font set Size to 15.0
In Appearance & Behavior > Appearance
- Disable Smooth scrolling (so that the scrolling position is on the border between lines)
- Disable System Settings > Updates > Show What's New in the editor after an IDE update
In Version Control > Git
- Set Update method to Rebase
- Enable Use credential helper — don’t save passwords in IDE (see IDEA-211251)
In Advanaced Settings
- Enable User Interface > Position mouse cursor on default button in dialogs
- Disable User Interface > Use native faile chooser dialog on Windows/macOS — because IntelliJ file chooser dialog is just better
- Enable Version Control > Toggle commit controls
- Disable Version Control > Enable Commit tool window

Registry

Invoke Find Action and search for “Registry…”:

Disable ide.allow.merge.buttons —
don’t merge buttons in VCS dialogs (see IJPL-87262, IJPL-80718, etc.)
Disable light.edit.file.open.enabled —
because light editor it’s not ready yet (see IDEA-236868)
Enable analyze.exceptions.on.the.fly —
automatically analyze clipboard content for stacktrace on IDE frame activation
Enable ide.completion.delay.autopopup.until.completed —
deterministic auto-completion results (see KT-29042)
Disable debugger.valueTooltipAutoShow — no debugger tooltip on mouse over
Disable show.live.templates.in.completion (see IDEA-216928)
Increase undo.globalUndoLimit and undo.documentUndoLimit
Enable toolwindow.disable.overlay.by.double.key —
because when using alt+... shortcuts (e.g. with IJKL shortcuts plugin) it can be annoying in presentation mode to accidentally open tool window overlay (see IDEA-112097)
Disable ide.switcher.tool.window.list and ide.recent.files.tool.window.list — don’t show tool windows in switcher and recent file popups (see IDEA-131137)
Disable project.tree.structure.show.url to hide the path next to the project name in the Project tool window

Keymap (based on “IntelliJ IDEA Classic”)

Use IJKL shortcuts plugin for navigation
Assign F1 to Quick Fix (see QuickFix plugin), remove it from Context Help
Assign alt+\ to Show Context Menu (i.e. the popup window displayed on the right click)
Assign alt+1 to Project tool window
Assign cmd+2 and alt+2 to Version Control tool window
Assign cmd+9 to Live Plugins tool window (see LivePlugin plugin)
Assign cmd+§ and alt+§ (i.e. to the key left of 1) to Main Menu > View > Tool Windows
Assign cmd+M to Scroll to Center
Assign cmd+F4 to Close Project
Assign cmd+shift+G to Execute Gradle Task
Assign cmd+ctrl+shift+P to Enter Presentation Mode
Assign alt+shift+9 to Put arguments/parameters on separate lines intentions
Assign alt+shift+8 to Put arguments/parameters on one line intentions
Assign alt+shift+7 to Add names to call arguments and Remove all argument names intentions
Assign cmd+alt+H to Window > Notifications > Close First / Close All (see IDEA-218434)
Assign ctrl+d to Show Diff for Lines
Assign ctrl+b to Version Control Systems > Git > Branches popup window
Remove Markdown > Create Link shortcut because it clashes with Toggle Case
Remove Increase/Decrease Font Size in All Editors shortcuts

macOS (Sonoma and above)

Disable accents defaults write -g ApplePressAndHoldEnabled -bool false
Max values for Preferences > Keyboard > Key Repeat and Delay Until Repeat
Map CapsLock to Control in Preferences > Keyboard > Modifier Keys...
Enable “Use scroll gesture with modifier keys to zoom” in
Preferences > Accessibility > Zoom and select Control as the modifier key for gesture
Select “Three-Finger Drag” as dragging style in
Preferences > Accessibility > Pointer Control > Trackpad Options...
Disable shortcuts in Preferences > Keyboard > Shortcuts that clash with IntelliJ (e.g. cmd+shift+/; there is usually a popup notification with the conflict after IDE install)
Disable option-space, see https://superuser.com/questions/78245

Note On Loudness

Thu, 18 Apr 2019 00:00:00 +0000

Theory

These days audio loudness is measured in LUFS (Loudness Units relative to Full Scale). This is a relative measurement, where, as I understand, 0 LUFS is the loudest audio you’re allowed to broadcast according to the standard by European Broadcasting Union. And -10 LUFS is 10 decibel quieter than 0 LUFS. Although LUFS is a bit more involved than decibel and measures “weighted perceived loudness”.

Youtube guarantees that videos are not louder than -13 LUFS, i.e. if you upload content louder than that, it will be normalised to -13 LUFS. In terms of the loudness of actual youtube videos, I looked at couple channels and found that some of them try to be as loud as possible at -13 or -15 (maybe because that’s the loudness of youtube ads), while other channels are not that loud, around -30 (in particular, I looked at conference talks).

I didn’t come to a conclusion what is the “right” loudness level for youtube but in general the recommended broadcasting level is -23 LUFS (see EBU R128 Introduction video for details).

Practice

You can measure LUFS of a video using ffmpeg. For example,

ffmpeg -nostats -i 'video.mp4' -filter_complex ebur128 -f null -

will print few lines to console and in the end a summary with something like:

Integrated loudness:
I:         -15.9 LUFS

where “integrated” means it was calculated for the whole video, not just a part of it.

You can also download videos from youtube using youtube-dl and then run them through ffmpeg to determine their loudness.

To increase volume of a video you can do

ffmpeg -i inputfile -vcodec copy -af "volume=10dB" outputfile

Dmitry's Blog

Mcp Sse The Hard Way

Tidy Kotlin

Contents

High-level declarations first

Maximum privacy

Keep variables close to their usages

Inline variables with single usage

Remove argument names when their types are distinct

Add argument names when types are the same or generic

Pass named arguments in the order of parameter declaration

Put parameters on one line

Put arguments on one line

Put parameters on separate lines

Stop the CONSTANT SHOUTING

Consider using tiny types

Sane Intellij Plugin Development With Live Plugin

Contents

Why?

What does it look like?

How does it work?

Notifications, logging, basic input

Actions

Action groups

Editor and Document

Intentions and Inspections

IntelliJ APIs mini cheat sheet

Final thoughts

Limited WIP — test && commit || revert

The idea

The most continuous integration

TCR with TDD?

Try it yourself!

Limited WIP — Auto-revert

What is auto-revert?

Thought experiments

What about real-world projects?

Try it yourself!

Limited WIP — Maximum change size

The idea

Calculating change size

Thought experiments

Try it yourself!

Hello Http4k

Hello handler

Hello routes

Hello filter

HTTP as a function

Summary

Squasher

The problem

Kotlin implementation

NASM implementation (with functions)

NASM implementation (with coroutines)

How is that different from threads?

Summary

Intellij Settings

Settings

Registry

Keymap (based on “IntelliJ IDEA Classic”)

macOS (Sonoma and above)

Note On Loudness

Theory

Practice