Behaviours, Not Tests

Daniel Terhorst-North changed my life.

I always recommend his Introducing BDD. It explains how Behaviour-Driven Development started from the intuition that TDD is not merely about testing. The word “test” itself points developers in the wrong direction, he says, toward verification, toward the past. He prefers using “behaviour”.

It’s a dramatic change of perspective. For Daniel tests document the system’s behaviour from the outside, from the point of view of its clients or users; therefore, they describe what the system should do, using narrative sentences.

They are in fact business requirements, written in the business language and, of course, conceived before the implementation even exists.

Why Test-First?

Some find it counterintuitive, if not crazy, writing tests before the system under test even exists. Yet, if you replace the word “test” with “requirement”, all makes sense, and it’s the opposite approach to sound crazy. What are we supposed to do? To write the code without knowing where to go and then, when we are done, to figure out what the requirements were?

If only TDD was called Requirement-Driven Development, no one would find it counterintuitive. “Test” suggests the idea of verifying something that is already done. Not the most fortunate pick, Kent…

As Daniel wrote:

A really useful way to stay focused was to ask:
“What’s the next most important thing the system doesn’t do?”

This question defines your next test; the test becomes a statement of intent, a promise. With the test in place, you commit (here’s the word!) to a specific, future behaviour for your product.

A Commit Is A Promise

I started thinking to commit messages the same way, and something clicked. Maybe the word “commit” in the Git lingo is not casual. Maybe making a commit means making a promise too.

It’s ironic how Git lets you commit to something when the code is already complete. A bit late, indeed. The real twist is how in Jujutsu it’s the exact opposite to be idiomatic: you first jj commit, then you write the code.
We’ll get this in few seconds. Back to the message.

How To Describe A Promise?

A commit message such as:

Use Set instead of List

is neither a commitment nor a behaviour description: it’s an activity report.
It makes more sense when written in the style of a BDD method:

The cart does not crash when it contains more than 10k items

That’s not a cosmetic difference. The former is about the actions performed by the programmer, the past, the how. The latter about the feature, the present, the what.

Tell Me What The Software Does, Not What You Have Done

The second form is what I would like to read in the Git history, and is in line with the idea of Conventional Commits, which promise to “Automatically generate CHANGELOGs”.

When I check out a commit, I know that someone worked hard on their keyboard to make the software behave like it should. Kudos, I’m grateful. But when it’s my turn to work on top of that commit there’s no point answering the question:

What did the programmers do during that work session?

It’s more useful to have an answer to:

What's the project behaviour NOW?

As Ferdinando Santacroce loves to say: the commit message should capture what cannot be deduced from the code only: the context, the motivation and the thought process behind the solution. To me, that alone is a good reason not to let LLMs write my commit messages.

Write It First

Then I stumbled upon this tweet by Eric Willeke:

A point of view of disarming simplicity. It’s test-first applied to version control. It instantly resonated with me.

If you are accustomed to write tests before the implementation, writing commit messages before coding will be just as natural. Messages become statement of intents, commits become commitments.

This is how the Squash Workflow works in Jujutsu. It revolves around the idea of creating a new empty revision, together with its description, even before touching any file. Not a coincidence that the command to do this is jj commit. The commit message as the intent and the unit of work.

First you write it, then you make it true.

What Happens When You Do It

In 2012, with my teammates we started applying this technique, with Git. It was a matter of doing

git commit --allow-empty -m <message>

followed by

git commit --amend

when we were done. Some Git GUIs makes this easy.

Over the years, this is what I experienced:

• It aligns pair members. Whether I am the driver or the navigator, I must agree with my pair on the message and on the goal. This stimulates a conversation. The moment we start programming, we genuinely agree on what we want to achieve.
  When I code alone, I have to find an agreement with my own understading. It’s equally challenging and it equally pays off.

• It’s easier to focus: tests and commit message are the coding session’s center of gravity. When I digress, the promise in the commit message is there to bring me back to the point.
  I’m less prone to lose focus.

• It sets a micro scope: Before, I used to ask myself: “Should I stop coding now? How much is enough? Am I done yet?”.
  Checking if I reached the planned goal is just easier.

• Simpler commit reviews: I always review my changes before making them final. Instead of asking “Well well well, let’s find out, what have I done here? I don’t even remember…” I can ask myself “Have I done all and only what I committed to?”.
  It’s a much easier and targeted question to answer.

• More accurate and faithful statements: when writing the message before coding, being very specific comes naturally. It’s easy to refrain from being vague and writing Fix or Update Repository.cs. The message and the resulting commit content naturally match, because the message was the goal since the very beginning.

• Each message triggers a micro design session: the mere act of defining the commit message requires reaching an agreement with my pairing partner. We need to concord on the words to describe the desired behaviour, on the names to give to domain concepts and, ultimately, on which component owns the behaviour. These are all design questions, and we cannot write the message until we’ve addressed them.
  Starting a pairing session, there’s nothing nicer than discussing about design.

• It creates a natural timebox: I make my best to commit to baby-step progresses, from a stable state to the next little goal. It’s easy to verify if a chance belongs or not to the goal.

• It goes well with short-lived feature branches: messages define micro-goals, and they live in the context of the wider goal defined by the branch name. Which is also pre-emptive, by design.
  These 2 levels of goals help me steering the direction.

• The commit history gains natural granularity. Each commit has a single goal (one could say it respects the Single Responsibility Principle). A feature branch becomes a readable sequence of behaviours, each commit a small step in a larger story.
  A natural changelog emerges.

This experiment was started in 2012 by Arialdo Martini, Mattia Piccinetti, Gian Marco Gherardi, Guglielmo Brasile, Francesco Pichierri, and Giuseppe Mariano. It was initially described in Pre-emptive Commit Comments.
It is described in details in the book Git Essentials by Ferdinando Santacroce.

References

• Jujustu
• Conventional Commits
• Pre-Emptive Commit Comments
• Ferdinando Santacroce
  • Git Essentials - Ferdinando Santacroce

The compiler of your beautiful new language — with which you want to bring the use of goto and null back to life — will have several components:

• A Lexer, for splitting the source code into tokens.
• A Parser, to analyze the token sequence and to convert it into a syntax tree, and to check the grammatical structure.
• An Intermediate Code Generator.
• A Linker, and so on.

Here we are focusing on the very first component: a piece of code able to analyze the source, to check its syntax against the esotheric formal grammar you defined, and to generate something very well structured for the next components to crunch.

It would be a (probably very complex) function with a signature like:

type SourceCode = string

type AbstractSyntaxTree =
| Goto of Label
| VariableDefinition of ...
| ...

val parser : SourceCode -> AbstractSyntaxTree

Most likely, you would like the parser to fail and to emit syntax errors in case there are any. Better return a Result, then:

type ParserError = { Expected: string; Encountered: string }
// or just: type ParserError = string

val parser : SourceCode -> Result<ParserError, AbstractSyntaxTree>

If you think about it, that’s not qualitatively different from deserializing a JSON string:

type JSON = string
type Error = string

val jsonDeserializer : JSON  -> Result<Error, MyObject>

Of course, it’s likely that a programming language grammar is more complex than the JSON grammar. But the two concepts are alike, and so are the signatures.

Tree-sitter too does something similar. It parses a string like:

"let x = 42;"

and emits a tree like:

(program
  (variable_declaration
    (lexical_declaration
      (identifier)
      (assignment_expression
        (number)))))

We can imagine the Tree-sitter grammar for F# as a function with this signature:

val treeSitter : SourceCode -> Result<TreeSitterError, TreeSitterSExpression>

I guess you see the pattern.
A parser is a function that takes loosely-structured data (usually — but not necessarily — text), and tries to build a more structured data out of it, accordingly to the rules of a formal grammar.

An inherent recursive structure

We say that the input data is loosely-structured because, in fact, it is not granted to adhere to the rules of the chosen grammar. Indeed, if it violates them, then we expect the parser to fail and to emit an error, to help the user identify the syntax errors.

There are multiple approaches to parsing, including the renowned Regular Expressions.
Monadic Parser Combinators are a particularly fascinating one: they are an example of Recursive Descent Parsers. This means that no matter how complex the parser for a grammar is, it is defined based on smaller, simpler parsers, and those in turn are defined based on even smaller and simpler ones, and so on recursively, down to the trivial parsers.
You can see the same from the opposite perspective: starting from the trivial parsers, by combining them together and then by combining their results, recursively, the parser for any arbitrary grammar can be built.

Now, if writing the trivial parsers is, well, trivial, the only challenge that’s left is to learn how to combine parsers. That is, how these Parsers Combinators work.

That’s the goal of this series.

How we will proceed

There are many similar series online, some specific to F# such as The “Understanding Parser Combinators” series by Scott Wlaschin. This post tries to stand out in a few different ways:

• Many tutorials begin with writing a simple parser — conventionally, the single-character parser. This does not.
  Instead, we will focus on combinators first, postponing the implementation of concrete parsers to the last chapter. When I was first introduced to parsers, I was just confused: what on earth does it mean to parse a single character? What’s the point, where is this leading me? My own ‘aha!’ moment came when I got how composition turns tiny parsers into something actually useful.
  I hope I can help you skip past that initial disorientation entirely.

• We will write code with Test-Driven Development.
  Isn’t it ironic that we developers often lament the absence of tests in our daily job projects and yet, when it comes to writing posts, tutorials and books, we never address testing at all?

Ready? Let’s go started.

References

• Recursive Descent Parser
• Tree-sitter
• Scott Wlaschin - The “Understanding Parser Combinators” series

Comments

GitHub Discussions

Functional Programming For The Rest Os Us

Interested in FP? Be the first to be notified when new introductory articles on the topic are published.

Let’s walk through the steps I took. As often happens with Emacs, along the path of exploring its source code, we will find some random pearls here and there to pick.

TL; DR:

(defun consult-line-symbol-at-point ()
  "Search for a line matching the symbol found near point."
  (interactive)
  (consult-line
   (or (thing-at-point 'symbol))))
   
(global-set-key (kbd "M-s .") #'consult-line-symbol-at-point)
(global-set-key (kbd "M-s M-s .") #'isearch-forward-symbol-at-point)

This issue is based on a lesson I got from Protesilaos.

You have no idea how powerful isearch is!

I was fascinated by the post You have no idea how powerful isearch is! by Bozhidar Batsov. In fact, I really had no idea.

One of the tricks he suggests is:

Type M-s . to search for the symbol at point.
(useful in the context of programming languages)

It’s very convenient: when your point is over a symbol, just hit M-s . to find other occurrences. Emacs will do its best to figure out what symbol means.

M-s . (that is, isearch-forward-symbol-at-point) is so more convenient than what I was used to do: once positioned on a symbol,

• C-= (er/expand-region) to select it
• M-w to copy it.
• C-s to trigger consult-line
• C-y to use it as the search pattern.

M-s . does the same in one single shot. With the only difference that, alas!, it uses isearch-forward instead of consult-line.
As I mentioned, once tried consult.el, I could not do without it anymore. Unfortunately, consult.el does not provide consult-line-symbol-at-point.

How hard is it to write it?

Edit: it turns out (thank you @Crandel!) that the consult.el README suggests obtaining this result simply with:

(consult-customize
 consult-line
 :add-history (seq-some #'thing-at-point '(region symbol)))

(defalias 'consult-line-thing-at-point 'consult-line)

(consult-customize
 consult-line-thing-at-point
 :initial (thing-at-point 'symbol))

Although next time I should RTFM!, I don’t regret not knowing this: learning it by hacking has been a rewarding experience.

How does isearch-forward-symbol-at-point work?

It makes sense to figure out first how isearch-forward-symbol-at-point works. Which you can do with either:

• M-x describe-key RET M-s .
• M-x describe-function isearch-forward-symbol-at-point

(in turn, describe-key and describe-function are conveniently bound respectively to C-h k and C-h f). You will get this documentation:

M-s . runs the command isearch-forward-symbol-at-point (found in
global-map), which is an interactive native-compiled Lisp function in
`isearch.el`.

It is bound to M-s ..

(isearch-forward-symbol-at-point &optional ARG)

From this help page, you can jump to the source code either:

• hitting s.
• running M-x help-view-source.
• hitting Enter while hovering the link isearch.el.

Fine. Here’s the code:

(defun isearch-forward-symbol-at-point (&optional arg)
  (interactive "P")
  (isearch-forward-symbol nil 1)
  (let ((bounds (find-tag-default-bounds))
        (count (and arg (prefix-numeric-value arg))))
    (cond
     (bounds
      (when (< (car bounds) (point))
	(goto-char (car bounds)))
      (isearch-yank-string
       (buffer-substring-no-properties (car bounds) (cdr bounds)))
      (when count
        (isearch-repeat-forward count)))
     (t
      (setq isearch-error "No symbol at point")
      (isearch-push-state)
      (isearch-update)))))

Spitting blood (I’m not that good at Lisp), I could extract the part that identifies the symbol at point with this function:

(defun get-symbol-at-point ()
  (let ((bounds (find-tag-default-bounds)))
    (cond
     (bounds
      (when (< (car bounds) (point))
	(goto-char (car bounds)))
       (buffer-substring-no-properties (car bounds) (cdr bounds)))
     (t ()))))

I’m honest: I cannot understand it completely. However, it’s easy to verify that it actually works:

(defun show-symbol-at-point ()
  (interactive)
  (message "%s" (get-symbol-at-point)))

Move the point on any symbol and execute M-x show-symbol-at-point RET. Cool: sounds like it is a good starting point.

Emacs’ geological layers?

My good friend Prot brought to my attention that in fact a function symbol-at-point is built-in in Emacs. It is part of thingatpt.el and a git blame reveals it has been introduced 19 years ago. Indeed, it is used here and there (check the code of org-open-at-point-global, describe-symbol, imenu--completion-buffer for example). It’s not completely clear to me why it has not been used in isearch-forward-symbol-at-point too. Maybe there are historical reasons. This confuses me, since isearch-forward-symbol-at-point has been introduced after thingatpt.el.

Whatever.
Let’s try it out, replacing my horrible custom get-symbol-at-point with the standard symbol-at-point:

(defun show-symbol-at-point ()
  (interactive)
  (message "%s" (symbol-at-point)))

Works like a charm! Good: we know how to get the symbol at point. We will have to pass it somehow to consult-line.

How does consult-line work?

Now, if we want to build a consult-line-symbol-at-point function, we’d better have a look to consult-line’s source code first.

Run M-x describe-function RET consult-line RET, then get to the code with M-x help-view-source RET (or s).

(defun consult-line (&optional initial start)
  "Search for a matching line.

Depending on the setting `consult-point-placement' the command
jumps to the beginning or the end of the first match on the line
or the line beginning.  The default candidate is the non-empty
line next to point.  This command obeys narrowing.  Optional
INITIAL input can be provided.  The search starting point is
changed if the START prefix argument is set.  The symbol at point
and the last `isearch-string' is added to the future history."

  (interactive (list nil (not (not current-prefix-arg))))
  (let* ((curr-line (line-number-at-pos (point) consult-line-numbers-widen))
         (top (not (eq start consult-line-start-from-top)))
         (candidates (consult--slow-operation "Collecting lines..."
                       (consult--line-candidates top curr-line))))
    (consult--read
     candidates
     :prompt (if top "Go to line from top: " "Go to line: ")
     :annotate (consult--line-prefix curr-line)
     :category 'consult-location
     :sort nil
     :require-match t
     ;; Always add last `isearch-string' to future history
     :add-history (list (thing-at-point 'symbol) isearch-string)
     :history '(:input consult--line-history)
     :lookup #'consult--line-match
     :default (car candidates)
     ;; Add `isearch-string' as initial input if starting from Isearch
     :initial (or initial
                  (and isearch-mode
                       (prog1 isearch-string (isearch-done))))
     :state (consult--location-state candidates))))

That’s it, not so huge, afterall. Basically, consult-line is just a thin wrapper around the internal function consult--read. We found something similar when we wanted to replace completing-read with a consult function, for getting real-time preview in Emacs: let’s zoom.
In the worst hypothesis, we could do the same and use this internal consult--read, passing it the symbol at point. But this is not even necessary. Notice the first consult-line parameter, initial? If it’s not nil, then it will be used as the initial pattern.
Let’s see how it works. Evaluate this with C-x C-e:

(consult-line "whatever")

Cool. It seems that really all we have to do is to feed consult-line with symbol-at-point as the initial value.

(defun consult-line-symbol-at-point ()
  "Search for a line matching the symbol found near point."
  (interactive)
  (consult-line (symbol-at-point)))
   
(global-set-key (kbd "M-s .") #'consult-line-symbol-at-point)

Try it out. Nope. We get an

apply: Wrong type argument: stringp, Try

This is because (symbol-at-point) returns a symbol, whereas consult-line wants a string. Fine: there must be a function to convert symbols to strings, right? I would try with:

M-x describe-function RET symbol

to list the symbol-related functions. symbol-name sounds like a good candidate. Its documentation says:

Return SYMBOL’s name, a string.

Let’t try it out:

(defun consult-line-symbol-at-point ()
  "Search for a line matching the symbol found near point."
  (interactive)
  (consult-line (symbol-name (symbol-at-point))))
   
(global-set-key (kbd "M-s .") #'consult-line-symbol-at-point)

Woah. We are almost there.

nil!

Does this work when there is no symbol at point? symbol-at-point declares:

Return the symbol at point, or nil if none is found.

(consult-line) happily works when the initial pattern is nil — or if it omitted. But the problem in our code is with symbol-name: (symbol-name nil) is "nil", as a string. So consult.el would search for that string. Mumble mumble, shall we raise an error, in that case?

DWIM

There’s a better alternative: to implement a Do What I Mean behavior. That is, if there is no symbol at point, M-s . shall act as a standard consult-line, if it’s over a symbol, it would search for that symbol.

We may use an if clause:

(defun consult-line-symbol-at-point ()
  "Search for a line matching the symbol found near point."
  (interactive)
  
  (let ((symbol (symbol-at-point)))
    (if symbol
        (consult-line (symbol-name symbol))
      (consult-line))))

Now, an if nested in a let can be shortened with a if-let:

(defun consult-line-symbol-at-point ()
  "Search for a line matching the symbol found near point."
  (interactive)
  
  (if-let ((symbol (symbol-at-point)))
        (consult-line (symbol-name symbol))
      (consult-line)))

But we can do better than this. We can remove the duplicated call to consult-line swapping if and consult-line, playing with the fact that:

(if condition
    (function when-true)
  (function when-false))

can be always written as:

(function
 (if condition 
     when-true
   when-false)

Here we go:

(defun consult-line-symbol-at-point ()
  (interactive)
  
  (let ((symbol (symbol-at-point)))
    (consult-line
     (if symbol
         (symbol-name symbol)
       ()))))

Now,

(if condition when-true ()))

can be shortened as:

(when condition when-true)

So:

(defun consult-line-symbol-at-point ()
  (interactive)
  
  (let ((symbol (symbol-at-point)))
    (consult-line (when symbol (symbol-name symbol)))))

Now, if you are curious as I am, it’s impossible not to take a closer look to the code of symbol-at-point. Which reveals a possible further improvement.

How does symbol-at-point work?

It turns out that also symbol-at-point is just a little wrapper, in this case around the more generic thing-at-point:

(defun symbol-at-point ()
  "Return the symbol at point, or nil if none is found."
  (let ((thing (thing-at-point 'symbol)))
    (if thing (intern thing))))

You see that call to intern? If you check intern’s documentation (C-h f intern RET), you will read:

Return the canonical symbol whose name is STRING.

Now we see why we needed to invoke symbol-name! symbol-at-point converts the string to a symbol, so we needed symbol-name to get the string back! It makes sense to directly invoke thing-at-point, then:

(defun consult-line-symbol-at-point ()
  (interactive)
  
  (let ((symbol (thing-at-point 'symbol)))
    (consult-line (when symbol symbol))))

Wait a second: what does thing-at-point return, when there is no thing at point? Move the point in an empty part of the buffer, then invoke it with:

M-x eval-expression RET (thing-at-point 'symbol)

It returns nil! We are lucky: it means that we don’t need all the ceremony about creating the variable symbol and checking its value with when. We can simplify our code as:

(defun consult-line-symbol-at-point ()
  (interactive)
    (consult-line (thing-at-point 'symbol)))

There is no greater pleasure than deleting code, is there?

Have I already told you are curious as a monkey?

I know what you are asking yourself: what is that 'symbol argument we passed to thing-at-point? Sounds like the kind of thing we want to detect.
And what about that part of the thing-at-point documentation that mentions other possible values?

THING should be a symbol specifying a type of syntactic entity. Possibilities include ‘symbol’, ‘list’, ‘sexp’, ‘defun’, ‘filename’, ‘existing-filename’, ‘url’, ‘email’, ‘uuid’, ‘word’, ‘sentence’, ‘whitespace’, ‘line’, ‘number’, ‘face’ and ‘page’.

And, finally, what about this intriguing invitation?

See the file ‘thingatpt.el’ for documentation on how to define a symbol as a valid THING.

I can not resist, can you? We have to try those other syntactic entity types, then we have to challenge ourselves defining our first custom thing-at-point.

You are repetitive, Arialdo!

Let me find out how many times I kept repeating “Try yourself” in his post.

Define:

(defun consult-line-sentence-at-point ()
  (interactive)
  (consult-line (thing-at-point 'sentence)))

Try yourself.

Notice how we are asking thing-at-point to detect a whole sentence, not just the single symbol at point.

Cool. I’m not that repetitive, after all.
Honestly, I have no idea how this would be useful. Maybe detecting sexps could come more in handy:

(defun consult-line-sexp-at-point ()
  (interactive)
  (consult-line (thing-at-point 'sexp)))

You got the idea.

This bears the question: how to define a new custom type of thing? We have no choice but to try.
The documentation is clear:

See the file ‘thingatpt.el’ for documentation on how to define a symbol as a valid THING.

So, let’s read the library code! M-x find-library RET thingatpt RET

Providers

Browsing around, you should stumble upon this variable:

(defvar thing-at-point-provider-alist nil
  "Alist of providers for returning a \"thing\" at point.
This variable can be set globally, or appended to buffer-locally
by modes, to provide functions that will return a \"thing\" at
point.  The first provider for the \"thing\" that returns a
non-nil value wins.

For instance, a major mode could say:

(setq-local thing-at-point-provider-alist
            (append thing-at-point-provider-alist
                    '((url . my-mode--url-at-point))))

to provide a way to get an `url' at point in that mode.  The
provider functions are called with no parameters at the point in
question.

Sounds like our culprit.

If you recap what we discovered until this point, you probably agree that this is a beautiful design trait of Emacs:

• isearch-forward-symbol-at-point focuses on searching things, seen as strings.
• thing-at-point focuses on returning that thing. It does that by delegating the job to a customizable collection of thing-providers.
• Each thing-provider implements some custom, arbitrary logic to detect a thing given the position in a buffer.

Indeed, we were able to easily create our consult-line-symbol-at-point because of this modular nature.

Time to challenge ourselves to define a thing that is not natively supported by isearch-forward-symbol-at-point.

A provider for literal strings, via Tree Sitter

Imagine we want to write consult-line-literal-string-at-point, to search for occurrences of the whole literal string at point, whatever buffer’s programming language defines as a literal string. You could use it to find duplicated strings in code:

What we should do is to define a new provider, ts-get-literal-string-at-point, a function whose goal is to return the literal string around the point. Notice the ts- in the name? That stands for Tree-sitter. Indeed, we will rely on whatever Tree-sitter grammar is active in the current buffer.

Once defined the provider, we will need to register it in the list of the global or the local thing-at-point providers. At that moment, we will get the chance to associate it to a symbol of our choice (we will use str_lit).

Once done that, thing-at-point will be able to detect literal strings at point (using the symbol we chose).

(Pause a minute to reflect again on the modular design of Emacs: Tree Sitter has been conceived in 2018, independently from Emacs and when Emacs was already 34. What we are really doing here is connecting some prehistoric Emacs machinery with Tree-sitter, the brand new kid on the block. There must be some deep beauty in the Emacs design if this works without any gimmick).

So, here’s the provider:

(defun ts-get-literal-string-at-point ()
  "Return the string node at point using Tree-sitter, or nil if none is found."
  (let ((node (treesit-node-at (point))))
    (when (equal (treesit-node-type node) "str_lit")
          (treesit-node-text node))))

It works like this:

• It asks Tree-sitter to get the node at point, with (treesit-node-at (point)).
• It checks the node type is of a literal string, (equal ... "str_lit"),
• then it returns its content (treesit-node-text node).
• Otherwise, it just returns nil.

And here is how we can (globally) register it:

(setq thing-at-point-provider-alist
            (append thing-at-point-provider-alist
                    '((str_lit . ts-get-literal-string-at-point))))

In a real case scenario, this would be done by a major mode, so most likely using setq-local instead of setq, inside a hook. If you are curious how major modes and hooks work in Emacs, you might give a read to Emacs: how to activate the functionality X for all files of type Y?.

Finally, let’s have a function connecting thing-at-point against str_lit with consult:

(defun consult-line-literal-string-at-point ()
  (interactive)
  (consult-line (thing-at-point 'str_lit)))


(global-set-key (kbd "M-s C-s .") #'consult-line-literal-string-at-point)

It really works! Try yourself.

(Ouch! I repeated myself)

References

• consult
• You have no idea how powerful isearch is!
• Protesilaos Stavrou
• thingatpt.el
• DWIM
• Bozhidar Batzov - Emacs Redux

For each of those themes, I will publish a little blog post to share the lesson learnt.

Today it’s the Imenu’s turn.

Imenu is an Emacs feature that produces menu items for accessing locations in buffers. Here’s the one for this very blog post:

If you are a programmer it is very likely that you have switched the visual menu off with:

(menu-bar-mode -1)

and that you prefer using the keyboard. Luckily for us keeb fans, Imenu happily works in the minibuffer too. Try yourself:

• Open a file for which you have a major mode installed.
• Run M-x imenu.

If you have consult.el installed, you would surely prefer M-x consult-imenu which offers narrowing and real-time preview.

Here’s how consult-imenu looks like while visiting imenu.el:

Why would an F# mode need Imenu in the first place? Because you would use it to jump to function definitions, variables and other structural elements of the code. Together with consult-outline this is one of the most convenient ways to reason about the structure of the code module you are visualizing.

So, our long term goal is to learn how to extract the structural elements of an F# buffer and to accordingly configure Imenu.

It turns out that:

• Imenu was initially conceived to work mainly using regular expressions.
• Tree-sitter offers a completely different way to extract structural elements from code, which saves major modes developers from dealing with regular expressions (because, as the saying goes “the plural of regex is regrets”).

We start our journey learning the classical way to use Imenu. We will cover the integration between Tree-sitter and Imenu in a second blog post.

Imenu

Let’s start with a simple case. Open the scratch buffer and past this content:

(defun greet ()
  (message "Hello, world!"))

(defun be-kind ()
  (message "OK, I will"))

Then run imenu: you should see those 2 functions listed. Selecting one, will get you to its definition. This is basically what Imenu does.

Besided what it does, we want to learn more how it actually works. So, let’s start delving into some details.

imenu–index-alist

At its core, Imenu just displays the content of the (private) variable imenu--index-alist. Try yourself. Again in the scratch buffer, paste and evaluate (C-x C-e) the following:

(setq imenu--index-alist
  `(("beginning" . 1)
    ("third char" . 3)
    ("fifth char" . 5)))

Then execute imenu again. No matter what the Imenu content was before, by overwriting imenu--index-alist you have just hijacked it. You can even display it in the visual menu bar:

(menu-bar-mode 1)
(imenu-add-to-menubar "My menu")
(menu-bar-open)

Select any of those 3 items to learn that the numbers in the imenu--index-alist alist are absolute buffer positions. Nice to know.

Notice that besides the 3 items you hard-codeed in imenu--index-alist there is a 4th item *Rescan*. Selecting it will restore the original Imenu items to the 2 greet and be-kind functions.

Another experiment you can do is:

• Evaluate again

(setq imenu--index-alist
  `(("beginning" . 1)
    ("third char" . 3)
    ("fifth char" . 5)))

• Run consult-imenu.

You will not see those 3 hard-coded items. Instead, it seems that consult-imenu invokes *Rescan* when it starts.

Wrapping up, our first intuition could be that:

• Imenu displays the content of imenu--index-alist.
• imenu--index-alist is populated by some function which is invoked when *Rescan* is selected.
• consult-imenu refreshes the content of imenu--index-alist when it starts.

Imenu scanning

So, when you visit a buffer imenu--index-alist does not start empty: it already contains some reasonable values. We can infer that some function must have automatically run, that inserted the proper items. If you reset imenu--index-alist:

(setq imenu--index-alist nil)

and you run again imenu, you can verify that greet and be-kind are back: Imenu has rescaned the buffer and detected them.

Which function in Emacs populates imenu--index-alist?
The answer is: it’s up to you. Imenu does not care, you can plug in any function you like, as long as it returns an alist to use.

If you poke into the Imenu code, you will find this snippet as part of the imenu--make-index-alist function:

(setq imenu--index-alist
      ...
      ...
      (funcall imenu-create-index-function))

This! We spotted the code where imenu--index-alist is populated!
So, Imenu expects that someone or something provides it with a function to scan the buffer and to return an index of menu item. This something is normally the major mode of the buffer.

To challenge this, let’s activate a major mode and let’s discover which function imenu-create-index-function targets:

• Visit a Markdown file and run M-x describe-variable RET imenu-create-index-function. It reveals that the function used by Imenu is markdown-imenu-create-nested-index. Apparently, an implementation very specific to Markdown.

• If you visit a C file, instead, the value of imenu-create-index-function is imenu-default-create-index-function, which sounds more generic.

• The same happens for a lot of other major modes.

So, interestingly: some major modes provide a custom way for scanning their buffers and indexing the Imenu; others rely on a default function, provided by Imenu itself.

This provides a hint about what a major mode is: a function providing a collection of variable values. (If you want to know a bit more about major modes, you might like Emacs: how to activate the functionality X for all files of type Y?).

It also gives us some suggestions:

• Should not we find a better method, when developing the Tree-sitter mode for F# we can always develop a completely custom Imenu index function, and we can plug it into Imenu through imenu-create-index-function.

• This imenu-default-create-index-function default implementation sounds inspiring. It’s used by a variety of independent major modes, therefore it must offer some flexible options to customize it. Let’s investigate.

So, our next 2 goals to challenge our discovery can be:

• We write a completely custom index function, and we plug it into Imenu.
• Then, we study what the default Imenu index function offers.

A custom imenu-create-index-function

Open the scratch buffer (M-x scratch-buffer RET) and evaluate:

(defun my-index-function ()
  '(("one" . 1)
    ("two" . 2)
    ("three" . 3)))

(setq imenu-create-index-function #'my-index-function)

Then reopen imenu and rescan the index (or just use consult-imenu which we learnt always rescans the index).
Yes! We see the 3 items.

Of course, our function needn’t return hard-coded values; it can contain any custom logic. It could generate an index in a cycle:

(defun my-index-function ()
  (cl-loop for i from 1 to 50
           collect (cons (format "Item %d" i) i)))

or it could actually scan the buffer searching for (defun FUNCNAME patterns:

(defun my-index-function ()
  (save-excursion
    (goto-char (point-min))
    (cl-loop while (not (eobp))
             for item = (when (looking-at "(defun\\s-+\\(\\S-+\\)")
                          `(,(match-string 1) . ,(point)))
             if item collect item
             do (forward-line 1))))

Cool. This gives up hope to integrate Imenu with Tree-sitter. We could:

• ask Tree-sitter to analyze the buffer and to return back the syntax tree for our F# file.
• use some custom logic to scan the syntax tree — instead of the chars in the buffer — building one item for each of the F# structural elements we want to index.

Way better than working with regular expressions. We will get to it.

The default imenu index function

Time to see what’s inside the default index function, provided out-of-the-box by imenu.el.

• Open the function documentation with M-x describe-function RET imenu-default-create-index-function RET.
• Access its code with M-x help-view-source RET (or just type s).

This reveals this implementation (for some reason, with a wrong indentation), which is not too complicated:

(defun imenu-default-create-index-function ()
  "Default function to create an index alist of the current buffer.

The most general method is to move point to end of buffer, then repeatedly call
`imenu-prev-index-position-function' and `imenu-extract-index-name-function'.
All the results returned by the latter are gathered into an index alist.
This method is used if those two variables are non-nil.

The alternate method, which is the one most often used, is to call
`imenu--generic-function' with `imenu-generic-expression' as argument."
  ;; These should really be done by setting imenu-create-index-function
  ;; in these major modes.  But save that change for later.
  (cond ((and imenu-prev-index-position-function
	      imenu-extract-index-name-function)
	 (let ((index-alist '()) (pos (point-max))
               (start (float-time))
	       name)
	   (goto-char pos)
	   ;; Search for the function
	   (while (and (funcall imenu-prev-index-position-function)
                       ;; Don't use an excessive amount of time.
                       (< (- (float-time) start) imenu-max-index-time))
             (unless (< (point) pos)
               (error "Infinite loop at %s:%d: imenu-prev-index-position-function does not move point" (buffer-name) pos))
             (setq pos (point))
	     (save-excursion
	       (setq name (funcall imenu-extract-index-name-function)))
	     (and (stringp name)
 		  ;; [ydi] Updated for imenu-use-markers.
		  (push (cons name
                              (if imenu-use-markers (point-marker) (point)))
			index-alist)))
	   index-alist))
	;; Use generic expression if possible.
	((and imenu-generic-expression)
	 (imenu--generic-function imenu-generic-expression))
	(t
         (imenu-unavailable-error "This buffer cannot use `imenu-default-create-index-function'"))))

Reduced to its core, it looks like the following:

(defun imenu-default-create-index-function ()
  (cond ((and imenu-prev-index-position-function
	          imenu-extract-index-name-function)
	          
              ;; [code omitted]
              ;; index using those 2 functions

	    ((and imenu-generic-expression)
	     (imenu--generic-function imenu-generic-expression))
	    (t
         (imenu-unavailable-error "This buffer cannot use `imenu-default-create-index-function'"))))

Basically:

1. It checks if the major mode provides 2 specific functions for indexing (one for positioning, one for extracting). If so, it uses them. We will see how this work in a minute.
2. Otherwise, it checks if the major mode provides a set of regular expressions (in the variable imenu-generic-expression). If so, it uses them, delegating the work to the function imenu--generic-function.
3. Otherwise, it raises an error.

Let’s start from the case 2, which is by far and large the most popular — but not necessarily the most powerful — usage of Imenu.

imenu-generic-expression

So, this is the case in which imenu--generic-function is passed the regular expressions defined in imenu-generic-expression. I’m curious to inspect both. Notice that imenu--generic-function is private, so it’s not meant to be modified. imenu-generic-expression, instead, is meant to be customized.

If you visit the scratch buffer and you inspect the value of imenu-generic-expression you will find a collection of very well crafted regular expressions:

  ((nil "^\\s-*(\\(cl-def\\(?:generic\\|ine-compiler-macro\\|m\\(?:acro\\|ethod\\)\\|subst\\|un\\)\\|def\\(?:advice\\|generic\\|ine-\\(?:advice\\|compil\\(?:ation-mode\\|er-macro\\)\\|derived-mode\\|g\\(?:\\(?:eneric\\|lobal\\(?:\\(?:ized\\)?-minor\\)\\)-mode\\)\\|inline\\|m\\(?:ethod-combination\\|inor-mode\\|odify-macro\\)\\|s\\(?:etf-expander\\|keleton\\)\\)\\|m\\(?:acro\\|ethod\\)\\|s\\(?:etf\\|ubst\\)\\|un\\*?\\)\\|ert-deftest\\)\\s-+\\(\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)" 2)
   (nil "^\\s-*(\\(def\\(?:\\(?:ine-obsolete-function-\\)?alias\\)\\)\\s-+'\\(\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)" 2)
   ("Variables" "^\\s-*(\\(def\\(?:c\\(?:onst\\(?:ant\\)?\\|ustom\\)\\|ine-symbol-macro\\|parameter\\)\\)\\s-+\\(\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)" 2)
   ("Variables" "^\\s-*(defvar\\(?:-local\\)?\\s-+\\(\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)[[:space:]
  ]+[^)]" 1)
   ("Types" "^\\s-*(\\(cl-def\\(?:struct\\|type\\)\\|def\\(?:class\\|face\\|group\\|ine-\\(?:condition\\|error\\|widget\\)\\|package\\|struct\\|t\\(?:\\(?:hem\\|yp\\)e\\)\\)\\)\\s-+'?\\(\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)" 2))

I don’t even dare an attempt to understand. I’m just too grateful to whoever took the time to write them. Chapeau!
I am more interested in answering the questions:

• In which file of Emacs is this value defined?
• How is this used by Imenu? What does imenu--generic-function do with this collecgtion?

To answer the first question, we can follow our intuition: it’s up to the major mode to configure Imenu. So, we can:

• Inspect which major mode is used by the current buffer, with M-x describe-mode.
• Navigate to its source code, from the file link mentioned in the documentation.
• Search for a reference to (setq-local imenu-generic-expression.
• If no result is found, we navigate to the parent major mode, following the expressions like (define-derived-mode current-mode parent-mode hitting M-. on parent-mode (again, read the post about hooks to learn more).

Doing this exercise starting from the scratch buffer takes us to the file lisp-mode.el where a giant variable lisp-imenu-generic-expression is defined:

(defvar lisp-imenu-generic-expression
  (list
   (list nil
	 (purecopy (concat "^\\s-*("
			   (regexp-opt
			    '("defun" "defmacro"
                              ;; Elisp.
                              "defun*" "defsubst" "define-inline"
			      "define-advice" "defadvice" "define-skeleton"
			      "define-compilation-mode" "define-minor-mode"
			      "define-global-minor-mode"
			      "define-globalized-minor-mode"
  ...
  ;; [code omitted]

Mystery solved.

Now, we just need to understand how this collection of regular expressions is used. If you remember, the default implementation of the index function imenu-default-create-index-function mentioned:

	    ((and imenu-generic-expression)
	     (imenu--generic-function imenu-generic-expression))

Let’s give a look to the code of imenu--generic-function:

• Do a M-x describe-function RET imenu--generic-function RET.
• Then M-x help-view-source (or just hit s).

Phew! This one is way more complex than the one before. What is relevant for us is:

• It maps over the elements of imenu-generic-expression, the collection of regexen. We expected that.

• For each regex, it starts with a (goto-char (point-max)). This means, it scans the buffer backwards, allegedly “for convenience of adding items in order”.

• Then, it collectes each matching candidate.

OK, not too alien. The reason why the function is so long is because it also takes into account error handling, duplicates, execution time, edge cases etc. We can skate over it.

Instead, let’s put it into play. Type in the scratch buffer:

- buy apples
- buy bananas
- sell ananas

(setq-local imenu-create-index-function #'imenu-default-create-index-function)

(setq imenu-generic-expression
  '(("to buy" "^- buy \\(.*\\)$" 1)
    ("to sell" "^- sell \\(.*\\)$" 1)))

Each element in the imenu-generic-expression shall follow this pattern:

(MENU-TITLE REGEXP INDEX)

where

• MENU-TITLE is the title of the submenu to host the item.
• REGEXP is the regular expression to capture the candidate. The regex may contain a group.
• INDEX is the group number to use as the item to be displayed. 0 means the whole match.

There are other options, which you can consult visiting M-x describe-variable RET imenu-generic-expression RET.

We will not deepen this approach any further: we know that the F# major mode will rely on the syntax tree provided by Tree-sitter, not on regular expressions.

Let’s see, instead, the other option offered by the default index function, imenu-default-create-index-function.

Find position and extract

Remember that we summarized imenu-default-create-index-function as:

(defun imenu-default-create-index-function ()
  (cond ((and imenu-prev-index-position-function
	          imenu-extract-index-name-function)
	          
              ;; [code omitted]
              ;; index using those 2 functions

	    ((and imenu-generic-expression)
	     (imenu--generic-function imenu-generic-expression))
	    (t
         (imenu-unavailable-error "This buffer cannot use `imenu-default-create-index-function'"))))

and that we said that the first branch of the condition:

• uses a combination of 2 functions, imenu-prev-index-position-function and imenu-extract-index-name-function. The former finds the position of the next candidate; the latter builds the index item for that candidate.

It’s time to say more about what the omitted code does. It was:

(let ((index-alist '()) (pos (point-max))
      (start (float-time))
	  name)
  (goto-char pos)
  ;; Search for the function
  (while (and (funcall imenu-prev-index-position-function)
              ;; Don't use an excessive amount of time.
              (< (- (float-time) start) imenu-max-index-time))
    (unless (< (point) pos)
      (error "Infinite loop at %s:%d: imenu-prev-index-position-function does not move point" (buffer-name) pos))
    (setq pos (point))
	(save-excursion
	  (setq name (funcall imenu-extract-index-name-function)))
	(and (stringp name)
 		 ;; [ydi] Updated for imenu-use-markers.
		 (push (cons name
                     (if imenu-use-markers (point-marker) (point)))
			   index-alist)))
  index-alist)

Indulge me while to simplify it in this naïve way:

(let ((index-alist '())
      (pos (point-max))
      (name nil))
  
  (goto-char pos)

  (while (funcall imenu-prev-index-position-function)
    (setq pos (point))
	(save-excursion
	  (setq name (funcall imenu-extract-index-name-function)))
	(and (stringp name)
		 (push (cons name (point))
			   index-alist)))
  index-alist)

We can interpret it as:

• It starts from the end. As before, then, it scans the buffer backward.
• As long as it finds candidates, it runs a cycle.
• For each step of the cycle it makes use of 2 custom functions:
  • imenu-prev-index-position-function, which has the goal of finding a candidate and moving the point there.
  • imenu-extract-index-name-function which, given the found candidate, has the goal to calculate the menu item.
• When it’s done, it returns the collection of items.

OK, fair enough. Let’s give it a shot with a silly example which creates a menu item for each line in the buffer:

(defun find-all-lines ()
  "Move point to previous lines, returning t each time.
Returns nil when reaching buffer start, signaling completion."
  (if (= (point) (point-min))
      nil
    (forward-line -1)
    t))

(defun get-line-number ()
  "Return the current line number as the index name."
  (format "Line %d" (line-number-at-pos)))

(setq-local imenu-prev-index-position-function #'find-all-lines)
(setq-local imenu-extract-index-name-function #'get-line-number)

• find-all-lines always indicates that the beginning of each line is a good candidate.
• Given the start of line position, get-line-number just returns a string with the line number as the menu item name to display.

Run it with imenu. Nice, it works. vvv

What about Tree-sitter?

So, which approach shall we take to integrate imenu with Tree-sitter? Sure enough, we won’t rely on imenu-generic-expression and its regular expressions. Shall we play with the couple of position+extract functions? Or will we have to develop a whole brand new custom index function to replace the standard imenu-default-create-index-function?

It turns out treesit, the built-in Tree-sitter package, has some delightful surprises for us. But this is the topic for the next episode.

Hang tight! Happy emacsing.

(Thank you Prot for guiding me in this journey)

References

• consult.el
• Emacs: how to activate the functionality X for all files of type Y?
• Protesilaos Stavrou

From Matteo Vaccari:

There’s no easy way to evolve the hardcoded values into general code.

• Alistair Cockburn said that this problem could be better solved by more mathematically-oriented thinking upfront about how many spaces should be generated where
• Emily Bache said that “recycling tests” is a valid way to iterate towards a difficult problem
• Other solutions came from Ron Jeffries, George Dinwiddie, Jon Jagger, again Ron Jeffries

Francesco Cirillo

Francesco Cirillo - The Diamond Kata Anti-IF Workshop

• Payed workshop.
• It promises that TDD and Anti-IF Programming “let design abstractions emerge […] in a way that prioritizes intentional design over algorithmic complexity, allowing for an adaptable and emergent solution.”
• It states: “No Thinking Ahead Required: Unlike conventional wisdom, this workshop will show you how to solve the Diamond Kata using TDD without the need for anticipatory thinking […] showing how abstractions emerge naturally through TDD.”

Seb Rose

Seb Rose - Recycling tests in TDD

• Seb uses the Print Diamond Kata in his trainings.
• He relies on the classical test progression (first the a case, then the b case, etc).
• He shows how some implementations are hardcoded (applying Fake It Until You Make It).
• He mentions the typical struggle with Example-Based TDD: “The code is now screaming for us to refactor it, but to keep all the tests passing most people try to solve the entire problem at once. That’s hard, because we’ll need to cope with multiple lines, varying indentation, and repeated characters with a varying number of spaces between them.”
• He suggests a second approach: “we start by decomposing the diamond problem into smaller constituent parts. This time I chose to write the following test:”
• He mentions the “the top half of the diamond”, without making it explicit it the code.
• He claims that “by recycling the second test multiple times, each time modifying the expected output to become closer to what we actual want, we’ve allowed ourselves to be guided gradually to a solution.”

Here are the used tests:

[TestFixture]
public class DiamondTest
{
    [Test]
    public void A_should_generate_a_single_character()
    {
        Assert.AreEqual("A", Diamond.Create('A'));
    }
 
    [Test]
    public void B_should_generate_the_smallest_diamond()
    {
        Assert.AreEqual(" A\nB B\n A", Diamond.Create('B'));
    }
 
    [Test]
    public void C_should_generate_the_next_diamond()
    {
        Assert.AreEqual("  A\n B B\nC   C\n B B\n  A", Diamond.Create('C'));
    }
}

and:

using NUnit.Framework;
 
[TestFixture]
public class DiamondTest
{
    [Test]
    public void A_should_give_single_character()
    {
        Assert.AreEqual("A\n", Diamond.Create('A'));
    }
 
    [Test]
    public void B_should_be_a_diamond()
    {
        Assert.AreEqual(" A\nB B\n A\n", Diamond.Create('B'));
        Assert.AreEqual(" A\n"  +
                        "B B\n" + 
                        " A\n", Diamond.Create('B'));
    }
 
    [Test]
    public void C_should_be_a_bigger_diamond()
    {
        string diamond = "  A\n"   + 
                         " B B\n"  +
                         "C   C\n" +
                         " B B\n"  +
                         "  A\n";
 
        Assert.AreEqual(diamond, Diamond.Create('C'));
    }
 
    [Test]
    public void D_should_be_an_even_bigger_diamond()
    {
        string diamond = "   A\n"   + 
                         "  B B\n"  +
                         " C   C\n" +
                         "D     D\n" +
                         " C   C\n" +
                         "  B B\n"  +
                         "   A\n";
 
        Assert.AreEqual(diamond, Diamond.Create('D'));
    }
}

Seb Rose - Diamond recycling (and painting yourself into a corner)

Alistair Cockburn

Alistair Cockburn - Thinking before programming

• “[…] a few professors had the idea that they might teach programmers to think a bit before hitting the keycaps. […]. They showed how to create programs (of a certain category) without error, by thinking about the properties of the problem and deriving the program as a small exercise in simple logic.”
• Accorging to Alistair XP and TDD “when practiced well, involve a different kind of thinking – not about the problem, but about the code. TDD has become a pablum to avoid thinking.”
• His approach is to combine the Dijkstra-Gries approach (thinking ahead) with TDD (“modern fine-grained incremental development”).
• In his premises, he seems to describe PBT: “What happens if we think about the problem and look for patterns and properties before starting, and then instead of typing the whole thing in at once, we “grow” the program using fine-grained incremental development and TDD?”
• Mentions the struggles about incremental development with TDD:

Step 1: Go out and buy really good sneakers.

Step 2: Do warmup exercises.

Step 3: Jump really really really high, like, to the moon. “

• He promises to reflect on the problem, not on the solution.
• He identifies some properties:
  • The diamond is 2x+1 per 2x+1.
  • For each row, there are 2x-1 spacers.
  • There is a formula for leading and trailing spaces.
• Yet, he implements those ideas following the classical steps (case 0, then 1 etc).
• The formulas he conceived are the implementation. So, the test did not guide the design, to the point that he writes the implementation before the first test.
• Interestingly, the name of his tests have the typical “ForAll” form of a property test:

def test_02_makes_and_fills_tray_of_any_size

• He claims that he did not type in all those tests at once, because “to do so is not only against the rules of TDD […]), but is actually dangerous”. I don’t agree with this: in fact, his implementation (the fomula he conceived) was already complete. The additional tests did not help evolving the formula.

• Even more interestingly, despite the initial good intuitions, his tests are a description of the solution, not the problem:

:test_01_fills_first_and_last_slot_in_tray
:test_02_makes_and_fills_tray_of_any_size
:test_03_makes_bumper_for_row_in_diamond
:test_04_glues_bumpers_to_tray_making_complete_row_in_diamond
:test_05_makes_diamond_the_right_size
:test_06_puts_a_row_in_both_upper_and_lower_half_of_diamond
:test_07_digit_diamond_fills_correctly

• Notice the use of expressions like “the right size” and “fills correctly”.

Jon Jagger

Jon Jagger - sliming and refactoring and deliberate duplication

• Classic progression: a case, b case, c case, all hardcoded.
• He claims “As the tests get more specific, the code should get more generic. I have three specific tests, but the code is equally specific.”
• Beautful recursive implementation. Really brilliant.
• Yet, not induced by a test. Definitely, not an emergent design.

George Dinwiddie

George Dinwiddie - Another Approach to the Diamond Kata

• He also starts from the trivial degenerate diamond (with a hardcoded implementation).
• “The next step is obviously to implement the B case”. Why “obviously”?
• He refuses to hardcode B. His implementation, though, is not the minumum code strictly necessary to make the test pass, but already a generalization.

Emily Bache

Emily Bache - Iterative and Incremental TDD with the Diamond Kata

• She believes that “recycling tests” is a valid way to iterate towards a difficult problem
• Nothing new in her proposed repo, though: tests are incrementally going from A to D.

Matteo Vaccari

Matteo Vaccari - The Diamond Kata Revisited

• He acknowledges that “the problem statement is simple, but solving it with TDD is not straightforward” because “there’s no easy way to evolve the hardcoded values into general code”.
• Rather than trying to arrive at a “single algorithm” that will solve the problem in one shot, he suggests a compositional approach.
• He analyzes some traits of the problem (not the solution!), decomposing the problem into sub-problems and in fact capturing some properties, such as the symmetry of quadrants.

Géza Mihala

Géza Mihala - Diamond Kata

• Similarly to Matteo Vaccari, he plays with decomposing the problem (using functional decomposition).
• He also identifies 2 properties: horizontal and vertical symmetry.
• The 4 properties that define the problem are:
  • a sequence of letters from ’A’ up to a given letter
  • arranged diagonally in a square, padded with blanks
  • mirrored to the left, not repeating the first column
  • mirrored down, not repeating the bottom row
• The only missing step in his post is: he does not use TDD at all.

References

• Resolved with Example-Based Testing
  • Francesco Cirillo - The Diamond Kata Anti-IF Workshop
  • Seb Rose - Recycling tests in TDD
  • Seb Rose - Diamond recycling (and painting yourself into a corner)
  • Alistair Cockburn - Thinking before programming
  • Jon Jagger - sliming and refactoring and deliberate duplication
  • George Dinwiddie - Another Approach to the Diamond Kata
  • Emily Bache - Iterative and Incremental TDD with the Diamond Kata
  • Emily Bache - DiamondKata repository
  • Matteo Vaccari - The Diamond Kata Revisited
  • Ron Jeffries - TDD on the Diamond Problem
  • Ron Jeffries - More about Diamond
  • Géza Mihala - Diamond Kata
• Resolved with Property-Based Testing
  • Nat Pryce - Diamond Kata – TDD with only Property-Based Tests
  • Mark Seemann - Diamond kata with FsCheck
  • Uldis Sturms - property based testing in javascript - diamond kata

Functional Programming For The Rest Os Us

Interested in FP? Be the first to be notified when new introductory articles on the topic are published.

Sirens

Test-Driven Development comes with 2 very seductive and ambitious promises:

• Development is really driven by tests
  Not only do tests verify correctness; they also help determine what code to write, in what order and how to structure it. In other words, TDD helps shaping the software and determining its development process.

• There is an Emergent Design
  The system’s architecture and design incrementally take shape, rather than being fully planned upfront.

It’s the alluring melody of a siren singing “Just write the tests and abandon yourself to them: correctness and design will naturally follow, incrementally”.

Reality

Of course, this is too good to be exactly true. As Kent Beck writes in TDD isn’t design:

nobody with any sense says TDD replaces the need for design

In fact, Emergent Design, in this context, has more the meaning discussed in the C2 Wiki article Can An Architecture Emerge:

The term ‘emergent’ has a specific meaning with respect to chaos theory. A structure, in the past, was always assumed to be imposed by some direct structuring force. This is known as skeletal or imposed structure. This would be an architecture that started perfect and stayed that way.

In this context, emergent means emerging from the system rather than being imposed on the system. Take the example of a river’s course. On the one hand the course of the river structures the flow of water. Yet the flow of water is also responsible for defining the course of the river.

The question, then, is: can TDD lead an implementation without requiring a design upfront?

Enter the Print Diamond Kata.

Print Diamond Kata

The Print Diamond Kata (presented by Seb Rose in Recycling tests in TDD) is a very interesting exercise in that, if you approach it with TDD by-the-book, it’s likely that you will struggle to be guided by tests and to experience an emerging design. Instead, you will be required to do some design upfront and to conceive a solution from a light-bulb moment.
Tests will help challenging your solution, not conceiving it. In a sense, the Print Diamond Kata demonstrates that Emergent Design is not always possible.

The problem is very simple: given a letter, you are supposed to return a string with a diamond starting with A with the supplied letter at the widest point.
For example: printDiamond('C') returns:

  A
 B B
C   C
 B B
  A

The conventional way to approach it in TDD is to first get the A case working — hardcoding the implementation — then to follow with B and then with C etc.

As Alistair Cockburn commented in Thinking before programming:

What struck me in Seb’s recounting of people’s approach to the problem is they first get the ‘A’ case working:

A

Then they shuffle around a bit to get the ‘B’ case working:

. A .
B . B
. A .

And then all the work that they did so far is of no use to them whatsoever when they hit ‘C’, because that’s where all the complexity is. In essence, they have to solve the entire problem, and throw away masses of code, when they hit ‘C’.

It reminds me of the story of incremental development and jumping to the moon:

Step 1: Go out and buy really good sneakers.
Step 2: Do warmup exercises. 
Step 3: Jump really really really high, like, to the moon. 

This is not the interesting form of incremental development. Can we do better?

To me, this sounds like Clem Fandango’s “How To Draw A Owl” meme:

Other developers way more experienced than me (see Review of other devs’ approaches) have observed more or less the same.

This begs the question:

Is it possible to apply TDD in a way that tests really drive the implementation?

We suck at TDD

I argue that the answer is a resounding “YES!”.
And that the reason why many have experienced the opposite is because of the way we classically apply TDD has some fundamental fallacies.

Once we revisit a bunch of the canonical assumptions we never dared to question, it is indeed possible to have emergent design.
That’s the goal of this article.

Along the way, I aim to debunk some myths:

• 100% code coverage and all green tests mean code correctness and production readiness.
• Understading how much testing is enough is a matter of skills and experience, not a science.
• Tests in TDD are requirements expressed with code.

Challenge

Before moving to the second part of this post, I invite you to try yourself solving the Print Diamond Kata.

Notice: the goal is not just to solve it. The goal is to challenge that design can be driven by tests. Therefore, I stress you to add the following constraint:

1. Tests should neither depend on nor impose an implementation.

In other words, make sure not to take design decisions while writing tests.

This resonates with the above-mentioned TDD isn’t design, that makes sense to quote more extensively:

Nobody with any sense says TDD replaces the need for design. The key question is when.

TDD offers:

• Immediate feedback for interface design decisions. You still have to make those decisions. [..]

• Separation between interface design decisions & implementation design decisions.

It also makes sense to have a second constraint:

2. Each test gets to a deliverable result providing a business value increment

Make your game. While coding, pay attention if either

• you can build on previous implentations
• or if you find youself in a How to Draw an Owl situation.

When you are happy with your findings, let’s meet on the second part of this post.

(Stay tuned…)

References

• Kent Beck - TDD isn’t design
• C2 - Emergent Design
• C2 - Can An Architecture Emerge
• Alistair Cockburn - Thinking before programming
• Seb Rose - Recycling tests in TDD
• Clem Fandango - How To Draw A Owl

Comments

GitHub Discussions

Functional Programming For The Rest Os Us

Interested in FP? Be the first to be notified when new introductory articles on the topic are published.

Investigating on this topic helped me realize that I have always used xUnit theories incorrectly. And that xUnit loves the idea of Property-Based Testing.

I’ve been tought to use xUnit theories as a mere trick to repeat a test with different inputs, without incurring in duplication:

[Theory]
[InlineData(2, true)]
[InlineData(4, true)]
[InlineData(3, false)]
[InlineData(7, false)]
void check_if_odd_or_even(int number, bool expectedResult)
{
    var actualResult = IsEven(number);
    
    Assert.Equal(expectedResult, actualResult);
}

Like in this dummy example, I’ve always been used have an expectedResult parameter.

Theory’s purpose

I then realized that this is not the style used in the xUnit manual. The chapter Write your first theory never mentions any argument like my expectedResult. Instead, it uses this style:

[Theory]
[InlineData(3)]
[InlineData(5)]
[InlineData(6)]
public void MyFirstTheory(int value)
{
    Assert.True(IsOdd(value));
}

bool IsOdd(int value)
{
    return value % 2 == 1;
}

It claims that a property (“to be odd”) holds for a set of inputs. Indeed, it was introduced by this sentence:

You may have wondered why your first unit tests use an attribute named [Fact] rather than one with a more traditional name like Test. xUnit.net includes support for two different major types of unit tests: facts and theories. When describing the difference between facts and theories, we like to say:

• Facts are tests which are always true. They test invariant conditions.

• Theories are tests which are only true for a particular set of data.

Under this light, Theories remind me of Properties in Property-Based Testing.
Ideally, xUnit designers could have decided to let theories run against a predicate, and let them assume success if the predicate is true, failure otherwise:

[Theory]
[InlineData(3)]
[InlineData(5)]
[InlineData(6)]
public void MyFirstTheory(int value) =>
    IsOdd(value));

Add a way to generate inputs and you would have Property-Based Testing.

Further details

Am I going too far?
I thought so. Then I stubled upon a comment to the xUnit Issue #2822 - Why the word “[Theory]” as opposed to something like “[MultiFact]”?:

Ruben Bartelink observed in his comment:

It’s just naming; The names are accentuating the role it plays vs a lower level description of what it technically does/is. […] you can argue it puts authors in the right frame of mind. Or you can argue that it’s cryptic. But xUnit is definitely designed with lots of though put into nuanced design aspects with a view to nudging one towards writing good tests, and shaping good systems.

So, this is indeed very intentional and about promoting a specific testing style. Going ahead (bold is mine):

In xUnit
A Fact is an individual relevant assertion. If you have 3 of them, they may or may not mean anything in aggregate.
A Theory is a set of facts that together prove an overall concept
In Property Based Testing, a Property is called that because that’s its role - It’s not a TestWithMultipleInvocationsBasedOnArbitraryData. We are specifting a property of the system under test.

So, I’m not alone observing a similarity with PBT.

Brad Wilson, xUnit’s caretaker, commented quoting the page Why Did we Build xUnit 1.0?:

The definition of how to run a test method can be extended. There are two example of this: the first, in xunitext.dll, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see https://homes.cs.washington.edu/~mernst/pubs/testing-theories-tr002-abstract.html.

Down the rabbit hole, here is an excerpt of the paper’s abstract (bold is mine):

Traditional test suites verify a few well-picked scenarios or example inputs. […] We propose theory-based testing as an adjunct to example-based testing.
A theory generalizes a (possibly infinite) set of example-based tests. A theory is an assertion that should be true for any data, and it can be exercised by human-chosen data or by automatic data generation.

Here are some code examples from the paper. In the first one:

public void additionIsInverseOfSubtraction(int x, int y) {
    assertEquals(x, (x + y) - y);
}

the assertion should hold over the entire valid input space. In the following:

@Theory
void equalObjectsEqualHashes(Object a, Object b) {
  assumeTrue(a.equals(b));
  assertTrue(a.hashCode() == b.hashCode());
}

the assumption expressed by assumeTrue further restricts which inputs are valid for evaluating the assertion.

In one of the last chapters, the paper introduces the idea of a Theory Explorer, an input generator based on Agitator, used to find inputs that will cause the theory to fail.

This Theory-Based Testing is really akin to Property-Based Testing. The paper even mentions the classical test on reversing a string and Haskell’s QuickCheck, which predates the paper by 9 years. No mention to PBT is present, though.

My conclusion

xUnit is not a Property-Based Testing library, and it’s unlikely to evolve into one anytime soon.
In Property-Based Testing for The Rest of Us I argue that the value of PBT lies not primarily in the random generation of inputs; instead, it comes from how it encourages developers to gain a deeper and reasoned undertanding of requirements, compared to example-based testing.

My take away is: it is worth to use theories the way they have been originally conceived and designed. I bet that the mere attempt not to use any expectedResult argument is very likely to bring to more profound and effective tests.

References

• Why Did we Build xUnit 1.0?
• xUnit Issue #2822 - Why the word “[Theory]” as opposed to something like “[MultiFact]”?
• xUnit manual - Write your first theory
• Theories in practice: Easy-to-write specifications that catch bugs
• QuickCheck
• Property-Based Testing for The Rest of Us

It will give us the chance to touch on:

• Face attributes.
• Asking the user to choose interactively from a list of options.
• The powerful consult--read.
• Lisp’s Dynamic and Lexical Binding.
• Few other little topics here and there.

Faces

In Emacs, the “face” is the visual styling attribute that can be applied to text: faces define how text appears on the screen, including the font, the colors, and the weight. And the height, the attribute we want to control in this little tutorial.

There is a notion of “default face” in Emacs: it’s basically the parent from which all the other faces inherit their attributes, unless they specify a specialization. Changing an attribute of the default face will propagate the change everywhere in Emacs.

Try yourself. Open your scratch buffer (M-x scratch-buffer RET) and write:

(face-attribute 'default :height)

Evaluate it: move right after the last ) and hit C-j (or M-x eval-print-last-sexp RET). This reveals the current height of the default face. In my case, it prints 180.

Good. Now, try to change it:

(set-face-attribute 'default nil :height 100)

Evaluate it: again, move right after the last ) and hit C-x C-e. As expected, any text in Emacs will inherit the new defined height.

Cool, you learnt how to programmatically control the global text height.
There are other attributes you can control, such as the foreground and background colors, the weight and the font. You can learn more about this topic from the chapter Faces in the Emacs Manual.
But let’s focus on height.

Controlling the current buffer faces

set-face-attribute operates globally. If you want to control the faces of the current buffer only, you need to play with remappings. I will skate over this topic, but just quickly:

(setq cookie
      (face-remap-add-relative 'default :height 280))

changes the height of the default face in the current buffer only. Indeed, it creates a remap of the face, that is a way to temporarily redirecting how a face is displayed.

It returns a cookie, that we save in the cookie variable. When you want to remove the remap, you have to provide that cookie to face-remap-remove-relative:

(face-remap-remove-relative cookie)

We won’t need to worry about this: our squint package will operate globally.

Controlling the height interactively

There are 2 ways for interactively modifying the size of characters on the screen:

They are straightforward to use: activate them, then follow the instructions on the screen, hitting:

That easy.

Grab your keeb

Enough with theory. Our goal is to create a new command, squint, offering a list of height presets to choose from interactively, with narrowing and realtime preview.

We will proceed incrementally, each time improving the result.

Step 1: set height from key/value pairs

The basic idea is to have a collection of heights in an alist (see Association Lists), that is a list of key-value pairs, such as:

(setq squint-heights
      '(("office" . 100)
        ("laptop" . 180)
        ("programming" . 200)
        ("presentation" . 300)))

and to set the corresponding height through its label, with:

(squint "presentation")
(squint "laptop")

It does not sound as a very challenging problem, does it? We need to define a function squint that:

• Given a label, retrieves the corresponding height from squint-heights.
• Once obtained the height, uses the set-face-attribute we saw before to adjust the font height.

Retrieving a value given a key from an alist is simply done with alist-get:

(alist-get "programming" squint-heights nil nil #'equal)

Here’s the signature:

(alist-get 
    KEY     ;; the key to search for
    ALIST   ;; the alist to search in
    DEFAULT ;; the value to return in case KEY is not found
    REMOVE  ;; whether the pair should be removed in case of success
    TESTFN) ;; the equality function to use for comparing keys

TESTFN defaults to eq, which is a bit unfortunate: eq does not return true if the two args are equal, but only if they are the very same Lisp object. In other words,

(eq "ciao" "ciao")

is false: although the 2 strings are equal, they are not the same object.
This:

(setq ciao "ciao")
(eq ciao ciao)

returns true.
Instead of eq we’d better use equal which, more conveniently, returns true if two objects have similar structure and contents.
Since TESTFN is a positional parameter, we are also forced to provide 2 nil values for the other parameters DEFAULT and REMOVE.

That’s all you need to know to finally define:

(setq squint-heights
      '(("office" . 100)
        ("laptop" . 180)
        ("programming" . 200)
        ("presentation" . 300)))

(defun squint (label)
  "Adjust font height based on predefined settings.
LABEL is a string that corresponds to a key in `squint-heights'."
  (let ((height (alist-get label squint-heights nil nil #'equal)))
    (set-face-attribute 'default nil :height height)))

Evaluate the code above (C-x C-e after each expression), then run:

(squint "presentation")

Such a large font! Sweet!

Step 2: symbols instead of strings

The quirk about using equal can be avoided using symbols (such as 'office and 'laptop) instead of strings such as "office" and "laptop".

With strings one must be careful when to use eq and when to use equal:

With symbols, equality is much easier:

Let’s try redefining squint-heights using symbols instead of strings for its keys:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 200)
        (presentation . 300)))

The usage of alist-get is a bit shorter:

(alist-get 'programming squint-heights)

Notice the use of 'programming instead of "programming".
This leads to the following simplified implementation:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(defun squint (label)
  "Adjust font height based on predefined settings.
LABEL is a symbol that corresponds to a key in `squint-heights'."
  (let ((height (alist-get label squint-heights)))
    (set-face-attribute 'default nil :height height)))

Invoke it with:

(squint 'presentation)

Still works.

Step 3 - Helper functions

Indulge me while I extract a few private functions. They will come in handy when we will change the behavior in the next steps.
In:

(defun squint (label)
  "Adjust font height based on predefined settings.
LABEL is a symbol that corresponds to a key in `squint-heights'."
  (let ((height (alist-get label squint-heights)))
    (set-face-attribute 'default nil :height height)))

we would like to have a function squint--height-from-label returning the height given the label. That’s easy:

(defun squint--height-from-label (label)
  (alist-get label squint-heights))

Then, we could abstract the details of changing the face attribute with:

(defun squint--set-height (height)
  (set-face-attribute 'default nil :height height))
  
(defun squint--set-height-from-label (label)
  (squint--set-height (squint--height-from-label (label))

Putting all together:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(defun squint--height-from-label (label)
  (alist-get label squint-heights))

(defun squint--set-height (height)
  (set-face-attribute 'default nil :height height))

(defun squint--set-height-from-label (label)
  (squint--set-height (squint--height-from-label label)))

(defun squint (label)
  "Adjust font height based on predefined settings.
LABEL is a symbol that corresponds to a key in `squint-heights'."
  (squint--set-height-from-label label))

That’s all. Verify that:

(squint 'programming)

still works.

Step 4 - Making squint interactive

Although you have defined squint, there is no sign of it in the list of commands displayed when you hit M-x. Yet, the function is defined globally: invoking M-x describe-function RET squint RET you can even read its documentation.

The fact is: M-x only lists commands, i.e., those functions that are explicitly declared as interactive.

Well, let’s make squint interactive then:

(defun squint (label)
  (interactive)
  "Adjust font height based on predefined settings.
LABEL is a symbol that corresponds to a key in `squint-heights'."
  (squint--set-height-from-label label))

Is this enough? Try yourself: hit M-x squint RET and you will get back the error:

funcall-interactively: Wrong number of arguments: 
  #[(label) (nil (squint--set-height-from-label label)) nil nil nil nil], 0

The problem is: the poor (interactive) function might even understand that it needs to ask the user for a label argument, but it cannot possibly figure out if this is a file, a string, a number or a symbol from a list, such as in our case. You must give it a hint.
Enter completing-read.

completing-read

Try:

(completing-read
 "Choose your poison: "
 '("arsenic" "digitalis" "strychnine" "belladonna"))

Evaluate it, make your choice and notice how it returns the selected string.
Does it work with symbols too?

(setq symbols
      '(office laptop programming presentation))

(completing-read
 "Choose your poison: "
 symbols)

Yes, it does! But notice: you gave it a symbols, it returns a string. Weird, but it is like it is.
What if you feed it with an alist?

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(completing-read
 "Choose your poison: "
 squint-heights)

It still works, and it still returns the key as a string. Yes, it would have been easier if it returned the value rather then the key, but that’s life.

You can combine interactive with completing-read:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(defun squint--height-from-label (label)
  (alist-get label squint-heights))

(defun squint--set-height (height)
  (set-face-attribute 'default nil :height height))

(defun squint--set-height-from-label (label)
  (squint--set-height (squint--height-from-label label)))

(defun squint (label)
  (interactive 
   (completing-read "Desired height: " squint-heights))
  "Adjust font height based on predefined settings.
LABEL is a symbol that corresponds to a key in `squint-heights'."
  (squint--set-height-from-label label))

Try M-x squint now. Good: now it shows an interactive menu to choose from.
Make your choice, hit RET and be disappointed seing the debug window screaming:

squint--set-height: Default face height not absolute and positive

What’s wrong?
There are 2 issues:

• The first is how we use interactive. If only we spent the time to consult its documentation (M-x describe-function RET interactive RET) we would have seen this line:

If the argument is not a string, it is evaluated to get a list of
 arguments to pass to the command.

We have:

  (interactive 
   (completing-read "Desired height: " squint-heights))

and we know that completing-read returns a string (indeed: the selected key, as a string), not a list. Indeed, interactive must work for an arbitrary number of parameters, so it makes sense it expects a list.
We can easily fix this surrounding completing-read with a list constructor:

(defun squint (label)
  (interactive (list
                (completing-read "Desired height: " squint-heights)))
  (squint--set-height-from-label label))

• The second problem could also be easily forseen: completing-read returns a string, but our squint--height-from-label expects a symbol, because the squint-heights alist uses symbols as keys.
  Here is where having refactored those helper functions away pays off. We can use the function intern, that returns a symbol given a string:

Just change:

(defun squint--height-from-label (label)
  (alist-get label squint-heights))

to:

(defun squint--height-from-label (label)
  (alist-get (intern label) squint-heights))

Putting all together:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(defun squint--height-from-label (label)
  (alist-get (intern label) squint-heights))

(defun squint--set-height (height)
  (set-face-attribute 'default nil :height height))

(defun squint--set-height-from-label (label)
  (squint--set-height (squint--height-from-label label)))

(defun squint (label)
  (interactive (list
                (completing-read "Desired height: " squint-heights)))
  (squint--set-height-from-label label))

Try M-x squint RET (or, if you developed a taste for writing lisp, evaluate (call-interactively #'squint)): it will equally work.

Step 5 - Preview! Preview!

completing-read does already a lot.

• It lets you select the preset with arrow keys.
• It sorts the options accordingly to their most recent usage.
• It lets you filter by typing (try typing "prog"" to see the list interactively narrow down to programming).

The next cool feature you might desire to have is a real-time preview. That is, the very moment you move on a candidate, even before confirming the selection, you would like to see its effect in place. Then:

• Confirming the selection would keep the new height.
• Canceling the selection, it would be nice to reset to the previous height.

This is much more challenging to implement. But maybe we can start by reusing what the magnificent consult package offers.

If you don’t have consult installed yet, show deep remorse, publicly express shame and amend by adding something like this to your init file:

(use-package consult
  :ensure t
  :demand t
  :bind (("M-g M-g" . consult-goto-line)
         ("C-x b" . consult-buffer)
         ("C-s"   . consult-line)
         ("C-S-s" . isearch-forward)
         ("C-x r b" . consult-bookmark)
         ("M-y" . consult-yank-pop)
         ("C-c r r" . consult-ripgrep)
         ("C-c g g" . consult-git-grep)
         ("C-c f l" . consult-focus-lines)))

You will not regret it.

Then, in your squint little package, replace completing-read with consult--read. Let’s see how.

consult–read

To break the ice, I suggest you to experiment with the following:

(consult--read
 '("arsenic" "digitalis" "strychnine" "belladonna")
 :prompt "Choose your poison: "
 :state (lambda (action candidate)
          (pcase action
            ('preview (message "preview: %s" candidate))
            ('return (if candidate
                         (message "return: %s" candidate)
                       (message "return: Nothing"))))))

consult--read is a bit more complex than completing-read, but for our goal it should suffice to know the following:

• Remember that completing-read happily worked with lists of strings, lists of symbols or even alists? Well, consult--read is slightly pickier, as it does not like lists of symbols. Curiously, it accepts association lists whose keys are symbols. So, you can keep feeding it with squint-heights as the list of candidates, just like you used to to with completing-read.

• Just like completing-read, after the candidate has been chosen, it returns the selected key as a string.

• For any action performed by the user, the lambda function defined at :state is invoked, with 2 arguments: action, and candidate.

This lambda requires a bit of explanation.
It is invoked during several stages of the execution. The argument action specifies which. The value of candidate indicates the selected candidate, or nil if no candidate matches or if the user decided to cancel. Or in few other cases. And this makes things a bit hairy.
In practice, just know that the couple of action / candidate shall be interpreted as follows:

I might be wrong, but although I think consult is a masterpiece, I would have prefered consult--read to be a bit more straightforward. Indeed, I find it a bit confusing.
Play with the example I provided above, keeping the *Messages* buffer open (M-x view-echo-area-messages or C-h e) and notice the following:

• As you select a candidate, the lambda is invoked passing preview as the action and the candidate name. This is what we will use to preview the selected squint-height.

• When you confirm the selection:

  • preview is invoked a last time, with nil as the candidate. What the hell?
  • Luckily, return is also invoked, with the selected candidate. Nice, it’s likely we will use this for the final selection.

• Now, try to cancel the selection, using C-g:

  • Also in this case, preview is invoked a last time, with nil as the candidate. It’s likely that we will need to workaround those funny empty calls.
  • Also in this case, return is finally invoked. This time, though, with nil as the candidate.

• Finally, try inputing garbage so that no candidate matches:

  • You will see the very same behavior as for legit candidates. This is also confusing.

A convenient option that helps handling some cases is :require-match t, which prevents the user from confirming the selection with a non-existing candidates. Unfortunately for our goals, preview is still invoked passing non-matching candidtes. Try yourself:

(consult--read
 '("arsenic" "digitalis" "strychnine" "belladonna")
 :prompt "Choose your poison: "
 :require-match t
 :state (lambda (action candidate)
          (pcase action
            ('preview (message "preview: %s" candidate))
            ('return (if candidate
                         (message "return: %s" candidate)
                       (message "return: Nothing"))))))

In conclusion: it seems that for the preview case we have to manually make sure that the input candidate is actually one of the valid choises. Since the candidate list squint-heights is an association list, we could use (assq key alist), a function returning the key/value pair if key is a key in alist, nil otherwise. This leads us to:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))
        
(consult--read
 squint-heights
 :prompt "Choose your poison: "
 :require-match t
 :state (lambda (action candidate)
          (pcase action
            ('preview 
                (if (assq (intern candidate) squint-heights)
                  (message "preview: %s" candidate)
                  (message "Nothing to preview")))
            ('return (if candidate
                         (message "return: %s" candidate)
                       (message "return: Nothing"))))))

An alternative could be to get the height directly, with out squint--height-from-label, then combining the definition of a variable and an if/then/else clause with if-let:

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(consult--read
 squint-heights
 :prompt "Choose your poison: "
 :require-match t
 :state (lambda (action candidate)
          (pcase action
            ('preview 
                (if-let (height (squint--height-from-label candidate))
                  (message "selected: %s, previewing with height: %s" candidate height)
                  (message "Nothing to preview")))
            ('return (if candidate
                         (message "return: %s" candidate)
                       (message "return: Nothing"))))))

Cool. It seems to work.
By the way, having obtained a height already at this point will allow us to use squint--set-height directly, getting rid of the squint--set-height-from-label helper function.

Let’s keep on analyzing. Remember that preview is being called a last time, with a nil candidate? That’s the case when we don’t need to preview anything. We’d better replace the if-let case:

                (if-let (height (squint--height-from-label candidate))
                  (message "selected: %s, previewing with height: %s" candidate height)
                  (message "Nothing to preview")))

with when-let, which does the same of if-let, without an else leg:

                (when-let (height (squint--height-from-label candidate))
                  (message "selected: %s, previewing with height: %s" candidate height)))

Reset

We are almost done. We just need to cover the return case.

Think about this: we will preview the selected height by invoking our squint--set-height: by itself, this is a destructive operation. I mean, although the intention is only to preview the new height, there is no magic way to ask Emcs “I joked: undo what you just did”. Most likely, instead, we will need to keep track of the initial face height, and to invoke squint--set-height with that value, if the user cancels.

So:

• If the user confirms the selection, we do nothing: we can keep the already previewed face height.
• If the user cancels, we need to reset the height to the initial one.

This all means that we definitely need to store the initial height somewhere before invoking consult--read. We could either use a global variable, or a local variable, which we can conveniently capture in a closure.

Let’s play with a closure. Here’s a possible approach:

(let ((previous-height (face-attribute 'default :height)))

  (defun squint--reset-height ()
    (message "%s" previous-height)
    (squint--set-height 180))

  (defun squint ()
    (interactive)
    (consult--read
     squint-heights
     :prompt "Choose your poison: "
     :require-match t
     :state (lambda (action candidate)
              (pcase action
                ('preview (squint--set-height (squint--height-from-label candidate)))
                ('return (when (null candidate)
                             (squint--reset-height))))))))

For closures to work as you expect, Lexical Binding must be active. You ither need to add:

;; -*- lexical-binding: t -*-

as th very first line of your file, or to set:

(setq lexical-binding t)

You probably like to have a little explanation here. Let’s see.

Dynamic and Lexical Binding

Consider this:

;; -*- lexical-binding: t -*-

(let ((v "captured value"))
  (defun run-me ()
    (message "v = %s" v)))

It first defines a variable v with some value. v is not a global variable; it only exists in the scope defined by its let, so basically in the 2 lines under let itself.

Inside that scope, a function run-me is defined. Notice that run-me’s body makes use of v, and that v itself, from the run-me function standpoint, is defined in the outer scope.

We say that run-me captures v from its surrounding scope.

Now: you know that local variables are accessible only from within the scope they are defined. So, the following does not work:

;; -*- lexical-binding: t -*-

(let ((v "captured value"))
  (defun run-me ()
    (message "v = %s" v)))

(message "Do we have a v here? v = %s" v)

v does not exist outside its let scope.
Interestingly, though, this works:

;; -*- lexical-binding: t -*-

(let ((v "captured value"))
  (defun run-me ()
    (message "v = %s" v)))

(run-me) ;; v = captured value

This works because, as we said, run-me captured the variable v. In other words, run-me keeps a reference to the scope it was created in, so that scope does not die when the let expression finishes its execution.
This is basically the idea of closures.

Now, consider this slightly different version:

;; -*- lexical-binding: t -*-

(let ((v "captured value"))
  (defun run-me ()
    (message "v = %s" v)))

(let ((v "some other value"))
  (run-me))

Not surprisingly, the last expression still emits v = captured value.

In fact: we are creating another, separate variable v, with a different value. When run-me is evaluated, though, it still reference its own v. Although sharing the same name, the two v variables exist and live in separate, independent scopes. Which scope a specific piece of code is using can be easily determined by looking at where it is defined in the code structure: in other words, it can inferred from their lexical context.
This behavior is called lexical binding.

Emacs also supports dynamic binding. See the difference:

(let ((v "captured value"))
  (defun run-me ()
    (message "v = %s" v)))

(let ((v "some other value"))
  (run-me))

Notice that I removed the:

;; -*- lexical-binding: t -*-

I could even enabled explicitly the dynamic binding with (setq lexical-scope nil).

Surprisingly, now the last expression emits some other value.

In Dynamic Binding, variables are resolved based on the runtime environment rather than on the definition-time environment.
When run-me is defined, it doesn’t capture v. Instead, it just remembers that it needs to use a variable named v.
When finally run-me is invoked, it looks for the current value of v in the caller scope, at run-time. It finds a v set to some other value, and it uses it.

You will find very few languages supporting this style. You can learn more on this reading Lexical Binding in the Emacs Wiki.

So, here’s the code we ended up with:

;; -*- lexical-binding: t -*-

(setq squint-heights
      '((office . 100)
        (laptop . 180)
        (programming . 190)
        (presentation . 200)))

(defun squint--height-from-label (label)
  (alist-get (intern label) squint-heights))

(defun squint--set-height (height)
  (set-face-attribute 'default nil :height height))

(defun squint--get-labels (squint-heights)
  "Return a list of string labels from squint-heights."
  (mapcar (lambda (pair) (symbol-name (car pair)))
          squint-heights))

(let ((previous-height (face-attribute 'default :height)))

  (defun squint--reset-height ()
    (message "%s" previous-height)
    (squint--set-height 180))

  (defun squint ()
    (interactive)
    (consult--read
     squint-heights
     :prompt "Choose your poison: "
     :require-match t
     :state (lambda (action candidate)
              (pcase action
                ('preview (squint--set-height (squint--height-from-label candidate)))
                ('return (when (null candidate)
                             (squint--reset-height))))))))

Evaluate it, and you will have your beautiful squint command at hand.

Step 6 - Make it a package

It would be nice to have squint loaded when Emacs starts. I’m also sure that, as you get better and better with Lisp, you will improve this little function making it so beautiful that every single Emacs user in the world will want to use it. Eventually, you will need to publish it on Melpa, and it will overtake Magit’s monthly downloads.

So, you need to make it a package.
This is indeed a very easy task.

First, add a very last line declaring that the file provides a package:

(provide 'squint)

Second, make sure that the package name (squint) is consistent with the file name (squint.el).

Finally, store the file in one of the directories Emacs parses when it looks for packages. Inspect the variable load-path to know which ones (M-x describe-variable RET load-path RET).
In my case I get a huge list like this:

("/usr/share/emacs/site-lisp/"
 "~/.config/emacs/local-packages"
 
 "/home/arialdo/.config/emacs/elpa/ag-20201031.2202"
 "/home/arialdo/.config/emacs/elpa/aggressive-indent-20230112.1300"
 "/home/arialdo/.config/emacs/elpa/auto-hide"
 "/home/arialdo/.config/emacs/elpa/avy-20230420.404"
 "/home/arialdo/.config/emacs/elpa/buffer-expose-0.4.3"
 "/home/arialdo/.config/emacs/elpa/corfu-20241030.1005"
 ...
 "/usr/share/emacs/31.0.50/site-lisp"
 "/usr/share/emacs/site-lisp"
 "/usr/share/emacs/31.0.50/lisp"
 "/usr/share/emacs/31.0.50/lisp/vc"
 "/usr/share/emacs/31.0.50/lisp/use-package"
 "/usr/share/emacs/31.0.50/lisp/url"
 "/usr/share/emacs/31.0.50/lisp/textmodes"
 "/usr/share/emacs/31.0.50/lisp/progmodes"
 ...
 "/usr/share/emacs/31.0.50/lisp/calc"
 "/usr/share/emacs/31.0.50/lisp/obsolete")

Notice the second item: "~/.config/emacs/local-packages": this is a custom directory I created for my personal packages. here’s what you need to do:

• Create a directory in your home or in the standard XDG directory (~/.config/emacs) where to keep your personal packages.
• Then, in your init file, instruct Emacs to add this directory to the list of directories to search in:

  (add-to-list 'load-path "<YOUR_DIRECTORY_HERE")

That’s it. Now you can load squint with:

(require 'squint)

If you prefer use-package like I do, go with:

(use-package squint
  :ensure nil
  :custom
  (squint-heights '((office . 1000)
                    (laptop . 180)
                    (programming . 190)
                    (presentation . 200)))
  :bind ("C-c s q" . squint))

Never mind that your code already contains a definition for squint-heights: use-package is smart enough to use the value you provide in :custom.

Cool! Your first package!
Of course, there would be so much to add and study about packages — such as how to write a proper documentation, how to declare dependencies on other packages and the like. But you can be proud of yourself already, can’t you?

Where to go from here?

Why should squint work with font heights only? Why not to have presets for any arbitrary face attribute?
The scaffold is set up: you will not find hard to extend what you wrote so far.

If you want to get inspiration and to learn from a real master, go checkout Fontaine, by Protesilaos. It does exactly that.
In fact, this post is an elaboration of what I learnt from some lessons I got from him, as I wanted to better understand how his packages work.

That’s all. Take care. Happy lisping!

References

• Emacs Manual
  • Text Scale
  • Faces
  • Association Lists
  • Symbols
  • Lexical Binding
    • Lexical Binding - Wiki
• consult
• Fontaine
• Protesilaos Stavrou

Binary Trees

Our journey starts with this problem: we want to count the number of leaves of an arbitrary Binary Tree. To keep the problem as simple as possible, we define a Binary Tree as that data structure having Nodes and Leaves, and in which every Node has exactly two branches.

This is a Tree with 2 Leaves:

     Node
     /  \
  Leaf  Leaf

and here is a Tree with 3 Leaves:

     Node
     /  \
  Leaf  Node
        /  \
    Leaf   Leaf

It is legit to ask ourselves: is a single Leaf a Tree?

   Leaf

Let’s agree that it is.
Do null Trees — Trees without Nodes and Leaves — exist?
Let’s do ourselves a favor and agree they don’t.

A Node always contains 2 Leaves. What does a Leaf contain? Nothing: it just is.
So, a Tree is either a Leaf or a Node made of 2 branches, each containing another Tree structure. Remember this definition, because we will translate it more or less literally to F#.

How deep can such Binary Trees be? Arbitrarily deep. For example, this is a legit Tree:

     Node
     /  \
  Leaf  Node
        /  \
     Node   Leaf
     /  \
  Leaf  Node
        /  \
      Node  Leaf
      /  \
  Leaf   Node
         /  \
      Leaf  Leaf

Of course, our code should work with any Tree, no matter how deep.

Counting Leaves

Good. We are diligent developers, so we start from a test. To keep it simple, let’s count the Leaves of a 3-leaves Tree:

module StateMonadTest

open Xunit
open Swensen.Unquote

let numberOfLeaves tree = failwith "Not yet implemented"

[<Fact>]
let ``counts the number of leaves in a tree`` () =
    let treeWith3Leaves = failwith "Not yet implemented"
    
    let leaves = numberOfLeaves treeWith3Leaves
    
    test <@ leaves = 3 @>

Let’s fill the gaps. As often happens with Functional Programming, it is convenient to start from modeling problems through their types.

Sum and Product Types

How to model a Tree? That’s an easy task in F#: it’s a matter of translating literally our definition:

A Tree is either a Leaf or a Node containing 2 branches each containing another Tree structure.

Step by step:

• A Tree is is simply translated to

type Tree =

• A Tree can be a Leaf:

type Tree = Leaf

• or a Node. The or part is translated with the | operator:

type Tree = Leaf | Node ???

• The Node contains 2 branches. The contain part is translated with of, the multiplicity is translated with *:

type Tree = Leaf | Node of (??? * ???)

• We know that each branch contains either a Leaf or another Node. We already have a type representing either a Leaf or a Node, dont’t we? It’s the Tree itself. So:

type Tree = Leaf | Node of (Tree * Tree)

That’s it!
As an aside, this expression makes use of the so called Algebraic Data Types, which include Sum (or Discriminated Union) Types and Product Types.

In C# this would approximately translate to:

abstract record Tree;
record Leaf : Tree;
record Node(Tree Left, Tree Right) : Tree;

In both cases, we have a recursive type, which both languages happily support.

Creating instances

Creating instances in F# is similar, but more concise, than in C# and Java: you just have to omit the new keyword and the parenthesis. Our 3-leaves tree:

     Node
     /  \
  Leaf  Node
        /  \
    Leaf   Leaf

can be created with:

let treeWith3Leaves = Node(Leaf, Node(Leaf, Leaf))

This completes the test implementation:

[<Fact>]
let ``counts the number of leaves in a tree`` () =
    let treeWith3Leaves = Node(Leaf, Node(Leaf, Leaf))

    let leaves = numberOfLeaves treeWith3Leaves

    test <@ leaves = 3 @>

Good. Now, let’s see how to make it green.

Type Constructors and Data Constructors

It is important to understand that in:

type Tree = Leaf | Node of (Tree * Tree)

Tree is a type, Leaf and Node are not: Leaf and Node are ways for creating an instance of type Tree. This is akin to the difference between a class and its constructors. In F#, we would call Tree, Leaf and Nodeas follows:

Haskell uses different names, but the meaning is the same:

We will shortly see why Tree is a Type Constructor, not just a Type: as soon as Tree will take a (type) parameter, we will learn to see it more as a function than just a type name.

The takeaway here is: given an instance of Tree, F# can pattern match on the Case which was used to create the instance. It’s like the instance remembers which constructor was used to create it. You can read more about it in Pattern Matching - Identifier Patterns.

This give you all the ingredients to develop a function to calculate the number of leaves.

Implementation

Pattern matching is easy:

let numberOfLeaves tree =
    match tree with
    | Leaf -> ???
    | Node (l, r) -> ??? 

Read it as a glorified if/then/else and ask yourself:

• If numberOfLeaves receives a tree, and that tree is a single Leaf, how many leaves does that tree have? In other words, the function needs to calculate the number of leaves of this tree:

   Leaf

Of course, it has just 1 leaf! So:

let numberOfLeaves tree =
    match tree with
    | Leaf -> 1
    | Node (l, r) -> ??? 

What about if tree is a Node? You only know that a Node contains exactly 2 branches. Each branch could be a Leaf, like in:

     Node
     /  \
  Leaf  Leaf

or it could be another arbitrarily deep tree, like in:

     Node
     /  \
  Leaf  Node
        /  \
     Node   Leaf
     /  \
  Leaf  Node
        /  \
      Node  Leaf
      /  \
  Leaf   Node
         /  \
      Leaf  Leaf

In the general case, you can see the received tree as:

     Node
     /  \
  Tree  Tree

This node contains as many leaves as the leaves in the right branch tree, plus the leaves of the left branch tree. Literally translating this sentence to F# leads us to:

let rec numberOfLeaves tree =
    match tree with
    | Leaf -> 1
    | Node (l, r) -> numberOfLeaves l + numberOfLeaves r

Notice that we added the keyword rec to the function definition, to make it clear that the function is recursive.

Does this work? You have a test. Run it.
Of course it’s green. As it often happens, if it compiles it works. Get used to this, it will happen over and over.

Putting all together

Let’s review what you obtained:

type Tree =
    | Leaf
    | Node of Tree * Tree

let rec numberOfLeaves tree =
    match tree with
    | Leaf -> 1
    | Node(l, r) -> numberOfLeaves l + numberOfLeaves r

[<Fact>]
let ``counts the number of leaves in a tree`` () =
    let treeWith3Leaves = Node(Leaf, Node(Leaf, Leaf))

    let leaves = numberOfLeaves treeWith3Leaves

    test <@ leaves = 3 @>

You can see how the:

    | Leaf -> 1

branch is the base case of recursion, while the:

    | Node(l, r) -> numberOfLeaves l + numberOfLeaves r

is the (double) recursive step.
Also notice how the results of the 2 recursive calls are composed with the binary + function.

F# <3 concisiness (and Haskell loves it too)

In:

let rec numberOfLeaves tree =
    match tree with
    | Leaf -> 1
    | Node(l, r) -> numberOfLeaves l + numberOfLeaves r

the parameter tree is only used to pattern match. To stress this, F# provides an alternative, more concise syntax, where

... tree =
    match tree with

is replaced by:

... function =

So, our function can be made shorter, as:

let rec numberOfLeaves = 
    function
    | Leaf -> 1
    | Node(l, r) -> numberOfLeaves l + numberOfLeaves r

A nice way to interpret this is to squeeze the eyes and imagine it’s a combination of 2 separate function definitions, one on Leaf and one on Node(l, r). Haskell would really let you define 2 separate functions:

numberOfLeaves Leaf       = 1
numberOfLeaves Node(l, r) = numberOfLeaves l + numberOfLeaves r

If you manage to understand this last snippet, bravo!, give yourself a pat on the back, take a little breather, and get prepared to the next chapter, in which you will invent (or discover) Functors.

Jump to Chapter 2.

References

• State Monad For The Rest Of Us - source code
• Microsoft Learn - Pattern Matching - Identifier Patterns
• Philip Wadler - Propositions as Types

Comments

GitHub Discussions

Functional Programming For The Rest Os Us

Interested in FP? Be the first to be notified when new introductory articles on the topic are published.

In chapter 7, when we wanted to introduce a new type for the result value of index we got to this:

// Tree a -> (Int -> (Tree (a, Int), Int))
let rec index =
    function
    | Leaf v -> fun count -> (Leaf (v, count), count + 1)
    | Node (l, r) ->
        fun count ->
            let li, lc = index l count
            let ri, rc = index r lc
            Node (li, ri), rc

I mentioned that there was a second perspective, leading to the same signature of WithCount:

• Manipulate the signature, separating the count-handling and domain logic on the type level.

Let us do this, because it will lead to some interesting refinements

Working on the signature

Observing again the original signature:

Tree a -> (Int -> (Tree (a, Int), Int))

it’s not hard to see what happened. Ideally, you wanted a simple function:

Tree a -> Tree (a, Int)

taking a tree of whatever content and returning an indexed version of it. This is the original domain logic, which you can easily make more generic with:

a -> (a, Int)

Then, you wanted to have a Functor to take care of the logic for traversing a tree. But then you found out that Functors are not powerful enough to apply an indexing function with map.

Instead of that, you ended up with:

or, generalizing:

Squint the eyes. This expression contains the domain logic and the count-handling logic mixed together. Let me move the count-handling logic to a separate line:

Tree a ->           Tree (a, Int)
          (Int -> (               , Int))

or if you want to make to use the more generic version:

a ->            b
     (Int -> (    , Int))

The:

Int -> (b, Int)

captures exactly what you did in chapter 6:

If you want to give this idea a name and to capture it with a type, it makes sense to define it with:

type WithCount<'b> = WithCount of (int -> 'b * int)

Of course, this could be more specific to trees:

type WithCount<'b> = WithCount of (int -> Tree<'b> * int)

or even more specif to our indexing use-case:

type WithCount = WithCount of (int -> Tree<string, int> * int)

but why should it be? On the contrary, it could be generic also on the type of the counter:

type WithCount<'b, 'c> = WithCount of ('c -> 'b * 'c)

where 'c is the type of the state being chained, and 'b is the original domain logic result type.
c is not a count anymore, it’s a generic state. Let’s call it s. And WithCount is a misleading name. We can easily call it State:

type State<'b, 's> = State of ('s -> ('b * 's))

This generalized version of WithCount is, ladies and gentlemen, the signature of the State Monad.
It’s definitely the time to implement it. Ready! Steady! Go with Chapter 11!

References

• State Monad For The Rest Of Us - source code

Comments

GitHub Discussions