Great programming is mathematics. … Except, all falsehoods are the same and error messages are not. Otherwise, great programming is mathematics.
    A quote that I wish someone whom I could quote had said

In this post I will talk about improving error messages. We will also discuss dogs which compose music.

I wrote about error messages in Haskell before, I decided to give it one more go. I am working on a slowly progressing task: rewriting code to improve the quality of error messages across projects I contribute to (… and log messages too, but I will focus on the error outputs here). I try to dedicate a few hours every sprint to it. This work often includes rethinking parts of aeson or Parsec code that use MonadPlus / Alternative when the resulting error message is likely to throw anyone for a loop, or re-implementing code that uses Maybe where something like Either would be a better choice, or where errors were never caught… This work also involves adding a decent amount of context to the messages. I have been trying to fix up the errors for several years now and I am starting to believe that this work may never end. You roll this rock uphill and it rolls back down. Can Functional Programming create quality error outputs? Of course it can! But, for this to happen on the level of projects … I think that the community needs to talk about it more.

The (low) quality of error messages I witness in functional code is something that has puzzled me for a very long time. I wrote about Maybe Overuse and Alternative Overuse in the past. The first received a very mixed response (including very positive and very negative), the response to the second was flat negative. I decided that the reasons for what I am observing are probably mostly not technical. This (at least partially) motivated me to look into cognitive psychology (Cognitive Loads in Programming), and I came up with “a theory” about Theorists vs Pragmatists. I cannot claim that I understand what is happening, I can only claim spending literally years thinking about it.

I want to try one more time to talk about my experience with errors and troubleshooting with some examples and thoughts. My current plan is that with this series (or with this post) I will end my blogging.

This post is also about conveniences. It shows a few examples where established parts of the Haskell ecosystem make it easy to be careless about errors or where providing decent error messages is simply hard.

I will mostly focus on aeson (the premier Haskell package for working with JSON) with short mentions outside of this library. This is because code that uses aeson has been my more recent refactoring effort and is fresh on my mind.

Historical notes

These are my (Haskell user) observations about the history of error messages in the Haskell ecosystem. If you have been using Haskell for a long time, you probably remember that aeson did not have eitherDecode at the beginning. eitherDecode was added in 0.6.1.0 (about two years after the initial release). What it did have (and unfortunately still does) is a more nicely named

decode :: FromJSON a => ByteString -> Maybe a

If I did my hackage archaeology correctly, an ability to output error messages was added in 0.2.0.0 with the introduction of parse :: (a -> Parser b) -> a -> Result b which has been hiding in Data.Aeson.Types. The commonly imported Data.Aeson module did not have an error message producing combinator until 0.6.1.0.

If you look over the documentation of the older versions of aeson you will see the following code as the suggested implementation for FromJSON:

-- A non-Object value is of the wrong type, so use mzero to fail.
   parseJSON _          = mzero

I am still finding (and fixing) similar code despite a past effort to eradicate these. It is not easy to troubleshoot a bug if the message handed to you says only “mzero”.

With respect to error messages, aeson clearly went a long way since the old days. If you look at aeson’s Haddock today you will find the use of mzero discouraged!:

"The basic ways to signal a failed conversion are as follows:

• fail yields a custom error message: it is the recommended way of reporting a failure;
• empty (or mzero) is uninformative: use it when the error is meant to be caught by some (<|>);
• typeMismatch can be used to report a failure when the encountered value is not of the expected JSON type; unexpected is an appropriate alternative when more than one type may be expected, or to keep the expected type implicit.

prependFailure (or modifyFailure) add more information to a parser’s error messages."

However, I still find the recommended use of <|> for working with errors an odd design choice. I will explain shortly why.

There are other libraries where an ability to get or provide crucial error information has been added only recently (e.g servant-multipart). At the same time, there are many examples where Maybe has been overused in the past and still is. My Maybe overuse post has a few examples like these¹.

Criticism outlined

Maybe criticism: Legacy Maybe combinators should be causing some concern. In programming, legacy is inertia. Maybe is not the correct type to represent something like a parsing failure, it can be useful to describe missing data but not for situations where we care about what went wrong (like parsing errors). A decoding function that returns Maybe should be marked deprecated and eventually removed. Functions like these are found in many libraries, not just aeson, and this is not just about parsing. One can even see it as a pattern across the whole Haskell ecosystem.

Anyone in a desperate need of dropping the error information can do that with an easy to create natural transformation like:

errInfoDon'tCare :: Either e a -> Maybe a

I am not trying to be sarcastic, IMO “who cares?” is a fair question to ask. It would be loud enough and useful in PR reviews.

Hyrum’s Law and friends: If you believe there is some truth to the Hyrum’s Law (the law which states that all, even the unintended ways to use a library will be exploited by its users) you will probably agree with my stance on this. I like to think about Hyrum’s Law using words that end with “use”: use, overuse, misuse, and abuse. Programming concepts often end up being misused and abused, it is enough for a library like aeson to provide an opening.

I do believe in (or rather, have been observing) something similar to Hyrum’s Law, namely:

  Developers are likely to choose convenience over correctness

I call it The Law of Convenience and note that Maybe is much more convenient to use then Either.

I also believe that writing code has a significant habitual factor. Ignoring error messages is a concerning habit to have.

And, finally, I believe that major libraries lead the ecosystem by example.

Haskell is converting from a research language to a language that is used commercially and topics like efficient ability to troubleshoot production issues are becoming important. The changes I am observing are good, I only hope that the community will get more aggressive on this front.
In particular, it would be nice to see more error types that are semantically richer than String. We do not want String to become the type of choice for errors and I am happy to see when this is not the case (Megaparsec, yaml, amqp, …). I would also love to see more of standard type level consideration for errors (e.g. standard typeclasses for working with them that go beyond the Exception typeclass)².

MonadPlus error laws

I am fixing a lot of code that uses Alternative / MonadPlus abstractions. The next section will show a code that produces wrong error messages by misusing these abstractions. In this section I will discuss MonadPlus in more general terms.

MonadPlus is a very convenient and easy to use Monoid like abstraction. It comes with mzero which is often used to represent a failure without any error information. It is supposed to be a principled abstraction that needs to follow certain monoid-like laws (see MonadPlus, Laws). Does this abstraction play well with computations that also can emit nontrivial errors?

To dig this rabbit hole a little deeper, let’s try to test the second law for mzero (v >> mzero = mzero) polymorphically by adding MonadFail constraint:

tst :: (MonadFail m, MonadPlus m) => m b
tst = fail "not mzero" >> mzero

Now I can try it with different monads to see if its error output is the same as mzero’s. E.g.:

{-# LANGUAGE TypeApplications #-}

import qualified Text.Megaparsec as MP
import Data.Void

-- |
-- >>> verifyMP
-- False
verifyMP :: Bool
verifyMP = runTest tst == runTest mzero 
  where 
    runTest p = MP.parse @Void @String @Int p "test" ""

Examples that fail “tst output is the same as mzero output” tests:

• IO
• Parser from Data.Aeson.Types
• Parec from say Text.Megaparsec
• Parser from attoparsec

Examples that pass such test:

• Maybe
• ReadP and ReadPrec from Text.ParserCombinators

Maybe has no error information, Text.ParserCombinators implement mzero as a no-message failure.

Question: Can we find an example where a monadic computation allows for nontrivial error messages and passes this test?

Answer: For a failing computation v, we would expect³ v >> anything = v. This, combined with the second mzero law (v >> mzero = mzero) implies that any failing computation is equivalent to mzero. So, we either need to think about the second mzero law “modulo errors” or we have to accept that any lawful MonadPlus computation will suppress error information.

I believe developers are divided into these 2 camps: those that think about and implement laws, and those who do not, but are nevertheless surprised when computations behave in unlawful ways. We consciously or subconsciously assume various computational properties when we reason about the computations. Partially lawful is concerning. If you care about error output, “lawful modulo errors” should be concerning, having such limitation undocumented is concerning too.

Principled computations give us abstractions to work with, like theorems are tools to a mathematician. We do not need to think about the details, just apply them to create new code. When we do that with MonadPlus error messages can fall through the cracks. I am dealing with quite a bit of code that has fallen into this trap. Next section will show one such example.

Who cares about errors?: I hope we will come up with principled abstractions that are error message friendly. I am looking forward to a day where aeson will stop recommending the use of <|> as an error message signaling abstraction.

errInfoDon'tCare :: Alternative f => Either e a -> f a
errInfoDon'tCare = either (const empty) pure

Alternative dog music. A use of <|> is considered harmful

Let’s sketch a contrived code to illustrate a use of <|>:

data Pet = MkPet {
  breed :: Breed
  , species :: Species
  , petname :: Text
}

data Composer = MkComposer {
    genre :: Genre 
    , composername :: Text
}

data Favorite = 
  MkFavoritePet Pet
  ... -- there are other favorite things 
  | MkFavoriteComposer Composer

-- Constituent types (Pet, Composer) intances are not shown
-- Assume these types have unique (different) JSON representations

instance FromJSON Favorite where 
  parseJSON v = 
     MkFavoritePet <$> (A.parseJSON v) 
     <|> ... -- parse other things
     <|> MkFavoriteComposer <$> (A.parseJSON v)   

Note the above Law of Convenience applies here: this code reuses existing JSON parsers to create the parser for Favorite, and this parser is very easy to implement. This also looks elegant, and seems to principally fit the Alternative very well. There is no special JSON representation of Favorite, rather we use JSON representations of the constituent types Pet, Composer, etc. This approach does not fuss with data constructor tags eliminating some JSON size overhead and looks ideal for structurally typed callers (e.g. TypeScript). But, this approach has issues.

Assume this has some frontend UI. Assume that a user enters information about her favorite four legged friend and that does not parse for some reason (e.g. frontend JSON encoding of Pet is incorrect). The error message from the parser will say something like

"Composer needs a genre"

(or whatever error FromJSON for Composer returns if it is given an unexpected JSON object).

We see a couple of problems: the message is misleading and it lacks context (there is nothing in this message to indicate that it came from the JSON parser for Favorite). I will focus on it being misleading because, believe me, this coding pattern can produce very confusing errors in real life. Code like this is something I am slowly working to fix in projects I contribute to. Fixing such code is often not easy.

Are developers aware of this <|> issue? Probably some are and some are not. Code like this is probably written because JSON parser errors are unlikely to be viewed by the end user, aeson makes code like this easy to implement, the code looks elegant, and error messages are the last thing on people’s minds.

Which leads to another question:
Q: How would we guard against issues like this? A common practice for avoiding program issues is writing tests. How do I write a non-brittle test that checks the quality of aeson error messages? Do I write message parsers?

Let’s forget about <|> for a moment and try to formalize what a parser error message is: Consider the input document specification as a collection of sets of detailed specs S_T, one for each parsed type T (e.g. “Composer has a not-nullable ‘genre’ field of type Genere” is an element of S_Composer) . An error message pin-points an⁵ element in one of these sets marking it as failed (e.g. “Composer needs a genre”).

To return a user-friendly error message, the parser needs to choose S_T wisely by matching the data the user is working on. Parser needs to have access to enough information about this context to compute which S_T to use (data constructor tags is an example of how such context is provided to the parser). Thus,

  thinking about user friendly error messages needs to be a part of software design and input specification.

The point I have been trying to make is that using Alternative / MonadPlus in computations where error information is important (like parsing) can be very tricky. It requires thinking about and testing error outputs, not something developers typically do.

  Hmm, I think Snuffy’s genre would be hard rock. But what if the dog’s name is Beethoven?

Overloaded errors

This section will be more subtle. Programs sometimes need to be selective about which error condition is handled.

We will try to write a program that checks if the local config file “.my.yaml” exists and if not, uses “~/.my.yaml”, and returns an error if there is an issue with any of the files.

We will use MonadPlus instance of IO. Here is standard library implementation of mplus or <|> for the IO Monad:

mplusIO :: IO a -> IO a -> IO a
mplusIO m n = m `catchException` \ (_ :: IOError) -> n

Let’s take a journey trying to do implement this and see some nuances and how complex using IO with <|> can be:

import qualified Data.Yaml as Y -- yaml package dep
import Control.Applicative ((<|>))
import System.FilePath ((</>))  -- filepath package dep
import Data.ByteString as BS    -- bytestring package dep
import Control.Exception ( throwIO )

-- MyConfig and its instances not shown, home directory is passed as argument for simplicity

-- will not alternate to home directory file no matter what the issue with the local file is
-- because Y.decodeFileThrow is not throwing IOError, it throws Y.ParseException  
won'tWork :: FilePath -> IO MyConfig
won'tWork homedir = 
    Y.decodeFileThrow ".my.yaml" 
    <|> Y.decodeFileThrow (homedir </> ".my.yaml")

-- Y.decodeFileEither :: FromJSON a => FilePath -> IO (Either ParseException a) 
-- uses ParseException to also signal readFile issues like missing file
-- this puts all problems in one bucket and alternates to home directory on any issue with the local file
conflateAllIssues :: FilePath -> IO MyConfig
conflateAllIssues homedir = decode ".my.yaml"
    <|> decode (homedir </> ".my.yaml")
  where 
    decode :: FilePath -> IO MyConfig
    decode file = Y.decodeFileEither file >>= either (ioError . parseErrToIOError) pure  
    parseErrToIOError :: Y.ParseException -> IOError
    parseErrToIOError = userError . show -- for illustration only


-- still not ideal, it conflates any IOError issued from BS.readFile and alternates on any of them
-- however invalid syntax in local file will now cause an error
isolateIOErrors :: FilePath -> IO MyConfig
isolateIOErrors homedir = 
    decodeFileIsolateIOErrors ".my.yaml" 
    <|> decodeFileIsolateIOErrors (homedir </> ".my.yaml")

-- override what yaml package provides
decodeFileIsolateIOErrors :: FilePath -> IO MyConfig
decodeFileIsolateIOErrors file = do 
    bytes <- BS.readFile file      -- possible IOError
    either throwIO pure $ Y.decodeEither' bytes -- not IOError

If you dislike this code, then I am with you. This example’s goal is to illustrate a thought process that goes into handling errors, so let’s focus on that process only.

conflateAllIssues example conflates (and silences) all of these things:

• local file is missing
• invalid yaml syntax in the local file
• local file yaml has valid syntactically but does not represent MyConfig
• other IO issues related to the local file, e.g. file access problems, file corruption …

The requirement is to alternate to the home directory file only when the local file is missing and output an error message otherwise. isolateIOErrors moves in this direction, but is still not right (it will alternate if there is anything wrong with readFile). Obviously there are ways to move forward, e.g. explore Y.ParseException constructors (there is more than one!) and make decisions whether to convert to IOError to alternate or not, or explore the content of the IOError returned from readFile and flip some of it outside of IOError.

I hope this shows that things can get complex.

My hidden goal behind this exercise was to have us notice something that applies to a wider range of MonadPlus / Alternative instances. In particular, it is related to the previous example. The impression I probably left on you in the previous section was: a naive use of Alternative results in bad error messages.

I look at the “Who cares about K9 composers” as a deeper issue of 2 conflated errors. The code in the previous section conflates errors from parsing JSON data representing one of the possible constituent types (parsing wrong branch), with errors from parsing JSON data that does not represent any of the constituent types (parsing unexpected data). This code cannot distinguish between these errors and alternates on both. Ideally we would only alternate on the first but there is no obvious way to do that (aeson errors are Strings).

Overloaded errors is a concern when programming parsers using MonadPlus instances. This is subtle and, probably, I have not explained it clearly enough. Please give it some thought before dismissing it.

Topics to discuss

In this post I wrote about things that irk me at the present moment. I think that the overall situation with error messages is getting better and better, but IMO we are far from being where we should be. Haskell does not have expressive stack traces or convenient debuggers. One would assume the community will try to compensate with clear error messages and great log outputs to make up for these limitations. I believe this topic needs more attention.

Here is a broader list of engineering topics that are IMO worth discussing:

• Overuse of String / Text as the error type.
  
• Programming approach where Either Monad / MonadError-like computations augment error outputs with additional context at every opportunity. Strategies for compounding error information.
• More about code that incorrectly uses wide ranging instead of specific errors and how abstractions fit into this.
• I dislike the non-termination throw catch games. Throwing errors effectively bypasses the type checker. If you think of types as propositions and programs as proofs, you can prove any nonsense by throwing an error. IMO, explicit Either type (or its close friends ExceptT/MonadError) are a better way to write code. To me, throwing errors is not FP (think about Idris or even Rust for alternative ideas). IMO, the same goes for effect systems: I prefer no throw catch games. I would like to see the use of error :: String -> a, or even things like IOError eradicated from the ecosystem, (e.g.  readFile :: FilePath -> IO (Either IOFileErr ByteString)). (I unloaded a lot from my chest here 🙂)
• Type level consideration for errors.
• Strategies for dealing with non termination caused by use of error :: String -> a (a pure function, I call it 😉 “pure evil”).
• More about Maybe, MonadPlus, Alternative when they are, in addition to being very convenient, completely OK to use.
• More about MonadPlus, Alternative when their use is concerning (e.g. are you using guard :: Alternative f => Bool -> f () in parsers? If so, how?).
• Strategies for refactoring code overusing Alternative in parsers. Writing parsers without using <|>.
• Monadic vs Applicative parsers comparison from the error messages standpoint.
• Strategies for input spec designs (e.g. for JSON representations, tagging constructors vs tagging types).

It would be nice to know if I am alone in my views and if these topics are of interest for anyone out there. If not, I will probably make this my last blog post, blogging is costing a little bit too much energy. If yes, I will select one of these topics and try to write more over the summer.

Was this post negative? IMO, there is a difference between negativity and frustration. Frustration can result in something positive, negativity cannot. Frustration seeks understanding, negativity does not. Frustration can unite, negativity can only divide.

If you agree with some of the things I wrote here, please try to focus on these and let me know! Thank you for reading and for your feedback.

I am a concerned Haskeller who loves and adores this language.

To all my readers: thank you for reading my posts and for your constructive comments and for your encouragement.

“When the going gets tough, the tough get empirical.” Jon Carroll

“If in physics there’s something you don’t understand, you can always hide behind the uncharted depths of nature. You can always blame God. You didn’t make it so complex yourself. But if your program doesn’t work, there is no one to hide behind. You cannot hide behind an obstinate nature. If it doesn’t work, you’ve messed up.” Edsger W. Dijkstra

I will discuss (on a very high level) empirical, experimental, and deductive aspects of programming. This distinction is fun to analyze (and a somewhat unusual way to look at programming) but it seems mostly a useless curiosity in itself. I have convinced myself that programmers tend to favor empirical or favor deductive. IMO, we are placing ourselves into 2 camps. I call these camps pragmatists and theorists. This division impacts how we program and communicate. Discussion of both mindsets is the main goal of this post.

The topic for this post came from a realization I had when thinking about cognitive loads (Cognitive Loads in Programming). It may be obvious to some of you, but it was not obvious to me: programming is largely an empirical process. I will argue that the pragmatic empirical mindset is also dominant.

What the others mean, how they reason, or what is important to them are the contexts in human communication. Communication without the basis of common interest is hard. Good communication requires an effort of understanding these contexts (we call it finding a common language). IMO, empirical, deductive, pragmatist, and theorist are a good terminology choice to analyze some of the current discourse (especially about FP). In this post I will present my observations about the empirical and deductive mindsets. You may disagree with me, please let me know if you do. The point is to get these contexts right, or at least to have all of us think about them a little.

Fairness and lack of bias are rare but beautiful if encountered in human interactions. In my current work I mostly use Haskell and I have been interested in functional programming for a long time, I am a mathematician who became a software developer (27 years ago, but these things stay with you). Thus, this post is likely to have some unintended bias. Also, these are my opinions, not an attempt at a scientifically sound reasoning.

I have not found much discussion about the empirical nature of programming, I am not following academic research in any related area. The topic of empirical software engineering is relevant to programming and the empirical method, but is not really what I will talk about. Retrospecting on my software programmer career, I recall good and bad things. The good had good communication, the bad had bad communication of some sort. Pragmatists vs theorists is just a part of a bigger puzzle, I am going to explore that part here.

So what is the point I am trying to make? My only real point is that both mindsets are important, my goal is to discuss empirical and deductive (programming), theorists and pragmatists (programmers) in as much debt as I can muster.

When reading it, keep in mind that my goal is to present different points of view developers are observed to have, I am not trying to make you agree with these viewpoints (that includes my side notes on formalism, mathematics, programming, and everything else in this post). The fact that we may disagree on these, is a corollary of the diversity I am trying to explore. This post is not about establishing consensus which viewpoint is better, it is about gaining some understanding of these viewpoints.

Empirical vs Deductive

In my experience, developers do not use this terminology. Hey, in my experience, I do not use this terminology so a recap is in order.

Scientists use these 6 (or more) terms: inductive, empirical, a posteriori, deductive, rational, and a priori. This sounds like the beginning of a catchy song. There is also informal and formal but it breaks the rhythm, so I skip these for now. Practical and theoretical are great terms to describe a mindset, while empirical and deductive describe the thought process. In this post I will mostly try to use empirical and deductive when referring to the thought process, I will use pragmatist and theorist when referring to people.

Empirical (to avoid using inductive¹) reasoning draws general conclusions from observations. E.g. certain software functionality will work because we tested it (notice how well software testing fits into this definition). Thinking about software testing as an empirical process allows us to consider things like bias² (e.g. the assumptions we make in unit tests), observation sample size (did we test enough?), correlation vs causality (e2e tests are unstable, is there something wrong with e2e testing itself?), if observations are balanced (adequate coverage of test scenarios across functional areas), establish observation baseline (golden testing for complex deterministic code)… Here is an exaggerated example of a correlation vs causation problem: “Each time I test the app it works just fine, yet users keep reporting issues. Something’s wrong with these people!”. But it is even more fun to think about both experiments and observations.

Deductive reasoning goes from general knowledge to specific conclusions. E.g. certain software functionality will work because of type safety or because it is a straightforward application of something else we believe works. In this post, deductive represents a wide range of thought processes: from tedious “observation-less” mental verification of values and types (oh, this value was supposed to be a positive number, why is it negative?, is the new refactored code equivalent to the previous?…) all the way to formal reasoning (e.g. equational proof that refactored code is equivalent).

In this essay, I use the term “formal” somewhat loosely. In particular, formalization of mathematics can mean something even much stricter than mathematical proofs, e.g. proofs that type check using a proof assistant. In this post mathematics is viewed as an example of formal thought already, my goal is to contrast it with, say, biology.

Empirical and deductive work in tandem and both are essential. IMO, we (as individuals) prefer to use one more than the other. Understanding more about these preferences will be the main topic of this post.

You may think that I am spending too much time explaining the obvious: any engineering will have a strong empirical aspect, engineers like to tinker with things. I assume that some readers are like me and have not thought about it before. The next section goes deeper into the empirical nature of coding.

Side note: formal deduction, mathematics, and immutability of knowledge

A logical thing to do here would be to discuss deductive systems in programming (e.g. operational semantics, equivalences between mathematical and programming concepts, etc). I decided against it, doing this would create a bias towards theorists and would probably alienate some pragmatists. I hope I am avoiding such bias by using mathematics as the example of formal thought.

My comments about mathematics being mostly immutable and additive and very different from empirical sciences that keep mutating to get things right got a lot of pushback from the readers. This section expands on this topic and make some points, previously buried in footnotes, more visible. Here is an example of feedback from a reader:

“Lol what? Mathematicians are fallible and often find faults in decades old proofs.”

I agree with the quoted text in general, but I do not agree with the often quantifier. We need to keep things in context. We are comparing mathematics to other sciences. I am not suggesting that published mathematical work is never wrong (e.g. the premises behind Homotopy Type Theory, Univalent Foundations ⁶, attempts to formalize maths with proof assistants ⁷ are all thumbs up). My argument is comparative, errors in mathematical proofs exist but are rare and exceptional. Comparatively speaking mathematics is a beacon of truth and an outlier.

Example: Why do we have infinitely many prime numbers? We are discussing formal deduction so let’s do it!

Theorem: Set of prime numbers is infinite.

Proof: Let S denote the set of all primes. Assume it is finite. Let P be the number obtained by multiplying all numbers in S PLUS 1.
By construction, forall p ∈ S, P mod p = 1 (P divided by any prime p has reminder of 1).
Thus, the prime factors of P are not in S. Thus, we have a contradiction. S has to be infinite. QED

This is the proof I remember being taught in school. How old is the proof itself? I believe that nobody knows. We can find it in Euclid’s Elements. So, it was known around 300 BC! I hope this example gives you some evidence of the immutability of “formal thought”. Yes, some areas of mathematics have changed around 1900, but a surprising amount of it has not changed for centuries.

In contrast, Empirical sciences are in the midst of replication crisis with reproducibility rates for published results below 50%⁸.

It is good to note that the above proof still does some hand waving. E.g. “Thus, the prime factors of P are not in S” implicitly assumes knowledge about prime factors and I am not referencing the “fundamental theorem of arithmetic”: every natural number greater than 1 is either a prime itself or can be factorized as a product of primes that is unique up to their order. I left it ambiguous on purpose as such ambiguities are often present in math proofs. If P itself is prime the proof implicitly defines its prime factors as a one element set {P} which is somewhat not standard. This all can be fully formalized but it would be tedious to do that and avoiding such tediosity is common in math. However, this looseness sometimes leads to overlooked errors. Again, such errors are rare. There is also the question of constructive proofs, this proof is not constructive. There is a concept of “proof relevant” logic, traditional mathematics is not proof relevant (all proofs are treated as equivalent and are not themselves an object of study). I am not delving into these areas at all.

Some mathematicians are trying to move towards formalizing such proofs. The new way is to write “informal” proofs for human consumption (like the one above) and accompany them with proof assistant type checked versions. This change, again, is additive as opposed to mutating. IMO, the above proof is likely to remain for another 2000 years.

I believe proof assistants and type checking the proofs will be used much more in the future. This formula is a quote (I do not remember the source):

lim_{t -> ∞} Math(t) = CS

I hope for this asymptotics:

lim_{t -> ∞} CS(t) = Math

and have a lot of thoughts about how this relates to LLMs and the future of human programmers. These would take too much space and do not belong in this post.

If you look at how most computer scientists approach programming and how most mathematicians approach writing proofs these areas remain very much divorced at the current moment. Even people who know a lot of math and do a lot of programming (e.g. data scientists) are using programming to do math but not use math to program.

“Once upon a time, there was a university with a peculiar tenure policy. All faculty were tenured, and could only be dismissed for moral turpitude. What was peculiar was the definition of moral turpitude: making a false statement in class. Needless to say, the university did not teach computer science. However, it had a renowned department of mathematics.”

John C.Reynolds Types, Abstraction and Parametric Polymorphism (John C. Reynolds) 1983

Coding by experimenting and observing

Consider these tools and processes: debuggers (observe execution of statements), loggers (record observed behavior), testing (observe app behavior and draw general conclusions about app correctness), TDD (pre-define observation outcomes for the code), design patterns⁹ (generalize code observed to work well, create coding protocols). I encourage you to think about tools and processes that are targeting the deductive, there are some!

Trial and error is how a lot of programming is done. We write some code (experiment) and then observe the result using tests, a debugger, looking at the logs, or simply observe how the app behaves ignoring other diligence. Personally, I am a little afraid of experimenting with certain types of code and I try to think through all scenarios, I could miss something but my mental process includes a lot of deductive effort. I have worked with many programmers who operate in a similar way. However, I still need to test the code to observe it working, make changes if needed, and rinse and repeat. Despite some deductive elements, this approach is still very experimental in its nature.

How about working in code that is a complete mess? We make experimental changes and then test / debug / trace the heck out of it¹⁰, right? We may try to reason about what solution is likely to work, but that is not much different from experimental alchemists trying not to blow themselves up. I can recall several projects where all my understanding was derived from debugging or tracing.

Let’s talk about process bureaucracy. Working in an empirical world means procedural protocols. Scrum is a procedural protocol: consider the continuous improvement process with retrospectives or team velocity calculation, these are all very empirical. Test plans, test cases, coding and formatting standards, git hygiene, even design patterns are also procedural protocols. In contrast, the deductive needs a cushy couch. Deductive and bureaucracy, IMO, do not mix well. Waterfall was a failed idea of applying an informal deductive approach to project management. There is a lot of real world complexity in project management and empirical is needed (see also Defined Process Vs Empirical Process). Waterfall reminds me of mocking in unit testing. Both do not work well for similar reasons.

Let’s talk about bias in empirical reasoning. Figuring out contributing factors and causality is often the hardest and the most important part in empirical reasoning. If your wrist hurts when you type, is this a pinched nerve in the wrist? or, are you looking down on your laptop and the nerve is pinched around your neck? or there is a pressure point somewhere in your arm? or maybe it is not a nerve issue at all? Practitioners of empirical have been known to assume wrong cause¹¹. Empirical is tricky.

Programmers are exposed to bias too. Performance issues or a bug could be caused by many factors, e.g. an application could misbehave only in certain scenarios or only in certain environment configurations, the underlying issue could be in the application code, library code, a configuration issue, an environment problem… It is not unlikely for a developer to go down a wrong path during the troubleshooting process. I view programming as an empirical process with accelerated feedback. You can go down a wrong path but you typically learn that fast.

I had a fun discussion with my wife, she is a data scientist working on pharmacological studies. We were discussing if a design of a clinical trial could be adjusted to do software benchmarks. There are some intriguing similarities. Obviously, humans are much more complex than programs and targeted approaches will work better in either domain. But it was a fun discussion and one that convinced me even more about the empirical nature of software development.

Why did we settle on using the empirical method in science? We do empirical not because we want to but because we have to. Empirical is the only way to study the unruly real world. Why did we settle on using the empirical method in programming?

Why is programming empirical?

I will focus on these 2 reasons: nondeterminism and code complexity. Let’s start with the second. An interesting question is how much the empirical nature of programming has to do with arbitrary (artificial) complexity. IMO, a lot.

Experimental process and high extraneous loads

I am framing this in the context of my previous (cognitive loads in programming) post. If you have not looked at that post, simply substitute high extraneous load with messy code or complex code. I think we are in a position to put a 1 and 1 together:

  The experimental nature of programming is a consequence of its high extraneous load and also one of its main causes

The bigger extraneous load the more we experiment. The more we experiment the bigger the extraneous load gets.
To work on a complex code we are effectively forced to experiment. Adding more experimental code only increases the complexity.

This is a feedback loop. To break this loop (and control the complexity) we need to rewrite or refactor parts of the code. These involve some deductive process. Some amount of deductive is essential, a deductive is what can break the feedback loop. Deductive and empirical thought processes are closely related to cognitive loads discussed in my previous post and to controlling arbitrary complexity.

There is one notable exception where piling up experiments does not result in extraneous complexity. It is called type safety.

Empirical FP

FP is considered theoretical for at least 2 reasons: (1) FP considers formalism to be important, (2) how it is being presented to the public¹⁵. But, is FP an empirical process as well?

Consider 2 cornerstones of scientific experiments: reproducibility and properly randomized sufficiently large observation samples, and restrict our attention to automated testing. FP-ers will immediately notice two related concepts: referential transparency and property testing (e.g. QuickCheck). Clearly FP has a lot going on for it if you look at it from the empirical angle.

Whatever your opinion about FP vs code complexity is, you have to agree that FP is about controlling nondeterminism. Functional programming tries to separate out the predictable deterministic part (pure functions) and limit (even stratify) the unruly effectful parts. Functional programming handling of nondeterminism is analogous to empirical study designs trying to control variance. I think that this is why FP seems so empirically-friendly.

Haskell is sometimes called the best imperative PL¹⁶. I propose that FP could become the best empirical programming method (yeah, could I be a little biased). However, it is “could become” rather than “is”. Empirical process benefits from good observability, e.g. decent debuggers, stack traces, rich amount of error information… This is an area where FP could improve in general (Haskell in particular)¹⁷.

partitionEithers :: [Either a b] -> ([a], [b]) 
partitionEithers [] = ([], [])
partitionEithers (Left a: es) = (a: ra, rb)
    where (ra, rb) = partitionEithers es
partitionEithers (Right b: es) = (ra, b: rb)
    where (ra, rb) = partitionEithers es

FP is a hybrid containing both empirical and formal. … But I got sidetracked a bit towards areas of my interest. You can classify this under IMO if you disagree with what I wrote.
Let’s finally get to my main topic: the human aspect.

Pragmatists and theorists

I am using terms theorist and pragmatist somewhat colloquially, but the meaning is close to how the terms are used elsewhere. In this section I will try my best to describe both mindsets. IMO, mindset and interest are very related terms. In this post I could almost use them interchangeably.

If you listen to a functional programmer talk, you are likely to hear these terms: “reasoning about code”, “principled computation”, “computation laws”, “correctness by design”, “type safety”. These people are likely to study things like lambda calculi, operational semantics, category theory, type theory… All these things come with formal proofs and could result in a very specific mental training. To this group programming is more of a deductive process.

These things do not resonate with the vast majority of programmers who have a more pragmatic mindset. You are more likely to hear these terms: “testing”, “TDD”, “hacking” (meant as a compliment, “I made it work”), even “design patterns” (though rarely these days). To this group programming is more an empirical process.²⁰

A theorist’s primary interest is in deductive reasoning. This implies an interest in the deductive itself. Deductive reasoning is often called top-down reasoning (general knowledge comes first). Formal reasoning (the pinnacle of deductive) has a very strong attraction for some individuals while being very much disliked by others. Some programmers dive deep into FP and learn formal reasoning. Sometimes a mathematician (this was the case with me) makes a career conversion and becomes a developer. Interest in the deductive and formal is rather rare, to many programmers formal methods are a foreign concept.

In contrast, a pragmatist’s primary interest is in accomplishing the tasks at hand. Pragmatists will experiment and go empirical to get there. “When the going gets tough, the tough get empirical.” Real world complexity is rarely fully amenable to formal reasoning. Unfortunately pragmatists take a stronger stance on this, a pragmatist may think that formal reasoning is never useful.

I need to emphasize that what I am presenting is an oversimplification. People are complex and cannot be easily labeled. In particular, data scientists, statisticians, programmers interested in probabilistic computing models have a lot of formal training but probably will develop a more empirical attitude towards programming. The attitude towards programming is shaped by many factors (e.g. work experiences, types of programming projects). Theoretical vs pragmatic mindset is IMO a good first approximation.

Pragmatists will say “it works because we tested it”. Theorists will say “it is correct by design” (or closer to Dijkstra: “we tested it and we know it does not work”²¹). Pragmatists will say “this abstraction is too complicated to use because it is too theoretical”. Theorists will say “this abstraction is too complicated because it lacks theoretical backing”. The 2 groups may find it difficult to communicate “this is nice” and “this is terrible” have a very different context when spoken by a pragmatist and a theorist.

Pragmatists want the tools to be mainstream and popular to pass their test of practical usefulness. Pragmatists will select tools that provide good ability to observe how the code runs. To a pragmatist reasoning about code often means studying its execution flow. Some pragmatists go a step further and will only select a PL that allows them the best control over execution (e.g.  JS is used by browsers so this is what they will want to write). This could be caused by a (somewhat justified²²) distrust for abstractions.

In contrast, theorists want tools that support, rather than inhibit, deductive reasoning. They typically want to reason on a higher level than the execution flow. Many will view programs and execution as decoupled concerns²³. This is where the imperative vs denotative discussion comes into play as well.

I consider programming to be a combination of both the engineering procedural processes and math-like science. Pragmatists are interested in the process, they want code standards, formatting standards, clean git history… Theorists will want functors, monads, higher rank types, higher kinded types, dependent types… Programming, obviously, benefits from both engineering and mathematics.

What makes us effective and confident when working with code? For a pragmatist the source of confidence is likely to be test coverage, for a theorist it will be type safety, abstractions, lawfulness. For both confidence implies some ability to understand the code (absorb the cognitive load). To put some of these thoughts in the context of the previous post:

  Theorist typically prefer germane, pragmatists are more at home with extraneous cognitive load
  Theorist typically prefer simple and hard, pragmatists prefer easy and will accept complex

One bizarre difference I have noticed between heavy deductive thinkers and empirical mindsets is their favorite approach to learning. When learning, some theorists will want to finish a section or chapter before writing a single line of code. The top-down thinking sometimes extends to top-down learning (a theorist wants to internalize the theory before applying it). My wife and I are in this group. I had to force myself to write some code early when going through Types and Programming Languages, I decided to create a public github repo to give myself incentive to code as I learn when reading Category Theory for Programmers and T(ype)DD in Idris. Some theorists treat learning as a murder mystery and want to learn ASAP who done it, pragmatists know it will be the butler.

On the flip side, pragmatists prefer hands-on learning from code examples (ideally, associated with project work) and typically expect immediate return on their learning investments. E.g. a pragmatist is unlikely to spend several months studying, say, a one line of code²⁴. Pragmatists are less likely to search for deep understanding (this is kinda definitional, having theoretical interests makes you a theorist).
Also, it seems logical to assume that learning from experience is more habituating. Pragmatists may have a harder time making mental shifts to how they work with code. Unlearning is hard on all of us, I believe it is harder on pragmatists.

The part of FP that has been the most disappointing for me is a typically low quality of error outputs. This may have to do with all falsehoods being equivalent in mathematics²⁵. However, we should not criticize the theory, rather the theorists for selecting abstractions that suppress or confuse error information (IMO). I wrote about it in the previous post and also before²⁶.

In contrast, to an experienced pragmatist error output is (typically) an important observation. It is a pragmatic thing to do to know what went wrong. However, the tendency towards the use of null, Option, Maybe suppressing available information is something I do not understand. IMO, goes beyond the topics we are discussing here²⁷.

While theorists may have a problem engaging with things outside of their theoretical model, pragmatists often have a problem engaging with things that are even mildly theoretical like computational laws or even referential transparency. Programmers often don’t care about computational properties, but will be surprised by software behavior when they are missing. Many gotchas can be described as a “natural” computation property that is violated.

Some theorists may not see a big difference between a prototype and a product that is fully implemented and maintained. This could be related to published equals done in academia and can be quite annoying.

Probably, the most interesting difference between both mindsets is:

  A bug means an observed malfunction to empirical pragmatists
  A bug means a logical flaw to formal theorists

In empirical science, rare is tricky. Rare can escape the empirical process. The term outlier, used in statistics, is relevant. The term “zebra” is used by medical doctors, “zebras” are hard to figure out, rare cases. Empirical reasoning sometimes equates rare with impossible. When dealing with real world we say:

  “the exception that proves the rule”
  In formal reasoning exceptions disprove the rule.

In software engineering an exceptional, rare event often stops being rare (e.g. when the data or usage changes expose it). I fixed many bugs caused by a programmer’s decision that a certain scenario is very unlikely and, thus, it is OK to cut corners. In my experience, ignoring rare scenarios saves hours and ends up costing weeks or even months later.

However, I am not convinced that cutting corners is unique to either mindset. A pragmatist may think “this is so unlikely, I will not waste time on it”. The rare case could be not represented by the theoretical model that a theorist is considering (e.g. error information in a bunch of FP code). IMO, pragmatists and theorists cut corners in different ways, however, cutting corners has no place in formal reasoning and I expect some programmers (most likely a subset of theorists) carefully think through rare cases.

Theoretical and practical mindsets are antipodes of programming. Even for people who “own” both, being a pragmatist or being a theorist is like wearing a hat. You can’t wear both at the same time, you would look ridiculous. I started making a conscious effort to understand which hat I have on.

This post argues that both traits are important. We will dig deeper into both ways of thinking by analyzing some examples.

Conversations

So far, this post has tried to upset people on both sides equally. This section will be biased towards theorists, it is hard for me to present the pragmatist’s viewpoint this way. Engineers are some of the cleverest people on this planet. If someone rewrote this section from an engineer/pragmatist point of view, I would love to read it.

Perhaps not surprisingly, theorists are not all equally disappointed about logical software defects. Explaining this diversity is a price I have to pay for bundling all theorists together. Think physics, it is very theoretical, yet theoretical physicists are happy to do hand waving arguments. In contrast, there is no hand waving in mathematics. IMO, this stricter view plays a role in how some of us approach programming.

Alice: “We have a concurrency issue in our code”
Bob: “Are you talking about a production issue, a failing test, or is it purely theoretical?”
If Alice gets a change to explain the race condition, she may hear this response:
Carol: “We did it like this before and everything was fine”.

Alice (nick name Negative Nancy, a theorist) considers all logical defects to be a disappointment. Bob and Carol are pragmatists and approach logical defects in a more relaxed way. It seems like a good idea for Bob and Carol to understand a little bit about how Alice approaches programming, and vice versa. Let’s analyze this dialog a little bit.

To Alice logical issues are kinda a big deal. She will consider it very hard to reason about code sprinkled with logical flaws. There is actually a good reason for this. Some logical flaws we examined in my previous post²⁸ are quite isolated but it is hard (if not impossible) to understand the full impact of many of them. I am like Alice, working in a complex imperative program or a poorly written functional code results in me forming a large mental repository of issues and their unclear impacts. Maintaining it is a tough mental effort.

Here is a story from my personal experience. A few years back I did a code review session with two (very capable) developers. I showed them one of my “bug stashes” in the project we all were contributing to. It had to do with a logically brittle use of inheritance. I demonstrated the process I go through to verify the brittle bits. This session was very productive, we all learned something from it, and this code was refactored later. Their response is something I still contemplate: “We do not go through such steps, we just assume it will work”. For me it was a learning experience I still think about, it made me realize how different our mindsets are.

Returning to Bob, he is a pragmatist. Notice that Bob has stratified all contexts he assumed relevant to Alice’s finding: production issue, failing test, and theoretical. To Bob, a logical issue in code is just a part of life. “It has bugs, it’s called software.” This empirical mindset, in some ways, is healthier²⁹. It is not unusual for empirical reasoning to dismiss theoretical concerns, however in this case this is likely to be wrong. It is hard to spot or even assess the impact of some bugs (e.g. race conditions) using testing or other observation based methods³⁰. The concurrency flaw Alice has identified can start manifesting itself all the sudden, this is what concurrency issues have been observed 🙂 to do.

Carol’s response suggests an empirical mindset as well. Carol has generalized previous observations of a working product and that generalization overrides Alice’s warning bells. Going from specific to general is what empirical process is about. Proper empirical reasoning will question, even invalidate, previous “hypothesis” if new evidence provides reasons for doing so, but Alice is not providing any empirical evidence. Alice’s argument is purely deductive, it could be helpful if she came up with a test that exposes the concurrency problem she has identified (this could be, obviously, very hard or even impossible to do).

Bob: “We are starting the new frontend project, I propose we keep using XYZ PL, but maybe we could add a new library to our setup?”
Alice: “XYZ is fundamentally broken, we should move to something sound, like Reason.”
Carol: “Alice, we do not know that PL, this will put the project at risk!”
Alice: “We know so many problems about XYZ, XYZ puts the project in jeopardy too”.

You may be wondering why I call JavaScript XYZ? Well 🙂, XYZ was intended as a placeholder. Alice would like to use tools that help, not inhibit her deductive process and are logically sound. Alice has witnessed her colleagues (and probably herself) trip over XYZ unsound design numerous times. She considers the use of XYZ akin to building a house on a broken foundation.
Bob and Carol insist on tools that have good IDE support, good debugging, and are familiar even if logically unsound. Pragmatists want to reason about execution flow, browsers use JS, thus, they may want a language very close to JS. Carol has a very valid point too, one that Alice may have hard time accepting. I admit, I sympathize with Alice, even if Carol is probably right here.

Alice: “You are memoizing a computation that is not referentially transparent”
Carol: “I have manually tested it and, besides, this code has a 100% test coverage”
Bob: “We are assuming that it is referentially transparent and want consistent results when we use it”
Alice: “Remember we patched a bug by updating shared state in the middle of this computation, did you retest this scenario?”

This code review session shows a benefit of having someone around who keeps a repository of potential issues in their head. I have noticed that developers are typically surprised when computation behavior keeps changing, yet are mostly not willing to engage with the concept of referential transparency. I also think some do not think about what 100% test coverage implies and what it does not³¹.

Bob: “I changed the interface, you can now pass new parameters to control how the data is processed”
Alice: “I changed the module, you can now use new functions (combinators) to manipulate the data”

Functions are great for reasoning about code, parameters are great for tweaking and experimenting.
Alice’s deductive approach can really be beneficial when writing code, probably more than when troubleshooting empirically implemented code. Let’s get a little philosophical:

Alice: “Ideal code to me is one I would still be proud of after 10 years”
Bob: “If you think about your 10 year old code as perfect, you learned nothing in these 10 years”
Alice: “I am looking for something as timeless as mathematics”
Carol: “Mathematics keeps improving and changing I am sure, everything does”
Alice: “No, it only grows, it has not changed its mind in over 100 years”

This is almost an exact copy of a conversation I had with some of my coworkers. The immutability analogy I have used before works well here: mathematics is immutable while empirical sciences mutate in-place (some theorist hold this view). Bob’s argument is partially valid as there is a lot of engineering going into coding and that is likely to keep changing³². Also, formal verification has a maintenance cost³³ making me question how realistic Alice’s dream is. Can you think about code examples that aged very well?

These conversation examples were not intended to be exhaustive. I invite you to think more about the differences between pragmatists and theorists in both creating and consuming the code. I invite you to think more about how each side thinks.
I need to emphasize, this is not a binary separation where everyone is either pragmatist or theorist. Many of us have both traits, just one is more dominant than the other and they tend to not manifest at the same time.

Negativity

There is quite a bit of negativity around us, the programming community is not exempt from it. I will wrap up with some loose thoughts about negativity in the context of deductive and empirical mindsets.

Venting frustration, IMO, does not improve how either the “venter” or the “ventee” feel. It only fuels the negativity. IMO, the best way to fight negative emotions is to employ the deductive. Figuring out the underlying context that causes negativity can save a conversation and simply engaging in that search can protect you. Using logic to confront emotions is a form of what psychologists call cognitive restructuring.

Criticism of bad code is bound to be unpleasant to its authors. Even worse, a lot of code can be criticized on purely logical grounds. This can be interpreted as a critique of the author’s competence and, I am sure, is a big source of negativity and tension. Let’s look at this differently. Look at some of the empirical sciences like neuroscience and see how much of the knowledge got adjusted if not invalidated³⁶. IMO, programming is on a very localized and very accelerated path of the same process. Is “bad code” phenomena partially related to the empirical nature of programming? I think it is³⁷.

Theorists are likely to devote a significant effort into learning. How does it feel to not be allowed to use what you worked hard to figure out? Examples like How to stop functional programming come to mind. This is something that can be changed on a small scale but there are only a few places who care to do so. There must be quite a few frustrated programmers out there³⁸. I have been in that position and I know it is mentally hard. I have argued that the deductive process plays an important role and, IMO, it is in the interest of the industry to treat the “theorists” minority better.

I have promised at the beginning that I will not try to make too many “points”. This section contained the exceptions that prove the rule 🙂.

Final thoughts

Did I sneak in any other side-“points” worth noticing? IMO, this one (if you agree that formalism is less prone to errors):

Strict backward compatibility implies a need for formalism
or, equivalently
Empirical, immortal, immutable, sanity: pick 3

The odd discourse between formal and empirical is not unique to programming. I still remember a few jokes about “a mathematician, a physicist, a chemist, …”. Empirical vs deductive, if one needs to budge then the empirical wins, mathematics has to move. Logical correctness is often more a guideline rather than a shackle. Some (mostly functional programmers) argue that computer science and programming could benefit from a stricter application of formalism. The difference is where the source of truth is. In a pure empirical world that source has to be what we observe. In programming there can be just enough determinism to benefit from treating logical soundness concerns more seriously. I believe, we need both empirical and deductive and I hope I made a convincing case for us to try harder for their peaceful coexistence. We can start by trying to understand the other viewpoint even if we do not agree with it.

Thank you for reading!

Unexplored

We can learn a thing or two about programming if we think about it as an empirical process. We can learn a thing or two about the empirical process itself if we examine programming as a case study. We did a little bit of both in this post. It seems that this synergy can be explored more.

Impact of education on the development of either mindset.

Related psychology, evolutionary biology: Humans survived and evolved by “observing” things and acting on these observations. Empirical process is in our nature. This also explains why we dismiss rare scenarios. I do not feel qualified to discuss these in more depth.

As we have discussed, developers approach bugs differently. This is how my interest in figuring out different programmer mindsets has started. There is a different way to look at this. Consider these 3 axes: “It has bugs, it’s called software” is the origin, testing is one axis, “correctness by design” abstractions and type safety is second, a mental repository of possible issues and their impacts is third. Pragmatists are on the first axis, theorists on the second. We need the name for the third group, let’s call them perfectionists. I came to FP on a correctness wagon, theory and improved coding efficiency are for me an added bonus. The question is how the perfectionists fit into this picture. This post bundled them with theorists, this was likely an oversimplification.

Implicit contexts in communication between programmers. There appears to be much more to explore here. Consider programming internet discussion forums (IMO, a Manhattan of communication skills, if you can make it there you will do really well in your project team). One can observe all kinds of context related bias issues (on the extreme end of this, some redditers do not consider reading to be a prerequisite to responding) or lack of context clarity (e.g. heavily downvoted posts with no comments). Moving away from discussion groups, teams tend to create their own localized contexts (unique vocabulary, proprietary technical solutions) which is often not ideal. IMO, context clarity is to communication what referential transparency is to programming. IMO, context is to communication what understanding of causation is to empirical science.

Program synthesis, you know the thing that is going to render programmers obsolete: It appears that empirical vs formal translates to 2 different approaches to do program synthesis. I have listened to a few presentations about the formal approach. I expect probabilistic models, deep learning approaches to be more mainstream and prevalent. It is interesting how this will play out, but I do not feel qualified to discuss this in much depth. “The effort of using machines to mimic the human mind has always struck me as rather silly. I would rather use them to mimic something better.” (Dijkstra, of course). Quite possibly we will see an equivalent of empirical vs deductive play out in this area.

Abductive reasoning and applicability of Ockham razor to software development seems like another fascinating philosophical topic that is orthogonal to what we have discussed³⁹.

LLM’s like ChatGPT cab be viewed as the ultimate “inductive learners”. Does this create opportunity for human minds capable of a more deductive thought process?

I am sure there are many more interesting angles to explore here.

  “My brain hurts”, a quote from a code review

This long post presents programming in a different light than what is commonly considered. We will look at cognitive aspects of interacting with code.

We will examine cognitive effort that goes into the implementation and cognitive loads on these poor souls who need to work on that code later. We will consider the programming language, its libraries, and implemented programs as instructional materials. We will view the developer as both an instructional designer and a learner. We will think about bugs as cognitive overload and a missed learning opportunity. We will discuss the cognitive impact of abstractions, types, and programming principles.

Cognitive load of working with code is rarely considered in actual project work. We ask “How long will it take?” (in fibonacci numbers, of course), we do not ask “How will it impact the overall complexity?”.
I had quite a few eye opening moments when thinking about these topics. This is the main reason I decided to write and share my thoughts. This post will be a high level rant discussing programming across the industry spectrum from JavaScript to Haskell. It is written as a set of loose notes about various cognitive aspects related to working with code. The main goals are to:

• show how considering cognitive loads in context of programming projects provides valuable insights
• present some useful terminology for reasoning about code complexity.

I will try to explain psychological terminology but this post assumes readers’ (high level) familiarity with concepts of FP and OOP.

My pet peeve is identifying specific patterns of erroneous code and what could be causing them, there is a human factor and a technical part to these patterns.
Mental processes involved in writing code are such a fascinating and broad subject. I am taking only a very narrow path through it.
I am planning another high level post to discuss programming from a different but relevant angle, it will be about empirical and deductive aspects of working with code. I believe these 2 aspects impact our cognitive loads in interesting ways. So, this post will focus on cognitive challenges caused by code. The next post will focus more on the human aspect.

This post reflects on my personal observations accumulated over 27 years of professional programming work, augmented by a few years of academic teaching.
I am not a psychologist, these are observations of a coder.

That dreaded YAML

I am perusing thousands of lines in Infrastructure as Code (IAC) yaml files. I am looking at an already refactored and improved version. It is a lot of templated YAML of k8s configuration at my work. The underlying reason for the complexity is the PL itself¹. Did the refactor break things? Of course it did. Complexity has consequences. With some effort, the issues were fixed. This is how things are, there is nothing we can do about it. There isn’t?

I want to contrast YAML with a configuration language called Dhall (one of my favorites). To use Dhall you may need to adjust to a Haskell-like syntax, maybe learn a few new concepts (like ADTs), think about configuration that uses lambda expressions. The return on the investment are Dhall safety features. Dhall even makes the process of refactoring safe, you can compare the previous configuration against the new and Dhall will tell you if both are equivalent or why not.

Dhall and YAML come with very different cognitive challenges.

Cognitive psychology

Cognitive load theory defines cognitive load as the amount of information that working memory holds at one time. The idea is that the human brain is limited in that capacity. Psychologists have identified the load to be about 3-5² “units of information” (also called “chunks”). This space appears to be quite limited.
I imagine the magic number is small in programming. However, I expect it to vary between individuals.
If we can load only a limited number of “chunks” into working memory, how big can these chunks be? The answer is interesting: it seems that it does not matter!³
In some situations, the magic number appears to be 3 (the concept + 2 constituent chunks)⁴.
Notice, it would be hard to enumerate chunks involved in a classic imperative program, but that number will be >> 5.

The idea of decomposing the program into fewer (but bigger) “chunks” that interact in a clear way has been around for as long as I can remember. We will examine this idea in terms of Cognitive Load Theory.

Cognitive Load Theory is concerned with instructional design and improving how information is presented to a learner. Controlling the learner’s cognitive loads is an essential part of this work.

Continuous learning is a part of what programmers do, but implementing and modifying project code is by far the biggest cognitive effort that programmers face.
I look at this as: the code itself is a very important instructional material, programmers are learners and instructional designers at the same time.
Programs are where the presentation of information happens. The concepts and findings of cognitive load theory seem still relevant after this adjustment.

Cognitive psychology considers 3 types of cognitive load: Intrinsic, Extraneous, Germane. All are related to information presentation and we will think about them in the context of code.

• Intrinsic cognitive load is the inherent level of difficulty associated with a specific (instructional) topic. Thinking about code, requirements are a good choice for a topic. A rough moral equivalent known to software developers is essential complexity (things are complex because they are, to reduce this load requirements would need to change).

• Extraneous cognitive load is generated by the manner in which information is presented to learners and is under the control of instructional designers. This term is often used to describe unnecessary (artificially induced) cognitive load. Thinking about code, a rough moral equivalent of high extraneous load is accidental complexity⁵ (things are complex because the program made it so).

• Germane cognitive load refers to the work that is put into constructing a long-lasting store of knowledge or schema. Schema is a pattern of thought or behavior that organizes categories of information and the relationships among them. Psychologists also use the term “chunk” and schema construction is the process of creating these chunks in memory.
  Thinking about code, this roughly translates to using abstractions, higher level concepts, types, programming principles. An OO programmer may try to define intuitive object hierarchies, employ design patterns to model the business domain. An FP-er may use denotational⁶ approach, look at how things compose (think about categories), design DSLs, blue-print the design using types…

  Cognitive load theory thesis is about reducing extraneous cognitive load redirecting it towards germane load.

Cognitive theory considers intrinsic load to be not movable, obviously requirements can be changed.

I need to emphasize that the information presentation under consideration facilitates understanding of the code itself and not so much the concepts (e.g. abstractions) used to create it. Knowledge of these concepts is a prerequisite. Prerequisites are an important caveat and one that ends up being contentious.

This sets the stage for what I want to discuss, but before continuing let me briefly review a few more relevant concepts.

Cognitive overload happens when working memory is overwhelmed by the 3 cognitive loads we have described, IMO, bugs are evidence of a cognitive overload.
However, psychology is not a simple arithmetic, some programmers learn how to process large cognitive loads sequentially, a few chunks of information at the time and get good at it. However, high cognitive loads will overwhelm even the most diligent among us, There is even a trivial combinatorial complexity to this (does working memory go through a binomial number reloads?).

I wanted to use cognitive debt in the title, intending it as a pun on “technical debt” because I am interested in discussing negative impacts on the team’s ability to understand and reason about the code. However, this term turns out to have a clinical meaning and I decided against using it.
Cognitive debt is a psychological term associated with repetitive negative thinking (RNT). Cognitive debt and RNT are hypothesized to have some very negative health consequences that can lead to depression or even dementia. RNT is described as

  “excessive and repetitive thinking about current concerns, problems, past experiences or worries about the future”

I do not claim to know a lot about clinical psychology but the definition clearly is very relevant to programmers and could partially explain why programmers are often unhappy⁷, or why programming discussion groups are often very negative.
Sadly, RNT seems to be a condition that really good programmers are likely to experience. Good programmers think about rainy day scenarios, notice design flaws, can anticipate program issues… My pet peeve, patterns of erroneous code, is an RNT. It seems important that we talk about RNT.

You may want to think about working memory and cognitive loads as something akin to RAM in a computer.
What is the equivalent of a CPU cost? In this post I use cognitive effort, the usage of this term is not very consistent⁸ in the literature.

These were cliff notes written by a non expert. There are many tricky and relevant bits like information retrieval from long term memory. Things I am progressively less and less competent to describe in psychological context.

Easy vs Simple

If you have looked at my TypeScript types series, you have seen me write about it already. I do not claim that my definitions are the only correct way that these terms should be interpreted. However, I have seen other programmers use a similar disambiguation so I am including it here.

I consider the terms simple and easy to have a different meaning.
Easy: A low barrier to entry (e.g. easy programming concepts). Hard is the opposite of easy and implies a high learning curve.
Simple: Low effort to correctly reason about (e.g. code written using learned concepts). Complex is the opposite of simple (e.g. code that is hard to understand). The term “arbitrary complexity” fits this definition very well.

Easy means fewer prerequisites and implies low germane load, hard means many prerequisites.
Simple means low extraneous load, complex means high extraneous load.

This differentiation could also be expressed as:

  Easy means low cost of creation, simple means low cost of consumption

except, in this post my interest is the cognitive effort only not the total cost⁹.

Achieving simplicity on a larger project is not easy. Easy does not scale well. There appears to be no free lunch, cognitive load needs to be somewhere. My big preference is trying to achieve hard and simple rather than easy and complex. In other words, I prefer to spend my cognitive bandwidth on germane load over extraneous load. This, I am pleased to note, is aligned with cognitive psychology.

Recall the advice from cognitive psychologists is to reduce extraneous load redirecting it towards germane load. This translates to:

  Move from complex to hard

An interesting way to look at easy vs simple is to think about a creative process like writing a book. Easy to write is clearly very different from simple to read. In programming these two are bundled together, a program created by one developer needs to be consumed by another dev who needs to modify it or interact with it. The term “readable code” comes to mind. I consider it different from simple. E.g. readable code does not mean no subtle bugs. Message is readable if you know what it conveys, but what it conveys could be complex or even misleading.

IMO, the popularity of easy and the unpopularity of simple are a systemic problem in today’s programming and elsewhere.

Next section discusses examples of code which was intended to be easy and ended up complex.

  “Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” Dijkstra

Extraneous loads that grow

What was the most complex code you’ve worked on?
I can think about a number of contenders, but my answer will be very unimpressive: I had to maintain a web page (just one page), it was implemented using Java Struts 1. This code used no advanced concepts, all ingredients were easy: control statements, instance variables, lots of protected methods with typically no arguments and void returns that read and set the instance variables.
The Java class behind it had about 200 mutating instance variables. Changing the order of 2 lines in this code was almost guaranteed to create an (often intermittent) bug, it was hard to even make sure that variables were set before they were read.
This code became infamous for its complexity very fast. Interestingly, Struts were blamed, not the needless overuse of mutable state.
I want you to channel your inner psychologist and answer this question: what is going to happen when a new functionality is added to a Java class with 200 instance variables? Right, I agree, we will have 201 instance variables.
This piece of code was eventually refactored. If I remember correctly, about 12 instance variables were kept, they were actually needed by Struts.

This experience seems to me a good example of a big extraneous load, I had to deal with a load of 200 coupled “chunks”.
Let’s think about such code as an instructional material. I can attest, it was virtually impossible to even know what this code is supposed to do from looking at it.
Ability to program using clear inputs and outputs (rather than void methods with no input parameters) requires a learning effort, I submit to you that this prerequisite is easier than the cognitive effort of maintaining such code.
Thinking about this as instructional material, clear inputs and outputs are great learning objectives. You know the app if you understand its inputs and outputs.

My second example is something that happened more recently. I worked on reimplementing a JS application. It was one of these apps that can be described as: was easy to write, is hard to maintain or understand. I am sure very few readers will be surprised by the existence of a hard to maintain JS application, but let’s put talking about this aspect aside. Is writing “easy” code the same as generating excessive cognitive load for the maintainers? I think it typically is, it is not that hard to incrementally develop a non penetrable maze. Maintaining some code structure to manage the cognitive load is not “easy”.
The new version is still close to JS (it uses TypeScript and vanilla React) but tries to enforce these 3 principles: referential transparency, clear, explicit types that also work as documentation, and async/await abstraction to avoid callback hell.
Referential transparency is an interesting dichotomy. Experiencing different results every time the code is executed typically causes surprise, in my experience developers rarely think about this during implementation. Thus, the code may feel weird and opinionated (e.g. React components avoid using hooks) but it remains accessible.

  IMO, high quality code shifts cognitive load from maintainer to implementer

This works great even if both are the same person.
Let’s consider the new JS app as an instructional material. Referential transparency creates learning objectives (inputs and outputs can be learned if outputs are predictable) while explicit types are an instructional material in itself (a blueprint). The biggest prerequisite for the implementers was knowledge about what to avoid.

Besides some common sense principles (feel free to add more), what else can we do to control extraneous load? Things are about to get more tricky.
Human cognitive load is limited but we can do abstract reasoning. It is simpler for us to deal with a few generalized abstractions than with a multiplicity of concretes¹⁰. And, as we know, abstractions are a better use of our working memory chunk space.
This suggests exploring the space of programming abstractions. Unfortunately, programming abstractions are nontrivial. That makes them hard to learn, but what is worse is that developers and language designers sometimes (if not often) mess them up. Instead of decreasing, this increases (or even explodes) the cognitive load. We will explore this topic in Extraneous nature of abstraction.

Types are, obviously, an important tool in controlling cognitive load. Types offload many code verification tasks from the developer. This is significant, developers can ignore a potentially high extraneous load of a program by trusting its type. As I already mentioned, types can be an instructional material, a blueprint. Using a type checker can, in itself, be an interactive learning process (e.g. using REPL to ask type questions about the code).
However, types are subject to similar limitations as abstractions: a learning curve, PL limitations, correctness issues (only, we call it soundness if types are involved).

Reducing cognitive load using abstractions and types is doable but requires navigating some tricky waters.

Bugs and metacognition

Let’s define a bug as an unintended program defect. That removes all the temporary hacks from consideration. But it is the programmer’s job to figure these things out. A bug implies some issue in the mental process.

I consider cognitive overload to be the main cause of bugs. Metacognition is an important concept in cognitive psychology. It is about knowing strengths and weaknesses in our own cognitive process.
I started analyzing and recording all defects I encounter at my work. My goal is to understand better what has caused and what could have prevented each issue. My records suggest that bugs uncover extraneous complexity. In other words, it is a good idea to ask this question: What is the underlying complexity that caused the developer to create this bug? The idea is to learn from bugs.
Types, obviously, can be very helpful in bug prevention. Programmers who start using a PL with powerful types (e.g. Idris, Haskell) experience this first hand: a lot of compilation errors, many uncovering an issue in the program. Notice, this is a very interactive process and an interactive learning experience in which developers can observe how and why they failed. Developers also observe what PL features prevent the bug from escaping.

  Programming is an interactive process of finding and fixing bugs.
  IMO, programming should be an interactive process of identifying and resolving the underlying causes of bugs.

“Insanity is doing the same thing over and over and expecting different results”. I promise you 2 things: When you start analyzing bugs, you will start seeing patterns (similar to patterns of erroneous code). Unfortunately, you will likely have problems in communicating these patterns to developers who do not go through a similar process. I found that a code review session showing the same issue in a few places works better than trying to explain this without a concrete context (oops).

How about typos, trivial overlooks that are sometimes so hard to spot? That mysterious brain of ours is good at creating these. A great reading on this, in the context of (non-programming) typos, is WUWT, Why It’s So Hard to Catch Your Own Typos.
Human brain has an ability to fill in gaps, self-correct things. Human brain is better at focusing on high level ideas and is perfectly happy skipping over minute details. This characteristic seems even stronger if we are on board with the big idea, and it seems fair to assume that programmers are on board with the features they are implementing. The main point is that our brain is not well designed to work at the level of statements and lexical tokens, it wants to work on big picture items.

Error proneness of programming at the level of PL statements is also consistent with the cognitive load theory. At this level a programmer needs to consider a multitude of details, most likely overwhelming the working memory limits.
An interesting piece of trivia is that Turing’s original paper (the one about universal machines and halting problem) had several bugs in it. If Turing could not get it right, what chance do we have?¹¹

Static compilation can prevent a lot of trivial errors and hopefully the prevented list will grow, but that list is not exhaustive.

Section Summary

My first point is that programmers should start considering cognitive aspects when thinking about bugs.

What is that we do when we discover a bug? We write a test, right? Does this reduce the cognitive load? Of course it does not. IMO, it is more important to spend time on some intro- and retrospection and look for ways to lower the extraneous load or build some type safety. If that is not possible, improving test coverage becomes important. I want to learn from bugs. Fixing bugs is the least important part of the process. I should also mention that this is, unfortunately, a repetitive negative thinking territory.

Here is an example that keeps popping into my mind when thinking about trivial errors. I have seen many stack overflow errors in my life, I have seen only 2 or 3 since I moved to Haskell but they were not easy to find. They all were caused by Haskell allowing this lambda expression:

let blah = blah 
in blah

This to me is a good example of extraneous complexity that could be prevented by the compiler. Many PLs (e.g. anything in ML groups like OCaml, Reason will not allow such code). Here is a relevant discussion on reddit: NoRecursiveLet. Thinking about reducing extraneous load could impact such discussion (in this case, by supporting the proposal).

My second point is the recurring one, types and abstractions can play a big role in reducing bugs. Hopefully types and abstractions themselves are bug free!

Extraneous nature of abstraction

Summary of previous sections: Our cognitive load is limited but we are capable of abstract reasoning and can work with big chunks of knowledge. Abstractions seem like our best hope in reducing the overall code complexity. But …there are a few caveats.
Programming abstractions are known for their germane load (for being hard to learn and for being less straightforward than imperative) but not so much for their extraneous nature (for being needlessly complex), the second aspect is much more interesting so let’s discuss it.

Poorly implemented abstractions

You spotted an intermittent malfunction in a code you maintain. Luckily, you see only one commit in recent history and you have a strong hunch something is wrong with that commit. Only some 50 code changes. The one that caused the issue is: var1 == var2 changed to var2 == var1. Would you be able to spot it? I call this type of issue a “gotcha”.
How about: your finder function seems to be not finding stuff, only that sounds too far fetched, the function looks correct, so you just ignore this as a possible explanation. The underlying issue is that sometimes x =! x and you have used an equality check to find things.

I like to think about this paraphrasing Gimli:

  “Computation Laws are upon you, whether you would risk them or not.”

Equality is an example of an abstraction developers implement and use, but not think much about. However, the list of surprising behaviors like these is quite long affecting all kinds of abstractions. Gochas create chaos in the cognitive process. Gotchas often become mystery bugs and are resolved using workarounds.
For abstractions to work as a cognitive load reducer, they need to be treated seriously by the implementer.

Developers I talked to often responded to such examples by saying something like: “This is just bad code, whoever implemented it should have been more careful”. Except, I can point to examples in standard libraries of popular mainstream PLs or popular frameworks¹². The issues come with no deprecation warning and, if documented, are considered a ‘feature’.
Are questions like “does a developer have a fighting chance of troubleshooting this feature?” even being asked?

_side_note_start
Poorly implemented abstractions could have to do with the empirical mindset that dominates the programming community. Programmers are typically inductive learners (learn from examples and generalize from examples). It is probably hard to think about counter examples when learning from examples.

Abstractions themselves causing cognitive issues

OOP creates a very high cognitive load, to a point that even compiler writers mess it up all the time¹³. I started my programming career as an OOP enthusiast and evangelist. OO programming has an appeal of simplicity and I was seduced by it for many years. It took me a long time to realize that OOP is not simple at all. Let’s talk OOP a little. Pick random training. You will probably learn that Cat is an Animal and that everything is intuitive.
You will not learn if any of these less obvious are (or should be) true:
  function accepting a Cat is a function accepting an Animal
  array of Cats is an array of Animals¹⁴
  function with no parameters is a function with one parameter¹⁵.
You will not learn about reduced type safety that comes with widening to a superclass¹⁶. I don’t even want to start on subtyping gotchas of variant (union and sum) types. OOP is approachable only because we hide the complex bits from the learners¹⁷. A relevant psychological concept is a cognitive bias called framing effect.

The concept of exception (i.e. throw and catch game) is another example of a risky complexity that impacts even Haskell¹⁸.
Types can reduce cognitive load of understanding the code, except exceptions provide a very accessible and virally used way to bypass the types. Other “bottom” types like null are in the same boat. In my experience, many developers turn a blind eye on error handling in general. This seems akin to omission neglect (psychological concept loosely described by this popular phrase: out of sight out of mind) and some optimism bias (focus on sunny day scenarios only).
I really like what Rust has done in this regard, you can panic but it is hard to recover if you do, otherwise errors are handled in an Either-like sum type called Result.
Hopefully we will see more of this pragmatic approach in future PLs.

You may notice that the examples of gotchas I am coming up with have something in common. These issues can be classified under: not trustworthy types. Misleading types will confuse any developer, that includes developers who work in dynamically typed languages and may not think about types explicitly.
We think in types more than we realize.

Are there any “gotcha” free environments? Haskell comes close but is not perfect¹⁹. Proof assistants like Idris come to mind, you get very sound abstractions, and these can even verify totality. That is kinda interesting, let’s pause for a bit here… Consider the levels of abstraction used in proof assistants. It appears that our brain needs something at the level of a dependently typed lambda calculus to work correctly²⁰. That could make sense, for things to be logical you need, well you need the logic itself. Proof assistants are not “gotcha” free though, they have different types of gotchas²¹.

Wrong abstraction for the job

Let’s talk about data structures a bit. The choice you make can impact extraneous complexity a great deal. An example which emphasizes this is CRDT.
Imagine that you are working on an app where 2 or more agents (human or not) can concurrently work on some list and your program needs to merge their work. Using a standard list type will be a cognitive nightmare, right? Think about one agent (R) removing items, another agent (A) adding items. How do you know if an item was removed by (R) or (A) just added it? So what do you do? You introduce some distributed locking mechanism? …Things are becoming complex very fast.
The choice of which data structure is used can have a big impact on extraneous complexity. This extends to other abstractions as well.

High levels of abstraction, an extraneous aspect

I have seen very abstract code where the abstraction was like trees preventing developers from noticing a forest. One source of such examples is error handling. Mathematics rarely concerns itself with error messages, falsehood is falsehood. I have blogged about it in my posts about Maybe Overuse and Alternative and errors.

Let’s focus on Haskell. One simple to explain and not very abstract example that still fits into this section is the guard²² combinator. I see it used and I also scratch my head when, say, a JSON parser error says only "mempty". Possibly, some programmers think about the abstraction called Alternative when they should be thinking about something like MonadFail, an abstraction that allows to specify error messages.
Abstractions seem to come with what psychologists call a commitment bias (hey, I am doing MonadPlus damn it!).
It is us, not the tooling, Haskell ecosystem offers a very expressive variety of abstractions. E.g. consider the error handling blind spot we talked about earlier. You can think about Either as a Bifunctor or an ArrowChoice argument, what typically gets our attention is its throw and forget Monad semantics.
Some of us really dig abstractions and are arguably very good at them. But we are kidding ourselves if we do not acknowledge that abstractions can also blind us.
IMO the one thing we can do about it is to be aware. More diligence + awareness is typically all it takes.

Section Summary
Some developers react to gotchas with something akin to omission neglect, while other developers appear to create a mental store of gotchas and their potential impacts. I am in the second group. Maintaining this store is not necessarily easy. I will also note a possible relationship to repetitive negative thinking.

Gotchas presented to us (thank you very much) by language designers or library implementers should technically be classified as intrinsic since a common bloke like me can’t do much about them other than look for a job that has a better tooling. If you look at programming as a whole, these are extraneous loads.

I have left the subject of abstraction vs imperative (abstractions being less straightforward and harder to map to actual execution) untouched. I plan to return to this in my next post.

Germane and intrinsic load of FP

I expect that nothing in this section will be surprising to a functional programmer, but FP has such a unique cognitive impact that it is hard for me to not talk about.

Functional Programming allows us to understand computations in ways that are not possible without FP. Understanding is a big cognitive simplifier²³. We are more at home with things we understand than with things we just know. Realizing that computations is something I can actually study to understand has been a game changer for me as a programmer.

Consider the following (middle-school?) formulas and how they relate to programming:

a^(b + c) = a^b * a^c
a^(b * c) = (a^b)^c

These, pattern match and currying formulas, suggest that computations relate to other things we already know in ways that are almost surprising²⁴.
From the cognitive load theory point of view, an ability to map to existing knowledge needs to be viewed as a big plus (and a missed opportunity in how we learn programming).

FP is hard and there are 2 reasons why. One: it is simply hard (has a decent surface area but is also deep), two: it is different.

I was learning FP while working as a Java / Groovy developer. It took me 8 years, I estimated about 7000 hours. This effort included Category Theory, Types (my main interest), PLT, and programming in a bunch of FP languages. This has been, obviously, a big personal investment. And, I still had to internalize a lot of this when I started my actual Haskell job. Please do not interpret these stats as an argument that FP cannot be learned incrementally, or that learning FP does not provide immediate benefits. I am including these personal stats as evidence of an overall effort but also as evidence of the multitude of learning opportunities. We should resist thinking about knowledge as a binary checkbox.

FP requires a shift in how developers think. This shift is especially hard if the developer can only practice imperative skills at work. The tools we use impact our cognitive function.

  “It is not only the violin that shapes the violinist, we are all shaped by the tools we train ourselves to use, and in this respect programming languages have a devious influence: they shape our thinking habits.”

The quote is from Dijkstra letter to The University of Texas protesting their Haskell -> Java curriculum change. If you are into technical sports, you may have heard the term “muscle memory”. It is often harder to unlearn or adjust a body movement than learn a new one from scratch. It is even harder to “own” the old movement and the new movement at the same time. Psychologists also believe that unlearning is hard²⁵.
The required mental shift for FP is the source of all kinds of additional problems. It can form a communication barrier, it can divide the community and teams.
At the same time, this cognitive shift is an opportunity to understand programs in a different way.

I will dig my hole a little deeper. This one line of code made a huge impact on me (it is called the Free Monad and is in Haskell²⁶):

data Free f a = MkFree (f (Free f a)) | Pure a 

I decided to dedicate a full summer to learning this line and it ended up taking longer than that. There is actually quite a bit to learn and understand here!
For example, how does it relate to this line (looks very similar, just replace Free with Fix and drop one constructor):

newtype Fix f a = MkFix (f (Fix f a))

Or, how is this a monad? Does it satisfy monad laws? What does free mean? Can other things than monads be free? Can Free-s with different f-s be combined? If so, are there easier and harder ways of combining them? What is freer? How do I program with it? How does it (and should it) relate to try-catch games? And finally, what libraries implement and use Free? The point I am trying to make is that FP computations are a different breed. They actually have properties and the learner can build an understanding of these properties.
Effect systems (the main application of Free) are a very powerful programming tool, they can add a lot of structure and cognitive simplicity²⁷. I use 2 of them at work, one of them we maintain. Effect systems allow us to organize code into DSLs and interpreters. This approach creates a very high level of code reuse, testability, and defines very explicit, self-documenting types.
Now, is it realistic to learn these concepts in a day or a week when starting a new project? Imagine a programmer who uses Java at work exploring this knowledge.

There has been some discussion about making Haskell itself more accessible (e.g. Elementary Programming) and some library effort in this direction as well (e.g. IHP).
Some teams separate hard micro services with high levels of abstraction from the rest.
Some places separate a possibly very advanced implementation from a simple to use API (I believe Facebook’s Haxl does it). Creating a progression from easy to hard is less trivial.

FP is a niche, I think FP has a stable niche in programming. Correctness and understanding of computations are problems almost nobody in the industry cares about but they are sometimes needed. This reminds me of a Terry Pratchett Diskworld character: Esmerelda (Granny) Weatherwax

  “Esme Weatherwax hadn’t done nice. She’d done what was needed.”

Wanted means popular, needed means stable. However, basic principles of FP will probably find a wider use (as discussed in Extraneous loads that grow).

Post Summary

My readers may get the impression that this post is a criticism of imperative programming. Applying cognitive load theory to programming does not translate to “imperative is complex”, rather it translates to “too much of imperative in one place (logically coupled) is complex”. IMO, some amount of imperative is often helpful.
I plan to return to this topic in my next blog.

I am sure you have noticed that I think a lot about code complexity. And, yes, I do not feel comfortable working in messy code. Assessing and controlling the level of code complexity is crucial to me.

It has dawned on me that my dislike of code complexity may not be shared by others. False consensus effect is assuming that everyone else thinks like me.
I remain convinced that some programmers react negatively to code complexity, but now I think that many programmers feel at home in code with a high cognitive load. This motivated me to work on this and the next post. IMO it is important that we try to understand each other a little better.

Are we doing a good job in managing code complexity? I think this a fair question to ask even if you think that simplicity is not crucially important. This post has argued that we are mostly failing on that front. In this post, we looked at how project complexity grows unnoticed, how bugs are a missed opportunity to learn about how we fail, and how FP changes the cognitive process but can be hard to learn. As a whole this post has been a bit of repetitive negative thinking, but I hope you found some positives and useful ideas in it as well. The main point of this post was to advocate for including cognitive aspects of programming projects into consideration and to present some useful terminology for doing it.

  “It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity” Dijkstra (of course).

There is much more to it

This post took a very narrow path through the very broad subject of cognitive aspects of programming.

These were observations of a programmer, this post does not try to cover research on this topic or provide a good list of reading materials. I do not feel qualified to provide either. This topic is also related to code quality and this has been vastly discussed.

My focus was coding rather than process. I did not discuss things like cognitive loads in pool requests, cognitive considerations during sprint planning, git hygiene, etc.

Size of program files is an obvious, related topic I did not discuss.

Monorepo vs single projects has interesting relevance. Dependency graphs of or sorts (version, library deps) are a similar interesting topic.

Coding efficiency and the 10X programmer in the context of cognitive loads is an interesting (but contentious) topic.

Low Code: The idea of distributing cognitive load across different components is not new. The terms “decoupling” or “isolation of concerns” are in this space. Low code is an idea of a very lopsided distribution in which most of the complexity falls onto the infrastructure. I started writing about it but decided to remove my notes as this post feels already too long.

Some PLs (Haskell is a good example of this) suffer from what some people call the Lisp curse. Instead of using established libraries, proprietary or one-off tools are often created. It is interesting why this happens and what to do about it. Could love of abstractions be causing it (reuse abstractions, not a code)? Is writing it from scratch a lower cognitive effort than learning and applying an existing solution? The end result, obviously, increases the cognitive load.

Cognitive load should be viewed as a resource problem, one that does not scale very well, and one that is not well understood. Cognitive load is greatly impacted by turn over rates, switching of code ownership, and by installed processes. Context switching is very expensive, the programmer’s inability to find contiguous blocks of time to focus could be viewed as an indication of an under-resourced project. Needless to say, under-resourced yields quick and dirty code.

Linting, formatting, aesthetics are all very interesting cognitive load topics. Most programmers seem to be very sensitive to how the code is presented, (e.g. would you ever use a light background in your code editor?). Similarly, syntax vs semantics, it seems syntax has a huge cognitive role even if we think about it as bikeshed.

Habit formation and unlearning are a big and very interesting topic.

Cognitive biases in the context of coding seem like very interesting topics too. In particular bandwagon effect (TypeScript is popular and hence must be very good), framing effect (new cool technology), commitment bias (we done it like this before, it has been tried and tested), functional fixedness (we do not need another PL), omission neglect (things we do not know are not important), groupthink (we want to work with people who think like us), bikeshedding (possibly most of this post 🙂).

Point-free code, I stayed away from discussing it.

Cognitive aspects of troubleshooting are something I only touched on.

Imperative vs denotative is something I only touched on.

One topic I do plan to discuss (in the next post) is a distinction between empirical and formal processes in programming and how it impacts cognitive loads and acts as a divider.

Cognitive loads are related to stress, I intend to return to this topic in the future as well.

This post did not run out of topics, rather I have run out of steam. I hope I gave you things to think about. Thank you for reading!

Please Leave Feedback in: git discussions

Previous post: Part 5. Advanced Types.
Back to the beginning post: Part 1. Typing in Anger

Nutshell

This post wraps up my series about types in TS. In this series, we explored type-centric approaches to writing code and pushed TS to its limits, sometimes a little beyond its limits.

In any mainstream programming language there is a group of users interested in using types. Similarly to the rest of the industry, this group is a very small subset of the TS community. Developers interested in types tend to be unappreciated and underutilized.

Types in programming get very formal and are very interesting for mathematically inclined developers. Mathematical inclinations are probably a necessary condition for enjoying types. This partially explains why types are such a niche, but IMO there are other reasons I will try to discuss some of them in this post.

This post will discuss these aspects of types: types are

• About Clarity
• About Productivity
• About Simplicity
• About Safety
• About Correctness
• About Maintainability
• Universal
• Unpopular

I will finish my series with a short rant about each of the bullet points. This will allow me to revisit and summarize some of the things we have discussed in previous parts and mention a few things this series did not cover.
This post will be mostly a high level rant. I want to talk a bit about what is possible. Some of the discussion will not be very relevant to TS as the language lacks the capabilities. I think these topics are still relevant to TS developers as the ideas behind these concepts can still be useful.
Simply put, my goal is to discuss how types (including advanced stuff) can be used in TS.

Like all of my other posts in the series, this one is a big longish and tries to cover a lot of ground. I hope you will find it worth the effort.

Some readers may disagree when reading this post. You may have valid reasons for disagreeing with me. Please let me know what they are.

About Clarity

What are coding conventions and standards? When I hear these terms being used, I know I will soon hear about code formatting and linting, importance of code comments, even things like readme files and git hygiene. However, I am unlikely to hear about types. It is not that types are not important, they are. They are also harder to discuss. This phenomenon has a name, it is called Wadler’s Law or bikeshed.

I have discussed using types to achieve code clarity in referential transparency and types as documentation sections of Part 2. Let’s revisit the topic here.

It is much harder to comprehend the whole program than it is to comprehend its types. Types can provide a high level information about the program the way that theorems provide high level information about proofs in mathematics. Types can give a valid high level representation of the app. Programs often can’t, they often contain tedious details, performance optimizations, and lots of other persisted developer’s sweat.
When done right, one can use types as specs, at least on a unit level types could be viewed as specifications.

Types have a synergy with FP. This series was not about FP but it was hard for me to completely stay away, the synergy is so deep. Types help express functional concepts clearly.

Advanced types come with a learning curve. It is important to acknowledge that clarity is subjective and can easily be replaced with confusion unless developers are familiar with the concepts.

“WTFPM: WTF Per Minute is an actual measurement for code value.”

I imagine some topics covered in Part 4 or Part 5 could have a high WTFPM number. IMO, types used in production projects should be accessible to the project contributors. This means pushing the envelope just a bit but not too far.

The following subsections examine a few concepts related to clarity.

Declare function return types

TS does not require it but why would you not do that?

Variables named x

Are variable names essential to clarity? It is a bit of a telltale. One of the most common criticisms of languages that heavily use types (like Haskell) is:

“Whats up with the variable names, why everything is x,y,z, xs,ys,zs, f,g,h or a,b,c?”

Part of the reason is that the code can be very general. If a variable can be literally anything, why not call it x? The other reason is where the reader gets the information from: variable names or types? Personally, I believe in more explicit variable names when implementing a specific business logic and the implementation is long. However, even in such cases, the info should be in the types. If it is not, the code probably can benefit from refactoring.

Enums

This section is now somewhat misplaced. I am keeping this edited version for consistency with my original post. It reflects, IMO, valid points made by the reddit community. My original position was that enums are obsoleted by union types.

Are enums clearer than literal types? This code has 3 types (“foo”, “bar” and the union):

type FooBar = "foo" | "bar"

This code defines one type (FooBar) and is more verbose:

enum FooBar {
  Foo = "foo",
  Bar = "bar",
}

What is the advantage of using this enum?

EDIT: I got a pushback to my criticism of enum (reddit). Here is a list of valid reasons to use enum in TS:

• Intellisense is likely to work better (go to definition, search for usage - these niceties may not even make sense with union types)
• Type inference is likely to be much better as the name of the enum is specified at the usage point
• Enums can be more descriptive than some literals (e.g. OK is nicer than 200)
• Enums are nominally typed which can be often useful. Two enums have different types even if they have the same content. (note that in TS even classes are structurally typed, class content matters, class name not so much)

However, most type safety appears to be identical. For example, switch statement exhaustive checks are equivalent. IMO literal string types are often more readable. Readers familiar with languages that have a nice implementation of Algebraic Data Types are likely to gravitate towards unions.

Clarity vs encapsulation

Encapsulation does not help clarity. I consider encapsulation to be very useful when designing micro-services, not so much when designing programs. Encapsulation often means not expressive types. Encapsulating is hiding things from the types. It often makes types simpler than they should be. To get the benefits of types, we need to give them a chance. Type checker will not type check what is invisible to it.

On the other hand, explicit types (types that contain a lot of information) have a higher maintenance cost.
I like to compare this to documentation. If app functionality changes you should change the documentation. You are likely to do that only in the most obvious places. Explicit types are different, they create domino effects forcing you to propagate the changes to all the relevant places.
This overhead is not always desirable and there are patterns and tools to minimize such cost (e.g. TS subtyping, existential types, FP concept called smart constructors). Changing functionality should create compilation errors but ideally these errors should not be hard to fix.

Encapsulated code does not test well and often requires mocking frameworks. You will know you are doing something right when you stop using mocks for unit tests.

Referential transparency, purity, and explicit types

I have discussed these concepts already in Referential Transparency (Part 2). I want to return to this topic for another rant.
Referential transparency does not have an agreed upon formal definition. It typically means:

A computation is referentially transparent if it can be safely replaced with its return value¹.

This is clearly related to clarity and simplicity. It is hard to reason about code that does different things every time it is called.

Function is pure if it does not perform any side-effects (e.g. does not mutate things)

These concepts are related but not equivalent, e.g. a function that finds a shortest path in a graph is likely to be referentially transparent even if its implementation mutates its local variables (most standard graph algorithms are imperative and mutate stuff). Such functions “look” pure from the outside and maybe in some cases that is good enough. I may want to care about referential transparency more than about strict purity.

I like to treat referential transparency loosely. In my loose approach, referential transparency simply means that what the function does is exposed in its type (so referentially transparent and has an explicit type become the same thing). Thinking in these lines makes referential transparency less of a checkbox and more a progress bar.

Consider the following versions of code that are supposed to establish a WebSocket using some imaginary API (we are implementing a PetStore):

//(1) gets config from a global place and globally stores WS connection
const initWs: (): void = ...

//(2) gets config from a global place
const connectWs: () => WsConnection = ...

//(3) gets config from passed parameter, incomplete return type
const connectWs: (conf: PetStoreConfig) => WsConnection = ...

//(4) gets config from passed parameter, incomplete return type
const connectWs: (conf: {loggerConf: LoggerConfig; wsUrl: Url}) => WsConnection = ...

//(5) null, option, optional, maybe ... types do not contain much error context
const connectWs: (conf: {loggerConf: LoggerConfig; wsUrl: Url}) => WsConnection | null = ...

//(6) gets config from passed parameter, complete return type 
//(most likely will involve subtyping at usage point) 
const connectWs: (conf: {loggerConf: LoggerConfig; wsUrl: Url}) => WsConnection | WsError = ...

Note that there is no much benefit between using PetStoreConfig and {loggerConf: LoggerConfig; wsUrl: Url} or between WsConnection | null vs WsConnection | WsError from the point of view of strict referential transparency. There is, however, a big difference if you think about the information contained in the types.
(1) and (2) are very opaque, (3) and (4) are similar but not all PetStoreConfig is relevant, thus (4) type is more transparent and precise.
In my experience, even programmers who know a lot about types end up not thinking about exceptions and will code some equivalent of (5). The goal is to get to (6). (6) is very explicit, IMO it is the best. Subtyping is likely to be used at some point as PetStoreConfig probably will be passed to it. However, the first (the least explicit) approach is probably more commonly used at large.

Readers working with React can alternatively think about a component that uses an internal state hook (encapsulates state) vs a component that accepts a setter callback and a getter property as input arguments. You can also think about React Context API or a similar approach and compare it with explicit setters/getters.

Each computation has an input and an output even if TypeScript / JavaScript code does its darndest to hide it. TS code can pull the inputs out of thin air (configuration, stuff stored on the window object, etc) and sink the output by saving it somewhere. The above initWs is guilty of both of these felonies. Still, there is a referentially transparent computation hiding somewhere. In the above example the last connectWs type describes the inputs and output within the heavily encapsulated initWs.

Inputs and outputs are essential to clarity. The developer should try to understand what these inputs and outputs are at the very least. Ideally, the explicitly typed computation within can be factored out. This is not just for clarity, you are likely to find future uses for it (e.g. the last example above could be factored out of the PetStore and used in other apps or used to open 2 connections). And, it will be easier to test.

About Productivity

We are not leaving clarity behind. Clarity and productivity are obviously related. I am only changing the angle.

When I write code in an untyped language, I still think about types, the only difference is that I do not have any verification from the type checker. Not having a type checker or working with poorly designed types slows me down. Moving to a strongly typed functional language has made me much more effective, possibly 4x-5x times more effective (I am sure such stats are very personal and depend on many factors).

IMO all developers, whether they admit it or not, use types in their heads. The question is: how effectively?

The following subsections examine some concepts related to productivity.

A walk in the park

Types can guide the process of writing code. I can write code by ‘following the types’ if the API gives me well designed types to follow. The analogy is following a path in the park.

We have seen examples of this in Part 2 where I twisted office.js arm to get the types right and was able to type predicate myself to a much faster to write and safer code.

We have also seen this in Part 4 (preventing information escape, phantom types) where types formed jigsaw puzzles allowing the computations to fit together in only certain ways.

Inference reversed and T(ype)DD

This section is not very relevant to TS, but I think it is interesting to note.

Type inference allows a programming language to compute the types without needing the developer to specify them.
Ideally, the future will bring tooling where the developer defines the types and the compiler computes the program.

TS will not automatically implement code for us, however, starting with types and following with (a manually written for now) programs is often quite productive. This is the TDD approach to programming, only T means Type. The simplest way to go about it is to start on a small unit level (define types for small building blocks first). It helps to know some solid building blocks (e.g. FP types) and to use a lot of type variables. About Maintainability section will say a bit more about TDD and Types.

About Simplicity

I consider the terms simple and easy to have different meaning. Easy: A low barrier to entry (e.g. language). Simple: Low effort to reason about (e.g. code written in that language). There is no free lunch, to get simplicity you need to accept that things will not be easy.
Simplicity is about ability to reason about things and as such is closely related to all other bullet points in this post.
IMO, the popularity of easy and the unpopularity of simple are a systemic problem in today’s programming and elsewhere.

I consider TS to be complex (the opposite of simple). I devoted Part 3 to explaining why.

On some basic level, simplicity is associated with strictness. Flexibility seems to cause complexity. Flow’s exact objects are strict and simple, existential types are more flexible and more complex, subtyping makes types very flexible and very complex. However, there are many modern concepts that programming languages are still trying to figure out, e.g. dependent types (Idris, Agda, Coq), linear types (Rust, Haskell v.9, Idris 2). These concepts should be filed under strict and complex today. I have a feeling that in 5 years I will consider them less complex than subtyping (see the current subtyping doc from Rust).

One aspect critical to simplicity that is easy to explain and one that we have not discussed yet is totality.

Total vs Partial

Another related term is non-termination. Does the function return a value as expected? Bunch of things can go wrong: function can throw exception, loop forever, have unbounded recursion, return unexpected null or undefined³. Functions that return a result for all inputs are called total otherwise are called partial.

Total is simple. Reasoning about partial functions is much harder. Any non-termination bypasses the type checker. Using partial functions means that the types are misleading.

Exceptions seem to be the most frequent reason for the non-termination. Developers who like types avoid throwing exceptions, they will favor TS union types instead.

Let’s think again about computations from the input-output perspective and consider conditional control flow of the program.
TS’s ternary can be viewed as a function. if-else not so much. if_else does not have a type. It was designed for mutating things and in today’s more immutable approach to programming it should feel antiquated. However, it is idiomatic to both JS and TS and is impossible to avoid.
I use if_else blocks only with return statements (with exception of void functions). I do not use if without the matching else even if the code looks repetitive (again, with exception of void functions).
If you think about the “referentially transparent computation within”, you will notice that if without else is partial. Several programming languages offer if_else syntax without the if only option.

One could go (IMO too) far with this approach and use if-else as a lambda:

// if-else as a lambda, this seems overkill
const x = (() => {
    if (condition) {
        ...
        return "yes"
    } 
    else {
        ...
        return "no"
    }
    }) ()

I fully expect some pushback on this. My view is opposite to what you can frequently find on the internet (else is sometimes considered evil). Developers consider a sequence of if-s better than if_else chains.
For simple control flows it should not matter. For more complex code using partial if-s is concerning.

About Safety

Here are some interesting examples of safety that could be provided by types: safe routing in a single page app (no broken routes), safe use of environment configuration (e.g. types prevent accessing arbitrary environment variables), safe backend communication (imagine the same types in frontend and backend with safety preventing broken urls and payload parsing errors).
Safety can be very interesting, we have seen some examples specific to TS e.g. no information escape, no unknown, no subtyping.

It should be noted that subtyping reduces safety. We have discussed it extensively in Part 3

We are seeing a slow industry shift towards a more sophisticated use of types, IMO, TS could play a role in that shift.

Monads

I have stayed away from the topic thinking that there are enough monad tutorials already, but it is hard to not mention this concept. Monad types provide interesting safety: monads can control the ability to leave monadic computation. A value can easily enter a monad but once there it is hard to leave. This is clearly interesting from the safety standpoint and can be used to achieve all kinds of interesting guarantees. Things get really very interesting in the jigsaw puzzle building department with the addition of dependent types⁵.

Monads allow for a very imperative code. However, this requires some syntax sugar that the programming language needs to offer. This is called do notation in several languages or for comprehension in Scala. TS does not offer it. That makes monadic computing far less accessible.

fp-ts library provides support for monads and other functional types in TS. Thumbs up to all developers who use it or work on fp-ts.
I am not using fp-ts in my TS project (even though Haskell development is my main job function).
Each project needs to decide on the level of abstraction it allows to make developers working in it productive.

About Correctness

This series is not about formal verification, types and correctness could be its own blog (or book) series, one I am not qualified to write. I will only point out that, gradual typing or not, in TS correctness and soundness are a baby thrown with the bath water.
Making things conceptually easy at the cost of correctness (e.g. incorrect variance, incorrect or at least very unclear narrowing semantics) should not be on the table.

Subtle falsehoods can sometimes be more concerning than the obvious once.
Here is a coding challenge: There is a common belief that compilation flags like strictNullChecks prevent escaped null and undefined. Exploit the incorrectness of variance in TS to create a partial function that has number return type but returns undefined for some of its input parameter values.

This series discouraged the use of TS’s any type. Undeniably, combining any with stricter types can lead to some very interesting and useful code if one is careful. In a way, I view any to be more straightforward and less damaging than other violations of logical soundness in the TypeScript language.

About Maintainability

Considering who is still reading this, I am now only preaching to the quire so I will keep this short. Clearly all the points I have made so far are very related to maintainability.

My favorite definition of high code quality is, simply, a code with a low maintenance cost. IMO, everything else is subjective. Types have a big beneficial impact on that cost.

Developers often go to great lengths to avoid compilation errors. Sure, committing code that does not compile is not very professional but this attitude sometimes goes beyond that. IMO, designing types to be resilient to changes in functionality is not what you want to do. Compilation errors are why we use types, compilation errors are a good thing. What you want are errors that are easy to fix.

It is well known that types can prevent trivial errors (like using a string instead of object). It is hard to catch all such cases in tests and they do show up in production. This is the reason, I believe, TS is used in most of its projects.
Let me point out a less trivial high level bit. Types can simplify adding new functionality! If you think about the app as a big union type of various requirements (this is an oversimplification but let me keep going), then adding a new piece of functionality to that union could give you compilation errors unless you fix all the relevant places. Think about TS-s switch or ts-pattern library exhaustive checks⁶.

Universal

Types are more fundamental than a programming language. For example, most FP languages are effectively a syntax sugar over some version of lambda calculus. Lambda calculi come with very well understood formal type semantics.
I am reminded about the Propositions as Types presentation by Phil Wadler himself. It makes a compelling and funny argument that the movie Independence Day got it all wrong. Aliens would not have used C. C is being created by an engineering effort, types and LC are being discovered⁷. Aliens would have discovered typed lambda calculi or have engineered something much different than C or Java.
This is very philosophical, but it also has a pragmatic implication. Discovered programs are, by definition, timeless. If Wadler is right (and if we will keep programming in the future) that would be kinda amazing. In Part 4, I have referenced the TAPL book, IMO, the best textbook to learn types. This book is 20 years old. Recursion schemes (Part 5) are 20+ years old. Rank-2 types discussed in Part 4 were studied in 1980-ties and 90-ties. Many language features we consider new and modern are really old ideas, some date back to 1970ties.

Robert Harper has coined a term The Holly Trinity of CS and types are one of the three.

Types are playing an increasing role in foundations of mathematics, the new and “hot” topic is HoTT.

This series was written by a TypeScript newb. I am using TS since November 2021 and only on one project.
We have covered a lot of ground that probably is not well known to many seasoned TypeScripters. I think the existence of this series provides a good verification for my claim: it is more about knowing the types than it is about knowing the programming language.

Types could unify how we think and talk about programs. Effective development teams are small, the threshold seems to be somewhere around 4-5. Why is that? I had worked once inside a team of 8 (two teams with different core competencies were merged to work on a new project). Design meetings, OMG, we had a hard time agreeing on anything.
Nobody disputes that natural numbers satisfy 2 + 2 = 4, and that has to do with types. One of my goals in this series was to sell the idea that types are fundamental to programming and are mostly not something open to endless debates. Types could help facilitate an agreement.

Advanced Types as Patterns

Advanced types are worth learning even if TypeScript is not able to support them. Advanced uses of types often come with very well behaving principled computations. TypeScript may not be able to express such types in full generality, but it is often possible to use the principled approach as a pattern.
An example is the Recursion Scheme code I wrote for Part 5. I see map being added to all kinds of types as a pattern. Monads are used as a pattern too. The concept of async, await uses monads as a pattern. fast_check library uses monadic computing as a pattern to accomplish randomized property testing.

The burden to understand the principles lies on the authors of libraries and APIs. For example, developers using async / await do not need to understand the concept of a monad. You need to understand it to create the async / await concept.
It is also much easier to learn the underlying concept after experiencing examples of its use.

Unpopular

There are two directions to writing high quality low defect rate software. Both approaches complement each other.

1. Increase project effort and cost (e.g. testing)
2. Increase effort / cost outside of project scope (e.g. learning types)

In the industry focused on short term goals 2 will be unpopular even if benefits of 2 are significant.
The ramp up time for the projects needs to be short. This explains why all mainstream languages look and feel alike. As far as programming languages go, the software industry is not innovation friendly. Any progress needs to be very gradual. Developers need to be able to “hit the ground running” when using a new language.

I have already mentioned Wadler’s Law and bikeshed. Types are about semantics. That puts them at the far end of the popularity ranking scale. I have mentioned the easy vs simple dilemma. Simple is less popular. Types are theoretical, that makes them less popular as well.

Let’s look at the job market. The job market for typed functional programming jobs is, frankly, dismal. At the same time, languages like Haskell and Rust top the weekend use stats based on stackoverflow surveys⁸.

How can we explain both of these phenomena? One issue is that only a small minority of programmers are interested in the more principled methods of writing code. Weekend learners playing with Rust appear to outnumber devs doing weekend project work in, say, PHP. That is good, but the numbers are still not there. There needs to be a critical mass of enthusiasts and there isn’t one. At the very minimum, managers expect to have a solid supply of headcounts. Managers will consider use of an FP language risky. You do not get fired or criticized for selecting Java.
The other issue is the correlation between interest in types with interest in mathematics. Current rush towards machine learning sways the precious few mathematically inclined CS students towards well paying data science careers.
Yet another issue is education and how mathematics and CS are being taught.

Let’s take a bit more controversial take on this. A stronger version of “Someone is wrong on the internet” is this statement:

Lack of popularity is a necessary (not sufficient) condition of doing something right.

“Popular => wrong” is a law (or hypothesis) of life that dates back to at least Socrates.
If you assume this to be true, you can view the progress as a process of being less and less wrong.
People look at the history of the software industry and see a never ending aggressive progress. A more insightful hindsight exposes a history of embracing bad ideas (e.g. null) and resisting good ideas (e.g. type parameters⁹).
You probably think of all of this as too hyperbolic. The benefit of taking this stance is a chance of noticing things that others don’t.

Lack of popularity can translate to some frustration for the type enthusiasts. The frustration comes in the form of rejected designs, rejected pool requests, failed job interviews. I heard stories and experienced some of it first hand. That is just part of life, the criticism can have validity as more advanced programming techniques could make the project confusing and not accessible to its contributors.
It also should be expected. One comment I received about Part 1 of this series said “this code is quite different from what we do”. “Different” could imply worth pursuing but unavoidably will at some point lead to a confrontation.

Types lack the critical mass of acceptance to become disruptive, they work well when the team is ready and/or when applied in a gradual way. Thumbs up to projects and developer teams who learn types and select the unpopular!

“Only in our dreams are we free. The rest of the time we need wages.” Terry Pratchett and Hwel. A good metaphor to describe the life of a programmer.

Gradual Progress

There is a steady and slow progress. Mainstream languages are introducing a little bit of types and FP.
async-await is now supported by many languages. Sum/variant types are supported by many languages (TS’s union types stand out for their readability). Record types are being introduced as well (e.g. Java 14 records, C# 10 record struts, …). C# has type safe equals. Advanced types in TS we have discussed in Part 4 and Part 5. The list slowly grows.

Final words

TS does a poor job implementing types. However, it has types and it even allows to do some advanced things with them. The last two installments (Part 4 and Part 5) allowed me to go places I would not be able to reach in most mainstream languages.

If developers start using types, the languages will expand support for them.
This will feed some gradual change. The hope for existence of such a feedback loop is what prompted me to write this series.

Should more advanced types be used in a project? Ideally (and IMO) that decision should be made by the developers. I have presented plenty of pros. The biggest obstacle is the learning curve. I am afraid this learning needs to happen outside of the project work. In reality, this means that the decision has to be made based on what the team knows already. So the answer for some teams could yes today, for some could be no today but a yes in the future.
My personal approach is to make sure that TS code is approachable and my goal is to make it principled within this constraint.
This is not very easy to do, it is much easier to use principled types than principled patterns. It is also easier to write principled code in an environment where principled is not considered odd.
It is also good to be able to scratch the itch and keep practicing the real thing, I have my backend work to do that, lots of people do not have that luxury.

This series was a long journey, I am happy I took it, but I am also happy the effort is now mostly behind me. Big thanks to all of you who stayed with me all the way to this end.
Thank you to everyone who messaged me corrections and comments. Please let me know your thoughts on this installment.
Good luck with your projects, I hope you will use types!

Please Leave Feedback in: git discussions

Previous post: Part 4. Programming with Type Variables.

Disclaimers: (imagine this is a very small font, read it very fast in a half whisper)
I assume strict compiler flags are on, something you get by default with scaffolding, e.g. using create-react-app my-project --template typescript is close enough.
The code examples have been tested with TypeScript v4.5.2.
This post is a pandoc output of a markdown document and code examples are not interactive.
Most of the code examples are published in ts-notes folder in this github repo: ts-experiments.

Motivating Quote for the series:

“TypeScript began its life as an attempt to bring traditional object-oriented types to JavaScript so that the programmers at Microsoft could bring traditional object-oriented programs to the web. As it has developed, TypeScript’s type system has evolved to model code written by native JavaScripters. The resulting system is powerful, interesting and messy.”

From typescriptlang TypeScript for Functional Programmers

Nutshell

This is the fifth post in the series devoted to types in TypeScript. In this series, I explore type-centric approaches to writing code and push TS to its limits in doing so. I am writing these posts for like minded developers who are interested in types and either use or consider using TypeScript.

In the last post I referenced the Types and Programming Languages book. Similarly to the previous post, this installment will be a little more advanced and a little TAPL-ish. I will also introduce a tiny bit of Category Theory. A great blog series (really a book) about Categories is Category Theory for Programmers, here it is on goodreads.

Recursive types

type JsonVal from Part 1 surprised me. It is recursive, the name JsonVal appears on both the LHS and the RHS of the definition. Here is this definition repeated:

type JsonVal = 
| {type: "object", val: Map<string, JsonVal>}
| {type: "array", val: JsonVal[]}
| {type: "string", val: string}
| {type: "number", val: number}
| {type: "bool", val: boolean}
| {type: "null"}

and there are TAPLish reasons why this is interesting:

JsonVal-like types appear to be hard on the TS type checker. I have played with some advanced recursive types and have experienced it first hand. I got quite a few

‘Type instantiation is excessively deep and possibly infinite’

compiler errors (e.g. code in https://github.com/rpeszek/ts-typecheck-peano). However, I did not succeed in creating a simple example to demonstrate this.

Here is another example of a recursive type:

type List<T> = 
| {type: "nil"} 
| {type: "cons", head: T, tail: List<T>}

That is a recursive definition of a functional cons list².

IMO, it is impressive that TS is able to pull these off. I consider this feature very useful and underutilized by the ecosystem. Here is some more advanced use of recursive types:

Type level programming

TS literal types are singletons (i.e. type "boo" has exactly one value "boo":"boo"). This allows singletons to magically connect types with values and values with types. That provides a lot of power to create very precise types.
Literal types should not be that hard to implement in a programming language and it is interesting why they are so uncommon. Kudos to TS for introducing these! They are, clearly, a great fit for JS.
However, TS literal types are very limited in scope (I remember reading somewhere that it was a design decision). For example, you can do some very basic type level string manipulation but you cannot concatenate strings or do any arithmetic on type level numbers and you have no way of defining any additional features on your own (e.g. DIY number addition).

TypeScript allows for type-level ternary (Conditional Types) as well as various type-level built-in functions (e.g. keyof).
Apparently, the type level programming in TypeScript is Turing Complete (see https://github.com/microsoft/TypeScript/issues/14833).
However, type level programming in TS is focused on creating type safety for various JS code idioms rather than creating a foundation for DIY type level programming. IMO this makes it harder to learn. The Turing completeness appears to be a completely accidental language feature.

Type level programming can be very useful, we have seen some of it in action in the previous post where we were able to prevent subtyping and prevent compiler from using the unknown type.

IMO the best language design direction is for the type level and the value level code to look the same (e.g. dependently typed language like Idris). The second best approach is for type level and value level to be very similar (e.g. Haskell).
TS cannot and should not do either. We do not want JavaScript (or very similar) on the type level!

At the same time, the lack of synergy between type level and value level programs makes things very complicated. E.g.:

//example type from https://www.typescriptlang.org/docs/handbook/2/conditional-types.html
type Flatten<Type> = Type extends Array<infer Item> ? Item : Type; 

const head = <T> (t: T[]) : Flatten<T[]> => {
    return t[0] //compiles (as expected)
}

const generalizedHead = <T> (t: T) : Flatten<T> => {
    if(Array.isArray(t)) 
        return t[0]  //still compiles (as expected)
    else 
        return t //compiler error: Type 'T' is not assignable to type 'Flatten<T>' (not as I would expect)
}

here is another one:

type HasContent<C> = {content: C}

type GetContent<T> = T extends HasContent <infer C> ? C : T

const getContent = <C, T extends HasContent<C>> (t: T): GetContent<T> => {
   //return t.content //compiler error:  Type 'C' is not assignable to type 'GetContent<T>' (not as expected)
}

It feels clunky. It feels like type level and value level have a broken marriage. This lack of synergy also feels very confusing.

I think TS type level programming will keep improving and we may see some very interesting use cases in the future.

Subtyping

This series has discussed subtyping already. I will keep this section comparatively short.

Personally, I try to avoid using subtyping features. Subtyping is related to Object Orientation. OO programming has an appeal of simplicity and I was seduced by it for many years. It took me a long time to realize that OO is not that simple. Today, I think about OO as very complex. Even language designers often get it wrong (this series has provided a lot of evidence for this statement in the context of TypeScript). The first 3 parts of this series could have been alternatively titles “Dangers of OO with examples in TS”⁴. This negative view of OO should be filed under IMO as many developers disagree.

Before continuing reading pass this code, please try to implement (at least in your head) the amIFooOrBar function:

function verifyExtends<T2 extends T1, T1>() {}

//more specific, fewer variants
type FooOrBar =  
| {foo: string} 
| {bar: string}

//A challenge: implement this function:
declare function amIFooOrBar(o: FooOrBar): "foo" | "bar"

declare function genFooOrBar(): FooOrBar

//more general, more variants
type FooOrBarOrBuz =
| {foo: string} 
| {bar: string}
| {baz: string}

declare function genFooOrBarOrBuz(): FooOrBarOrBuz

const fooOrBarOrBuz: FooOrBarOrBuz = genFooOrBar() //compiles assigns specific to more general 
//const fooOrBar: FooOrBar = genFooOrBarOrBuz() //will not compile tries to assign general to more specific

verifyExtends<FooOrBar, FooOrBarOrBuz>() //compiles, FooOrBar extends FooOrBarOrBuz
//verifyExtends<FooOrBarOrBuz, FooOrBar>() //does not compile, FooOrBarOrBuz does not extend FooOrBar

The thing to remember is that {foo: string} | {bar: string} extends {foo: string} | {bar: string}| {baz: string} not the other way around.

Did you implement amIFooOrBar, great, let’s move on.

Subtyping in object types will feel familiar to OO developers. Roughly speaking, you can assign object with more properties to object with fewer properties:

type FooAndBar = {foo: string, bar: string} //more general
declare function genFooAndBar(): FooAndBar

type FooAndBarAndBaz = {foo: string, bar: string, baz: string} //more specific
declare function genFooAndBarAndBaz(): FooAndBarAndBaz

const fooAndBar: FooAndBar = genFooAndBarAndBaz()  //specific assigned to general is valid assignment
//const fooAndBarAndBuz: FooAndBarAndBaz = genFooAndBar() // will not compile, tries to assign general to specific

verifyExtends<FooAndBarAndBaz, FooAndBar>() //compiles, FooAndBarAndBaz extends FooAndBar
//verifyExtends<FooAndBar, FooAndBarAndBaz>() //does not compile, FooAndBar does not extend FooAndBarAndBaz

Subtyping gets more involved if you combine adding properties to objects and variants to union types. In TS subtyping extends to functions which makes things even more complex (leading to what I called compilation bloopers in Part 1). But I think the above examples cover the basic idea.

Now let’s revisit the above challenge. What will your function return in this call:

// challenge check:
// what does your function return when used on this value?
// NOTE this does compile, you can assign FooAndBar to FooOrBar, since 'and' implies 'or'
const whatIsThat = amIFooOrBar({foo: "foo", bar: "bar"}) 

This is just one of the many gotchas associated with subtyping.

Thunks and callbacks, never and unknown.

To finish this post I want to pick 4 concepts fundamental to TypeScript: variables, callbacks, never and unknown types and discuss how they relate in a somewhat more theoretical setting. I believe the relationship between these concepts is not commonly understood.

We have seen <T> () => T before, we called it a type hole _: <T>() => T. Now I am changing my mind and want to call it a generic thunk.
We can think about it as a ‘lazy’ value.
Instead of defining const t: T (which, incidentally, TS does not allow on the top level⁵) I can define a function that, when called, will return me that t. Basically thunks are variables you put the () after. A referentially transparent (no side-effects) <T> () => T thunk is morally equivalent to a variable of type T.

A thunk produces a value of type T. A generic callback <T> (_: T) => void consumes a value of type T. There is, clearly, some type of duality between thunks and callbacks.
Incidentally, many programming languages define a unit type often denoted as () instead of the C-style void. If this was the case for TS, we would have written: <T> T => () for the callback and <T> () => T for the thunk. You can get from one type to the other by reversing the arrow =>. These concepts become dual in the categorical sense. This post is not about Category Theory but this section has just a tiny bit of it.

In TS, the generic thunk <T> () => T type is equivalent to never.
You may remember that the type hole _(), was implemented by throwing an error (that is never in TS).
never assigns to everything but you cannot assign anything else to it. Well, except for the generic thunk:

//thunk assigned to never
const nevr : never = _()

export const __neverFn: () => never =  _

In other words <T> () => T and () => never can be assigned to each other, thus, I consider them equivalent.

If you replay the same argument with arrows reversed, you will establish equivalence between the generic callback <T> (_: T) => () and the unknown callback types:

declare function someUnknownCallback(t: unknown): void 
const overbar: <T>(_:T) => void =  someUnknownCallback

declare function someOverbar<T>(t:T): void
const unknownCallback: (_: unknown) => void = someOverbar

The never type is the TS’s bottom type (can be assigned to anything), while the unknown type is the TS’s top type (anything can be assigned to it). These concepts are also dual in the sense of reversing the direction of assignment.

Let’s think about referential transparency again. There are no interesting referentially transparent functions that return void. To do something meaningful, such a function would need to mutate some shared state or do some other effectful things. E.g. when coding in React, a callback could compute a new state (let me call it r: R) and invoke a state hook to make the change. I like to think about such a callback as having an imaginary type <T,R> (t: T) => R.

The duality between variables/thunks and callback is quite fascinating and has some depth.
Let’s fix the type variable T to, say, Person. Any type would do, I just want to remove the quantification (remove the genericity) to simplify my explanation.
JS / TS programs often use higher order functions that accept callbacks as parameters. Consider a callback that accepts a callback (f: (_: Person) => void) => void and computes the same value. The imaginary referentially transparent type for it could be

//TS-like pseudocode
<R> (f: (_: Person) => R) => R

As it turns out, this type is equivalent (isomorphic) to the thunk () => Person (or, ignoring side-effects, to just Person)!
They are not equivalent based on assignments, they are equivalent because one can be easily converted to the other.

It kinda makes sense for a dual of a dual to end up back where we started.
However, this equivalence is a bit stronger, in a sense that it holds for every fixed type T. It is also weaker, since what we get is only isomorphism⁶.

This equivalence is a special case of Yoneda Lemma in Category Theory⁷.

I can express this succinctly in TS (note a higher rank type is used) as:

//all of these compile
type Yoneda<T> = () => <R>(f: (_: T) => R) => R
type Thunk<T> = () => T

//Yoneda<T> is isomorphic to Thunk<T>
//here are functions defining the isomorphism:
const toYoneda = <T> (th: Thunk<T>): Yoneda<T> => {
   const res = () => <R> (f: (_: T) => R): R => f(th())
   return res
}

const fromYoneda = <T> (y: Yoneda<T>): Thunk<T> => {
    const res = (): T => y()(x => x)
    return res
 }

Programmers are divided into 2 camps when exploring this type of information: some consider it fascinating and important, some consider it a lot of useless nonsense. If you are still reading this series, chances are you are in the first camp.

Category Theory is very related to types and to programming in general, I found it only fitting to finish this installment with a note that discussed a little bit of it.
Bartosz Milewski’s CTFP book linked above starts with code examples in C++ and in Haskell. Bartosz gives up on C++ very fast. I think it would be possible to stay on a little longer by selecting TS instead of C++. Kudos to TS!

Next and the final Chapter

I will finish the series with some final thoughts and rants.
The last 2 installments got a little on an advanced side of things. One question I have been asking myself is: When should more advanced types be used in a TS project?

Here is the link: Part 6

Please Leave Feedback in: git discussions

Previous post: Part 3. TS Complexity.

Disclaimers: (imagine this is a very small font, read it very fast in a half whisper)
I assume strict compiler flags are on, something you get by default with scaffolding, e.g. using create-react-app my-project --template typescript is close enough.
The code examples have been tested with TypeScript v4.5.2.
This post is a pandoc output of a markdown document and code examples are not interactive.
Most of the code examples are published in ts-notes folder in this github repo: ts-experiments.

Motivating Quote for the series:

“TypeScript began its life as an attempt to bring traditional object-oriented types to JavaScript so that the programmers at Microsoft could bring traditional object-oriented programs to the web. As it has developed, TypeScript’s type system has evolved to model code written by native JavaScripters. The resulting system is powerful, interesting and messy.”

From typescriptlang TypeScript for Functional Programmers

Nutshell

This is the fourth post in the series devoted to types in TypeScript. In this series, I explore type-centric approaches to writing code and push TS to its limits in doing so. I am writing these posts for like minded developers who are interested in types and either use or consider using TypeScript.

This post will be a little more advanced and will focus on programming with type variables.

Types and Programming Languages is the book about types I recommend to everyone (… even if not very successfully). Reading TAPL will be a big eye opener for many developers. The good news is that types dramatically increase programming efficiency so learning them is a good investment.
This section of the post will be a little more TAPL-ish with some more advanced CS. The topics I am about to present are IMO very useful and I will try my best to present them in a digestible way.

I will discuss type variable scoping, rank-2 types, and existential types. Some examples show a level of safety that I did not expect to be able to pull off! As it turns out, we can even prevent subtyping in TS.

Before we start I need to build up some tooling. I will start with a tiny bit of type level programming.

Safety preventing unknown

In previous posts, we have seen examples where TS decided to widen types to unknown rather than report a compilation error.
Interestingly, TS allows enough type level programming so we can try to fix such issues ourselves.

type IsUnknown<T> = unknown extends T? true: false

function verifyUnknown<T>(p: IsUnknown<T>, t: T): T {
    return t
}

verifyUnknown(false, "test")
const unk: unknown = {}
verifyUnknown(true, unk)

//Compilation Error
//Argument of type 'false' is not assignable to parameter of type 'true'.ts(2345)
verifyUnknown(false, unk)

In my first post, I had an example of incorrect code body4 inferred as unknown instead of a string. Wrapping such code in verifyUnknown(false, body4) would have alerted me with a compilation error.
You may point out that a much simpler solution is to just type annotate: const body4: string.
I agree. However, having a more generic solution at our disposal is also useful. We will see shortly why.

Here is a short TAPL-ish explanation of what just happened. TS allows me to use type level ternaries. IsUnknown<T> is a type level function (TAPL’sh term for this is Type Family) that maps types T to literal boolean types true or false. These types have only a single (a singleton) value: true: true and false: false. If I write verifyUnknown(false, someExpression), TS will figure out that it has to use false as the type. false matches the second part of the type level ternary and, thus, implies that the ternary predicate unknown extends T is not true. Hence T is not unknown.

I will use verifyUnknown to do some type level trickery. You may wonder if we can extend this approach to other types, not just to unknown. I will get there in this post as well.

Type variable scoping

Type variable scoping has two aspects. Let’s start with the most obvious one. The type variable being visible inside of the implementation body. This is simple stuff, I just want to share an obvious gotcha that got me at some point:

export const bodyScopeExample1 = <T>(value: T | undefined | null): void => {
    if(value) {
        const t: T = value //you can access type variables in function body
    }
}

This approach to defining type signatures (actually my preferred way to write function type signatures) puts T out of scope:

export const bodyScopeExample2: <T>(_: T | undefined | null) => void = value => {
    if(value) {
        const t: T = value  //Cannot find name 'T'.ts(2304)
    }
}

For the type variables to be visible in the implementation they need to be on the RHS of =.

The other aspect of type variable scoping is much more interesting:

Higher Rank types

Consider these 2 function declarations:

declare function fn1<T> (f:(t:T)=> void): void 
declare function fn2(f: <T>(t:T)=> void): void 

In fn2 the scope of T is much narrower. In TAPL-ish this would be called a rank-2 type.
So what is the difference? Let’s try to use both:

const useStr = (s:string): void => {}
fn1(useStr)

//Compilation Error:
//const useStr: (s: string) => void
//Argument of type '(s: string) => void' is not assignable to parameter of type '<T>(t: T) => void'.
//  Types of parameters 's' and 't' are incompatible.
//    Type 'T' is not assignable to type 'string'.ts(2345)
fn2(useStr)

Basically fn2 requires the argument to be fully generic and useStr is not.

I can play the same games with generic arguments that return T

declare function fn4(f: <T>() => T): void

but will not do that here as these tend to be less practically useful.

Here is how I think about it:

Higher rank means generics are first class

Existential types

In TAPL-ish this is called existential quantification and it has to do with the ownership of definitions. In OO you would say “code to interfaces, not implementation”, it is also related to the OO concepts of inversion of control and dependency injection. Here is how the story goes:

Replacing factory pattern

interface Foo {
    foo: string
}

class MyFoo implements Foo{
    foo: string
    constructor() {
        this.foo = "bar"
    }
}

We want to be able to hide which implementation of Foo we are passing to a callback.
Our first approach tries to use a vanilla TS generic function with a callback argument.
The input function parameter uses some <T extends Foo> of an unknown exact implementation type:

function factoryWithCallback<T extends Foo> (f:(_:T) => void): void {
    //Argument of type 'MyFoo' is not assignable to parameter of type 'T'.
    // 'MyFoo' is assignable to the constraint of type 'T', but 'T' could be instantiated with a different subtype of constraint 'Foo'.ts(2345)
    f(new MyFoo()) //Compilation Error
}

It does not work and it should not work! We need to use a rank-2 definition:

//Compiles!
function existencialFactory(f: <T extends Foo>(_:T) => void): void {
    f(new MyFoo())
}

This simulates what is called an existential type. The function that accepts a callback owns the definition of the exact type that is passed to the callback. The callback itself needs to be generic and accept any possible implementation.
Note the scoping of T inside the type defining the function parameter.

This inverts the control from the implementation of the callback to the caller.

declare function fn1<T> (f:(t:T)=> void): void 

declare function fn2(f: <T>(t:T)=> void): void 

Preventing information escape

I am drawing a blank trying to think about an OO analogy for this. It is somewhat related to friend classes in C++, package-private scope in Java … only not exactly.
This example will accomplish more than the above ‘factory’ pattern and will not use any interfaces or classes:

// Using higher rank to protect data
// Imaginary world without debuggers, JSON.stringify, etc
type Api = {getGoodies: string[]}

//provides access to API, password needs to be protected
declare function login<Password>(p: Password): Api 

//provide password to a computation, that computation should be able to use the password but shouldn't return it
const secretive = <R> (fn: <Password> (p: Password) => R): R  => {
   const s : any = "topsecret"
   return fn (s)
}

The example is somewhat contrived with the main goal of illustrating the point.
This code exposes building blocks that work together. To get the access to the Api type, you have to use login and you have to use it inside the provided secretive function. Working with an API like this is like assembling a jigsaw puzzle. Types prevent from jamming a square peg into a round hole.
Note, Password is a type variable and we have used the existential type trick.

This code uses the building blocks:

const goodProgram = <Password>(p: Password): string[] => {
    const api = login(p)
    return api.getGoodies
}

const stealPassword = <Password>(p: Password): Password => p

secretive(goodProgram)
secretive(stealPassword)

Unfortunately, secretive(stealPassword) compiles. Somewhat typical of TS, instead of providing robust type safety, the compiler infers unknown and accepts my questionable code. Hovering over secretive shows me this:

//const secretive: <string[]>(fn: <Password>(p: Password) => string[]) => string[]
secretive(goodProgram)

//const secretive: <unknown>(fn: <Password>(p: Password) => unknown) => unknown
secretive(stealPassword)

That is why I have created the verifyUnknown safety in the previous section:

const valid = verifyUnknown(false, secretive(goodProgram)) //valid: string[]

//Argument of type 'false' is not assignable to parameter of type 'true'.ts(2345)
const invalid = verifyUnknown(false, secretive(stealPassword)) //does not compile!

To make it a bit nicer we can package verifyUnknown and secretive into one function:

const verySecretive = <R> (_: IsUnknown<R>, fn: <Password> (p: Password) => R): R  => {
    const s : any = "topsecret"
    return fn (s)
 }

const valid = verySecretive(false, goodProgram) //valid: string[]

const invalid = verySecretive(false, stealPassword) 

This creates some interesting safety. Obviously you could still do a lot of mischief if you wanted to. There is a need for some ‘gentlemen’s agreements’ to not use casting, JSON.stringify, to not use true in verySecretive etc. However, if you think about creating clear contract APIs, this approach could be very powerful.

Existentials are not exactly equivalent to OO. However, using existential types can often accomplish a lot of the same things and often in a cleaner way. Using existentials and disabling OO features like unknown feels a bit contrived, but IMO is still useful. It would be nice if TS provided a cleaner way to disable the use of unknown.
I do not know how robust this type of coding is. I have not played enough with this approach in TS to give you a list of gotchas. In my very limited experience, this seems similar to the rest of TS, TS stops working if I start pushing harder.

Safety preventing subtyping

Many TS users have observed the need for this. The term exact type is floating around, I believe flow introduced this name. I have seen solutions like this one being proposed:

function exact<T>(item:T): T {
    return item
}

type Hello = {hello: string}

//Argument of type '{ hello: string; since: number; }' is not assignable to parameter of type 'Hello'.
//  Object literal may only specify known properties, and 'since' does not exist in type 'Hello'.ts(2345)
exact<Hello>({hello: "world", since:2002})

This safety is fragile (and a TS design inconsistency IMO) as the following example shows:

const helloSince = {hello: "world", since:2002}

exact<Hello>(helloSince) //complies

To create something more robust, here is a code that combines the above unknown verification idea with existentials:

type Same<P,T> = P extends T? (T extends P? true: false): false

const verifySame = <P> () => <T> (_: Same<P,T>, t:T): T => t

verifySame<Hello>()(true, {hello: "world"}) //'true' indicates that type matches
verifySame<Hello>()(false, {hello: "world", since : 2020}) //'false' is needed to acknowledge types are different 

//Argument of type 'true' is not assignable to parameter of type 'false'.ts(2345)
verifySame<Hello>()(true, {hello: "world", since : 2020})
verifySame<Hello>()(true, helloSince)

You may have noticed a case of typing euphoria here. I used rank-2 construction because it allows me to type annotate with only one type variable. This is nice but often not essential.

Here is an implementation of safePush that acts invariant, it does not use any existential tricks:

//Note to get 'safePush' I ended up with casting, this is a quick and dirty example and can be done slightly better
// However, this cast could be an indication that we are changing how TS compiler works
// Kinda makes sense, to overrule the compiler I may need to cast
const safePush = <P, T> (_: Same<P,T>, ps: P[], t: T): number => ps.push(t as any)

const intlist: number[] = [1,2,3]
const unklist: unknown[] = intlist  //exploits array covariance
unklist.push("not a number") //unsafe 'push' adds a 'string' to 'intlist'

safePush(true, intlist, 1) //this is safe

safePush(true, unklist, 1)    //this is risky and will not compile 
safePush(true, unklist, "not a number") //this is risky (here wrong) and will not compile 

Note, to be even safer I would need to prevent unknown as well:

const unkstr: unknown = "not a number"
safePush(true, unklist, unkstr)  //unfortunately compiles

//An even safer version of 'Same'
type SameAndKnown<P,T> = P extends T? (T extends P? (unknown extends T? false: true): false): false

const verySafePush = <P, T> (_: SameAndKnown<P,T>, ps: P[], t: T): number => ps.push(t as any)

verySafePush(true, intlist, 1)  //this is safe

verySafePush(true, unklist, unkstr) //this is risky and will not compile!

We have discussed problems with the TS approach to variance in the previous installment. We have a DIY approach to fight back!

Side Note: The linked github repo has an existentially typed version of safePush (safePush2) that has just one top level type variable. That version is more cumbersome to use. TS ends up not working well with it.

Another fun exercise:

const safeEq = <P, T> (_: Same<P,T>, a: P, b: T): boolean => a === (b as unknown)

safeEq(true, {hello: "word"}, {hello:"dolly"})

safeEq(true, {hello: "word"}, {hello:"word", since:2022}))
safeEq(true, 1, "str")

We have discussed problems with TS approach to === narrowing in the previous installment. Again, we have a DIY approach to fight back.

This section is related to a number of feature requests: TypeScript issue 12936 and TypeScript issue 7481. Hopefully a future version of TS will provide a simpler way to achieve invariance and disable subtyping.

Phantom types

TypeScript is somewhat unique in supporting Structural Types. Types like type Person = {firstNm: string, lastNm: string} are structural. That means the name Person is only an alias, what defines the type is the RHS of the definition, not the LHS. Contrast this with an OO class definition in a language like Java. Two structurally identical classes are still considered different types (this is called nominal typing).

It is sometimes convenient to be able to define different types that share the same structure. Phantom types are a way to do that. We say phantom because these types have no impact on runtime values.

Somewhere around 2006, haskell wiki published a write-up about phantom types. The write-up was expanded in 2010 to include a form validation example. Since then all blogs (in any programming language) about phantoms show a validation example. I decided to be as unoriginal as everyone else. This will allow me to better focus on how it is done in TS.

My first attempt at phantom types in TS will fail. But this code should make the idea behind phantoms clear:

//Marker type
type Validated = {type: "validated"}

//For simplicity this is just a string
type ValidationError = string

//Extra phantom type variable 'T' 
type Person<T> = {firstNm: string, lastNm: string}


//Validate person in some way returning 'Validated' phantom marker
declare function validate<T>(p: Person<T>):  ValidationError | Person<Validated> 

//Function to be used only if phantom 'T' is the 'Validated' type 
declare function doSomethingValidated(p: Person<Validated>): void

Again, these types are trying to create a jigsaw puzzle. One I can assemble in a specific way only.
If the puzzle machinery works, I will have to call validate first to be able to use doSomethingValidated.

Only, this machinery does not work. The following code compiles:

function validatedOrNot<T>(p: Person<T>): void {
    doSomethingValidated(p)
}

type ClearlyNotValidated = {type: "notvalidated"}

function notValidated (p: Person<ClearlyNotValidated>): void {
    doSomethingValidated(p)
}

The fix is to provide a value level information about T in an optional property.
This type definition replaces the one above:

//Modified definition adds value level representation `phantom?: T` 
type Person<T> = {firstNm: string, lastNm: string, phantom?: T}  

//provide a way to create person that ignores the additional 'phantom' property:
const createPerson : <T>(fst: string, lst: string) => Person<T> = (fst, lst) => {
    return {firstNm: fst, lastNm: lst}
} 

Now this compiles:

function validated(p: Person<Validated>): void {
    doSomethingValidated(p)
}

But these no longer do:

// Compilation Error
// Argument of type 'Person<T>' is not assignable to parameter of type 'Person<Validated>'.
//   Type 'T' is not assignable to type 'Validated'.ts(2345)
function validatedOrNot<T>(p: Person<T>): void{
    doSomethingValidated(p)
}

// Compilation Error
// Argument of type 'Person<ClearlyNotValidated>' is not assignable to parameter of type 'Person<Validated>'.
//   Type 'ClearlyNotValidated' is not assignable to type 'Validated'.
//     Types of property 'type' are incompatible.
//       Type '"notvalidated"' is not assignable to type '"validated"'.ts(2345)
function notValidated (p: Person<ClearlyNotValidated>): void {
    doSomethingValidated(p)
}

I believe phantom types are used by some FP libraries in TS, e.g. fp-ts, these libraries use somewhat different techniques to get phantoms. There may be advantages to doing phantom types differently than what I have presented. The above approach is the simplest I can think of.

//Sort status as a phantom type,  'List' has type level information about its sort status.
type List<T, SortStatus> = ...

interface Comparator<T> {
    compare (o1: T, o2: T): number
}

declare function sortAscending <T extends Comparator<T>, AnyStatus> (list: List<T, AnyStatus>):  List<T, "ascending">

declare function doSomethingWithSortedList <T extends Comparator<T>> (list: List<T, "ascending">): void

Phantom types could be a very powerful API building tool.
I am sure you can think about many other interesting use cases, … like state machines¹.

Next Chapter

I want to talk about recursive types and type level programming. It will be more of a review of TS capabilities in these areas.

I need to take a break from writing posts. The next installment will take me longer, maybe a month or a little more, to finish.
Thank you for reading. Happy New Year!

Here is the link: Part 5.

git discussions

This blog has used a vanilla Hakyll styling with slightly modified standard pandoc CSS for code blocks.
Some of the readers have experienced very weird font size irregularities, making it hard to read my blog.

I did a full CSS reset using meyerweb reset.css and have restyled all posts. Sans-Serif fonts are now a hard-coded default (before the font-family was not specified leaving it to the browser defaults to do whatever mischief they fancied).

Please let me know in git discussions if you still experience styling issues.

Thank you to everyone who alerted me about the styling problems.

Please Leave Feedback in: git discussions

Previous post: Part 2. Typing Honestly.

Disclaimers: (imagine this is a very small font, read it very fast in a half whisper)
I assume strict compiler flags are on, something you get by default with scaffolding, e.g. using create-react-app my-project --template typescript is close enough.
The code examples have been tested with TypeScript v4.5.2.
This post is a pandoc output of a markdown document and code examples are not interactive.
Most of the code examples are published in ts-notes folder in this github repo: ts-experiments.

Motivating Quote for the series:

“TypeScript began its life as an attempt to bring traditional object-oriented types to JavaScript so that the programmers at Microsoft could bring traditional object-oriented programs to the web. As it has developed, TypeScript’s type system has evolved to model code written by native JavaScripters. The resulting system is powerful, interesting and messy.”

From typescriptlang TypeScript for Functional Programmers

Nutshell

Happy New Year! Let’s hope 2022 it will be way better than 2021. It has to be.

This is the third post in the series devoted to types in TypeScript. In this series, I explore type-centric approaches to writing code and push TS to its limits in doing so. I am writing these posts for like minded developers who are interested in types and either use or consider using TypeScript.

In this post we will see TS struggle. We will see compilation inconsistencies and surprising type checker behavior.
My main goal is to point out the complexity of what TS is trying to accomplish and share my understanding of it.
On a positive note, I will introduce additional tools for asking TS type questions.
Also, I promise, the next installment will be about good things in TS. It will be about programming with type variables.

Before we discuss the messy bits, let’s briefly talk about some cool type safety features.

Interesting safety

TypeScript implements special narrowing semantics when processing parts of JS code. These semantic rules provide very surprising and useful type safety features. TS can effectively narrow types used in a number of JS operators such as typeof, ===, == and apply this information to if-else, switch statements. This post has already shown a few examples where this, almost magically, prevents placing code in a wrong branch of conditional if-else blocks.

Here are some of my favorites with IMO on their use.

apple !== orange type safety

This JavaScript code (I keep reusing type Person = {firstNm: string, lastNm: string} from the first post):

//Bad code
function blah(lhs: string, rhs: Person) {
  if (lhs === rhs) {
    //Do something
  } else {
    //Do something else
  }
}

is a programming bug and will not type-check in TypeScript. You can just replace it with:

//Actual equivalent
function blah(lhs: string, rhs: Person) {
  //Do something else
}

TypeScript prevents from using === if it can guess¹, by looking at the types, that === will always be false. This is true in general, not just inside if-else, but the if-else use is the killer app IMO.
One cool example of === type safety combines type narrowing with literal types: 1 === 2 will not compile!

This is a big deal. === is often used to compare things like string or number id-s or hashes and it is not that uncommon to accidentally try to compare something like an id with something completely different.
I have seen analogous issues in many programming languages including even Scala.

switch exhaustive check

if-else does not provide any mechanism for the type checker to verify that the program checked all possible conditions.
Interestingly, we can use the switch statement in TS to solve this problem:

//This compiles!
const contrived_better = (n: 1 | 2): number => {
    switch(n) {
       case 1:
        return 1
       case 2:
        return 2 
    } 
}

//Compilation error
//Function lacks ending return statement and return type does not include 'undefined'.ts(
export const contrived_better_ = (n: 1 | 2 | 3): number => {
    switch(n) {
       case 1:
        return n
       case 2:
        return n 
    } 
}

That is another nice example of TS enhancing JS with a nice type safety feature.

IMO an even better solution is provided by the ts-pattern library. See this blog post: Introducing ts-pattern v3.0

null / undefined safety

We have seen null safety already. There is a semantic difference between null and undefined but most code does not care. My personal preference is to unify these two.

In my very first example in the series, getName(p: NullablePerson), was not undefined safe, only null safe. Using it with undefined (e.g. on expressions typed as any) will cause an error.

My coding preference would be to rewrite my first example like this:

//Reusable utility type
export type Undefined = null | undefined

export const isUndefined = (d: unknown): d is Undefined =>
   (d === null) || (d === undefined) //I prefer not to use '=='

const getName2 = (p:Person | Undefined): string => {
    //const tst1 = p.firstNm //will not compile
    if(isUndefined(p)){
        //const tst2 = p.firstNm //will not compile
        return "John Smith"
    } else {
        return p.firstNm + " " + p.lastNm //compiles
    }
}

This is just my personal preference, I also use this approach when typing optional ? object properties. E.g.

type Person2 = {firstNm: string; middleNm?: string | Undefined; lastNm: string}

The extra safety features are what surprised and excited me about TS. They reminded me of a functional programming language.

Complexity of TS types

Throughout the series, we encountered a few examples where the TS type checker did not work as expected, we will encounter more of TS quirkiness in this section. This note suggests a reason for this: type complexity.

My original plan was to write about TS needing to implement a separate ad-hoc semantics for various JS operators. I was not able to present anything very insightful and I have abandoned that idea, e.g. these type hole expressions do not even compile:

//Compiliation errors: Object is of type 'unknown'
_() + _()
_() * _()
_() / _()

Taking the quote from the top of this post to heart, I concluded that TS is about providing support for OO and other idiomatic uses of JS. I decided to narrow the focus of this note to subtyping and the === operator semantics.

=== semantics, rejected overlap

I have picked === because we discussed it already in my previous note about the unknown type. Selecting == would produce a very similar presentation.

Here is an example of safety around the === operator:

//This condition will always return 'false' since the types '"world!"' and '"Dolly!"' have no overlap.ts(2367)
"world!" === "Dolly!" //does not compile

Let’s try to figure out the semantic rules around ===. What does “not having an overlap” mean?
I have not seen a formal (or even a somewhat precise) definition of the semantic rules for the ===.
(Please comment in git discussions if you know about any place that defines these.)
The informal definition (from typescriptlang documentation) points to a “common type that both x and y could take on” but this statement clearly has some loose ends.

The first part of the error message “This condition will always return ‘false’” suggests a way to start:

(EQ-SAFETY attempt 1): TypeScript prevents using === if it can prove, by looking at the types, that the result of === would always be false.

This is a very high level and does not explain how TS does it. But is this even true?

function testEqSemantics(a: {bye: string}, b: {hello: string): boolean {
   //This condition will always return 'false' since the types '{ bye: string; }' and '{ hello: string; }' have no overlap.
   return a === b //does not compile
}

Let me temporarily comment the not compiling code:

function testEqSemantics(a: {bye: string}, b: {hello: string}): boolean {
   //This condition will always return 'false' since the types '{ bye: string; }' and '{ hello: string; }' have no overlap.
   //return a === b
   return true
}

const helloBye = {bye:"world!", hello:"world!"}
testEqSemantics(helloBye, helloBye)  //compiles, here is the overlap!

TS has effectively prevented me from using === even though there are legitimate cases where the === would have returned true! This seems like a major blooper.

We have falsified the error message from TS.

OO is complex and type design issues are not uncommon among OO languages, this could be one of them.
On the other hand, preventing {bye: "world!"} === {hello: "world!"} from compiling seems useful from a pragmatic point of view. It is possible that this behavior is intentional.

I see 2 possible conclusions

1. This is a bug caused by a complexity of TS’s semantic rules
2. This is a feature indicating that the rules are indeed complex

This appears to be one of the “Working as Intended” or at least known issues (see footnote 4).

=== semantics, what’s an overlap?

Let’s focus on this part of the error message: “types … and … have no overlap”.

(EQ-SAFETY attempt 2): x === y compiles if x: X and y: Y and the compiler successfully computes some special non-never Overlap type that widens to both X and Y

X is the computed type for x, Y is the computed type for y, how do we compute Overlap type for both? I think we can assume that widens simply means extends.
The 64K dollar question is how is the Overlap computed? It is clearly not the same as intersection (the type operator &), we have falsified that hypothesis in the previous section. Let’s try to look at some patterns:

const helloDolly: {hello: string} = {hello: "Dolly!"}
const datedHello: {hello: string, since: number} = {hello: "world!", since:2022}
const one = 1 //const one: 1
const two = 2 //const two: 2
const onenum: number  = 1
const twonum: number  = 2
const world: string = "world"

//fails, different literal types do not overlap
"Dolly!" ===  "world!"
//fails, different literal types do not overlap
one === two
//fails, string and number do not overlap
one === world

//compilies, note both have the same type
onenum === twonum
//compiles, note 'typeof datedHello' extends 'typeof helloDolly' 
helloDolly === datedHello

//compiles, the overlap seems to be the 'Person' type
function tst (x: number | Person, y: string | Person) {
    return x === y
}

//compiles, the overlap seems to be `{hello: string, since: number}` 
function testEqSemantics2(a: {hello: string} | 1, b: "boo" | {hello: string, since: number}): boolean {
    return a === b
}

A possible rule for calculating Overlap could be (this is just a rough, high level heuristics, please comment if you know a better definition):

• for intersection types X and Y, if X extends Y take X else if Y extends X take Y otherwise reject
• for union types X = X1 | X2 | ... and Y = Y1 | Y2 | ... recursively check if any Xi and Yj overlaps (this heuristics ignores performance cost)
• for complex combinations of union and intersection types? I DUNNO, I have not tested it enough.

I have not played with this assumption for a very long time, but so far these rules seem to hold with these exceptions:

//All compile
1 === null

1 === undefined

function tst2 (x: 1, y: null) {
    return x === y
}

Does 1 have an overlap with null and undefined? What does that even mean? With the strictNullChecks compiler flag, null should be well separated from other types.
This particular quirkiness is actually useful, it allows for a program to do conservative null checks even if the type indicates that it is not needed.

I hope you agree. This is complicated.
I will hopefully bring this point even closer to home by the end of this post.

Hidden blooper (side note)

If you remove type annotations from the above definitions, the helloDolly === datedHello still compiles:

const helloDolly = {hello: "Dolly!"}
const datedHello = {hello: "world!", since:2022}

helloDolly === datedHello //still compiles

From a pragmatic standpoint this is very strange. "Dolly!" === "world!" is statically rejected, but {hello: "Dolly!"} === {hello: "world!", since:"2022"} is not.

This surprising situation is caused by the type inference widening the types. The types inferred in the expression "world!" === "Dolly!" are the literal types "world!": "world!" and "Dolly!": "Dolly!", while the helloDolly and datedHello infer a string and number for their properties:

//IntelliSense view of helloDolly
const helloDolly: {
    hello: string;
}
//IntelliSense view of datedHello
const datedHello: {
    hello: string;
    since: number;
}

TS allows to define the above object types using as const, e.g. const helloDolly = {hello: "Dolly!"} as const and const datedHello = {hello: "world!", since:2022} as const. It this is done helloDolly === datedHello will no longer compile but IMO, widening object property types is an arbitrary complexity.

DIY equality

The question is how far can I get by trying to reproduce safety around the === on my own.

declare function eq<T>(t1: T, t2: T): boolean

This generic function (it could be implemented by simply using ===) forces both arguments to have the same type. That should give me at least some level of extra safety and prevent from comparing apples and oranges.
Let’s see, starting with these type holes:

//type holes shows a string, not bad, I would prefer the literal "foo".
eq("foo", _())
//type holes shows unknown, Another unexpected 'uknown' widening issue? 
eq(_(), "foo")

Let’s ignore the second type hole disappointing quirkiness and move on.

//These all compile
eq(1 as 1, null)
eq(1, 2)              //NOTE we lost the type safety of ===
eq(1 as 1, 2 as 2)    //NOTE we lost the type safety of ===
eq({bye: "world"}, {hello: "world"})  //NOTE we lost the (possibly erroneous) type safety preventing {bye: "world"} === {hello: "world"}

How come these compile? These are all different types but TS can unify them into a supertype (next section will discuss it). These are all legitimate statements. Unfortunately, the type safety has been lost. This explains why the semantic narrowing around the === operator is needed. It is needed because structural subtyping can unify types even if types are very different.

However, quirkiness alert, these do not compile:

//Argument of type '"boo"' is not assignable to parameter of type '1'.ts(2345)
eq(1, "boo")
//Argument of type '1' is not assignable to parameter of type '"boo"'.ts(2345)
eq("boo", 1)
//Argument of type '{ hello: string; }' is not assignable to parameter of type '1'.ts(2345)
eq(1, {hello: "world"})
//Argument of type '{ hello: string; }' is not assignable to parameter of type '"boo"'.ts(2345)
eq("boo", {hello: "world"})

This is very unfortunate, you want generic functions to work consistently across types. IMO this is a bug or an arbitrary complexity.
The quirkiness seems to be related to the type inference working inconsistently and failing to widen the types if a string literal type is involved (next section will discussed it).

The narrative has run away from me, but the point should be somewhat clear: Generics provide only limited type safety in TS.
E.g. enhanced safety semantics around === does not transfer to a DIY safety that a library solution could expose.

Subtyping

How come this compiles?

eq(1 as 1, 2 as 2) 

The type checker widens the types of both arguments to 1 | 2. This is because of a subtyping rule that says that 1 extends (1 | 2) and 2 extends (1 | 2).
Here is a somewhat clever trick to see that:

export declare function unify<T>(t1: T, t2: T) : T

//hovering over unify shows me:
//(alias) unify<1 | 2>(t1: 1 | 2, t2: 1 | 2): 1 | 2
unify(1 as 1, 2 as 2)

if you do not believe me that 1 extends (1 | 2) you can check it for yourself with another trick:

export function verifyExtends<T2 extends T1, T1>() {}

verifyExtends<1, 1 | 2>()

However, TS appears to be not consistently good about inferring these subtyping rules. TS apparently did not notice that 1 extends (1 | "boo") and "boo" extends (1 | "boo"). Hence the blooper

verifyExtends<1, 1 | "boo">()
verifyExtends<"boo", 1 | "boo">()

//Argument of type '"boo"' is not assignable to parameter of type '1'.ts(2345)
eq(1, "boo")

Let’s try to force TS into compliance by type annotating everything:

const booone : 1 | "boo" = "boo"
const oneboo : 1 | "boo" = 1

//Argument of type '1' is not assignable to parameter of type '"boo"'.ts(2345)
eq(booone, oneboo) 

//finally compiles with type application on 'eq'
eq<(1 | "boo")>(booone, oneboo)

We have seen that === narrowing is partially consistent with the intersection (& operator).
Let’s look at & semantics a little closer.

We can try to double check how the & intersection works by doing this:

//both compile suggesting that Person is equivalent to the intersection  (number | Person) & (string | Person)
verifyExtends<Person, (number | Person) & (string | Person)>()
verifyExtends<(number | Person) & (string | Person), Person>()

However this does not compile, and it does look like a bug (see second line of the error message):

//Type '(1 | "boo") & ("boo" | Person)' does not satisfy the constraint '"boo"'.
//  Type '1 & Person' is not assignable to type '"boo"'.ts(2344)
verifyExtends<(1 | "boo") & ("boo" | Person), "boo">()

Complexity is a super food for bugs.

Here is my quick summary: subtyping is complex and it weakens type safety. TS tries to recover the safety by building complex narrowing semantics around a selected set of JS operators. There are many inconsistencies in both the implementation of subtyping and the implementation of narrowing semantics.

verifyExtends<() => number, (_:string) => number>()
verifyExtends<(_:string) => number, (_1:string,_2:boolean) => number>()>

Comparative complexity rant

A “type enthusiast” will associate types with correctness, even formal verification. To me, the words “messy” and “type” are self contradictory. TS “types” support some interesting features but are a mess.

I want to contrast the above === and eq examples against a programming language that has been designed around types from the beginning. An example could be an FP language like Elm, PureScript, or Haskell (I am not that familiar with ReasonML or OCaml)².
These languages have much simpler types. The safety around equality does not require any special narrowing semantics. You get it for free in any DIY function that has 2 arguments sharing the same generic type (only they call it polymorphic not generic).

One underlying reason for this is the lack of complex subtyping and OO features. eq(x,y) will not compile if x and y have different types. There is no way to unify x and y to some supertype because there are no subtypes or supertypes.
But, you may say, JS object polymorphism is very useful. All the 3 languages listed above provide support for polymorphic record types³, only they use much simpler techniques than subtyping to achieve it.
These languages also come with well thought out semantic rules that are often formalized and come with soundness proofs.
The types in these languages are much simpler (not necessarily easier but simpler).

Type complexity translates to a confused type checker and to a confused developer.
Programming in a language in which I do not fully understand the types equates to me writing programs I do not fully understand.

I expect that to become a seasoned TS developer, one needs to remember a big dictionary of idiosyncratic compiler behaviors. Common Bugs that aren’t bugs is, I think, just a warm up reading to achieve such mastery.
Were you surprised about the gotchas we have uncovered in Part 1? Is the above overlap issue a well known problem⁴? Call me weird, but I would rather be learning PLT or Type Theory than these gotchas.

It is worth noting that TypeScript has over a million users. FP languages have tens of thousands of users (if combined). TypeScript has more resources to improve. What makes for fewer bugs, lots of dollars or clean types?
I do not think there is a clear answer to this question. However, resources can’t solve all the problems. Programming languages are almost paranoid about backward compatibility and backward compatibility does not like changing things, even if the change is fixing bugs.
So I am afraid, a simple language like Elm will always be cleaner and more robust.

Forgetting about the popularity context, I view it as a trade-off: suffer because of the type complexity and reduced type safety but see a readable JavaScript and trivially integrate with the rest of JS ecosystem vs introduce a language that has nicer types, greater type safety, predictable compiler, but lose generated JS code clarity and suffer when integrating JS libraries.
This trade-off is IMO not trivial and very project dependent. Clean types vs clean JS, I typically select the clean types. The ecosystem compatibility issue is a little harder to ignore and the main reason I am writing code in TS. Projects with a high correctness requirement, IMO, should select an FP language, the optimal choice for other projects is less clear.

Variance problems

I will finish with some examples that may feel even more surprising.

const bye = {bye: "world"}
const hello = {hello: "world"}

declare function eqArrays<T>(t1: T[], t2: T[]): boolean

eqArrays([{bye: "world"}], [{hello: "world"}]) //compiles

//Compilation error
//Property 'bye' is missing in type '{ hello: string; }' but required in type '{ bye: string; }'.ts(2741)
eqArrays([bye], [hello])

Here is another example:

interface Payload<T> {payload: T}

// ... we would see the same behavior for:
//type Payload1<T> = {payload: T} 

declare function eqPayloads<T>(t1: Payload<T>, t2: Payload<T>): boolean

eqPayloads({payload: {bye: "world"}}, {payload: {hello: "world"}})  //compilies

// Compilation error:
// Property 'bye' is missing in type '{ hello: string; }' but required in type '{ bye: string; }'.ts(2741)
eqPayloads({payload: bye}, {payload: hello})

My first instinct was to assume that this weird behavior is caused by TS treating T[] and Payload<T> conservatively as invariant. Unfortunately, this is not the case. The above quirkiness looks to be just another type inference issue and there is a deeper safety problem.

TS implements variance incorrectly and makes both T[] and Payload<T> covariant (e.g. TS assumes that P extends T implies Payload<P> extends Payload<T>). Here is a well known Java language bug reimplemented in TS:

//how to put a string into a list of numbers
const intlist: number[] = [1,2,3]
const list: unknown[] = intlist
list.push("not a number") //compiles

//array is incorrectly covariant
verifyExtends<typeof datedHello[], typeof helloDolly[]>() //datedHello extends helloDolly type

I see the same incorrect subtyping on the Payload interface:

//interface Payload is incorrectly covariant
verifyExtends<Payload<typeof datedHello>, Payload<typeof helloDolly>>()
verifyExtends<Payload<typeof datedHello>, Payload<object>>()

Implementations of interface Payload<T> do not need to behave in a covariant way.
An example in the linked github repo exploits interface Payload<T> covariance and ends up passing a number to a function that accepts string input.

Invariance would have been a better (a more conservative) choice for both interface Payload<T> and the array.
A careful reader may notice that the structurally typed type Payload1<T> = {payload: T} should also be invariant since the payload property is mutable (getters are covariant, setters are contravariant). TS incorrectly makes it covariant.

I will sound like a broken record now, subtyping is clearly very complex.

I did more digging into it after writing this note. It appears that the intention was to keep TS conceptually easy (issue #1394).
The result may be easy but is definitely not simple.

Incorrect is never simple.

Summary

This was a very hard note to write. I rewrote it several times. How do I write about complexity and make it simple to read?
Seems like a catch-22 problem.

“One does not simply explain TS types”

Boromir about TypeScript

Again, my main claims are:

• subtyping adds significant complexity and lowers type safety
• ad-hoc semantic narrowing around JS operators partially recovers safety, but is complex by itself and scope limited

Languages with simpler and more reliable type systems are not a superset of JS syntax and are idiomatically far from JS⁵.

We have observed some compilation issues and irregularities. To summarize these:

• issues inferring literal types widened to a union (eq(1, "boo"))
• issues preventing intersecting unions involving literal types ((1 | "boo") & ("boo" | Person))
• unexpected widening of literal object property types (hidden blooper)
• inconsistent widening of function arguments (top of variance problems)
• incorrect handling of variance (variance problems)
• === rejects the & overlap of intersection types, while claiming the opposite in the error message (rejected overlap)

I cannot identify TypeScript documentation or tickets relevant to these bullets. The subset I have checked against TS issue board is either in the known issues and / or “Working as Intended” category. My question about known issues is: known by whom?

Introduced tools

declare function unify<T>(t1: T, t2: T) : T
function verifyExtends<T2 extends T1, T1>() {}

can be used to ask TS subtyping questions.

Next Chapter

This post has been about the “messy” in TS. The next installment will focus on programming with type variables and will present TS in a better light. I decided to split advanced topics into 2 smaller posts. I plan to discuss phantom types, type variable scoping, a pattern emulating existential types, and rank 2 types. I consider these to be quite useful typing approaches. I will also show a trick that prevents unknown and supertype widening.

Here is the link: Part 4.

Happy New Year to all of my readers. Thank you for reading.

Summary of final edits

• Added information about as const in Hidden blooper note
• Added note about tickets relevant to the overlap issue (see footnote 4)
• Added side note about arity in Subtyping.

Please Leave Feedback in: git discussions

Previous post: Part 1. Typing in Anger.

Disclaimers: (imagine this is a very small font, read it very fast in a half whisper)
I assume strict compiler flags are on, something you get by default with scaffolding, e.g. using create-react-app my-project --template typescript is close enough.
The code examples have been tested with TypeScript v4.4.4 and v4.5.2.
office.js examples are based on https://appsforoffice.microsoft.com/lib/1.1/hosted/office.js and @types/office-js@1.0.221 (these match the current scaffold for office.js/React).
This post is a pandoc output of a markdown document and code examples are not interactive.
Most of the code examples are published in ts-notes folder in this github repo: ts-experiments.

Motivating Quote for the series:

“TypeScript began its life as an attempt to bring traditional object-oriented types to JavaScript so that the programmers at Microsoft could bring traditional object-oriented programs to the web. As it has developed, TypeScript’s type system has evolved to model code written by native JavaScripters. The resulting system is powerful, interesting and messy.”

From typescriptlang TypeScript for Functional Programmers

Nutshell

This is the second post in the series devoted to types in TypeScript. In this series, I explore type-centric approaches to writing code and often push TS to its limits in doing so. I am writing these posts for like minded developers who are interested in types and either use or consider using TypeScript.

This post will cover TS’s type predicates, the notorious any, and its safer cousin the unknown. These are well known and heavily blogged topics. My goal is provide a little different perspective with a more type-centric view point.
This series uses office.js as a source of code examples. This post examines the correctness of office.js types and fixes them using type predicates.
My main code example is something I am excited about. It demonstrates a case where TS made me completely rethink a previously written JS code.
I will discuss some safety concerns about unknown (no, this is not a typo, I mean the unknown type) and will set the stage for my future note about complexity of TS types.
I will finish in the realm of coding conventions discussing transparent, self documenting type definitions.

Can I trust the types?

I am going to discuss the obvious gotcha in a gradually typed language like TS: runtime values do not satisfy statically defined types.
Despite it being an obvious concern, the issue is something a developer who spends most time in a statically typed language (e.g. me) will not have on his / her mind when working in TS.
The following seem to be the prevalent reasons for why values do not match types: overconfident TS code (e.g. type casting, any type), issues with converted JavaScript (declaration files out of sync or containing otherwise incorrect definitions). I am going to show a real life (or close to real life) example of each.

The series started with an example defining the Person type, to avoid jumping back and forth I will repeat it here

type Person = {firstNm: string, lastNm: string} 

This will be a good conversation starter:

//Questionable JSON parsing example
const p: Person = JSON.parse('"John Smith"')

Your experience with consistency of JSON data may be different from mine. I rarely see JSON issues in a frontend - backend conversation. On the other hand, my experience with using 3rd party REST APIs is not exactly stellar. JSON data problems do happen.

The above code illustrates what I used to call ‘fail late’ and now I call ‘a type I cannot trust’ case. It is a nasty situation where runtime errors are nowhere near the actual problem. Looking at the example, JSON.parse function is declared to return the TS’s notorious any type. Using any bypasses type checking and the code assigns the result to Person. The actual run-time value of p will be a string, while the type checker is now convinced it is p:Person.

Now, look at the top rated answer in this stackoverflow: how-to-parse-json-string-in-typescript. It appears that the above code matches the top rated answer. Yes, safer approaches are available (look at less popular answers, we will discuss a much safer way as well).
I am not claiming this to be a prevalent problem in TS code, but it is an interesting issue caused by the coexistence of the typed and the untyped.

Now, since I already may have angered a large part of the TS community (did I? I hope not.), let’s beat a little on office.js.

office.js is a source of code examples for my series. Looking into office.js release history suggests that a bond between office.js and TypeScript. That bond developed very early. It looks like these projects grew up together. office.js might have even been one of these Microsoft projects that spearheaded the development of TS.

Short Recap We are using office.js to interact with Outlook emails. office.js provides us with item: Office.MessageRead allowing us to retrieve data from an email opened for viewing in Outlook. (Recap End)

I imagine it is not that uncommon for a TS library to have a non-nullable property that is undefined at runtime.
The IntelliSense tells me that item: Office.MessageRead contains an overloaded item.body.getTypeAsync method. I was hoping to use it to retrieve the type (plain text vs html) of the email body.

(method) Office.Body.getTypeAsync(options: Office.AsyncContextOptions, callback?: ((asyncResult: Office.AsyncResult<Office.CoercionType>) => void) | undefined): void (+1 overload)

getTypeAsync is undefined at runtime. It looks to me like the TS declaration files are not in sync with JavaScript. My hypothesis seems to be confirmed by the item.body.getTypeAsync documentation suggesting that this method is available when email is open in compose mode (not when using Office.MessageRead). (I am using office online and the latest office.js as of the time of this writing.)
Please message me in git discussions if you think I am misrepresenting it.

It seems like office.js types are a little off.

We should look at the type definition of the office.js Office.context.mailbox.item a little closer.
This property is overloaded to be one of the following types (let me call them facets):

Office.AppointmentCompose (composing calendar entry)
Office.AppointmentRead (reading calendar entry)
Office.MessageCompose (composing email)
Office.MessageRead (reading email)

These facet types are all different. For example, to get email subject you use item.subject:string if you are working with Office.MessageRead or item.subject:Office.Subject if you are working with Office.MessageCompose.
Office.Subject contains getAsync, setAsync methods and is absolutely not a string.

The type of item provided by office.js is not, as I would expect:

//Type I expected
AppointmentCompose | AppointmentRead | MessageCompose | MessageRead

Rather it is closer (I have not listed all the &-s) to:

//Actual Type with some & parts removed 
AppointmentCompose & AppointmentRead & MessageCompose & MessageRead

Basically, the type office.js chose for item mashes all the available properties, methods, overloads into one type. This is simply an incorrect type for the item property. Runtime values do not satisfy the intersection type, they satisfy the union type. Type checked programs will fail at runtime. office.js type declarations are incorrect.

office.js types are off for sure.

In a weird way, this explains why the undefined item.body.getTypeAsync has not been noticed. Without a corrective reassignment to, say, Office.MessageRead many other methods are undefined at runtime and it is harder to single this particular one out.

Gradual typing over the wild-west JS has to come with maintenance challenges.
Nonetheless this is surprising. What are the types good for if they’re not accurate?

“You take the blue pill — the story ends, you wake up in your bed and believe whatever you want to believe.
You take the red pill — you stay in Wonderland, and I show you how deep the rabbit hole goes”

Morpheus about not believing types in a gradually typed language
… nightmares of JavaScript running on my walls and ceilings make me wake up screaming

Note about the any type

My first example in this post used the infamous any type. Let’s have a closer look.

any type is crazy. It behaves like the top (you can assign any other type to it). It also behaves like the bottom (it can be assigned to any other type, maybe except of never). Ideally, the bottom type is empty, this one clearly is not.

As a result, any value can have any type.

We should have some fun with this.

//express yourself with _any_ (notice no casting, only assignments)
const sad: any = "emptiness and sadness"
const sadVoid: void = sad

const myCallback = (n: number): void => {
    return sadVoid;
}

You can have your own favorite null that is not null value, you can define your own undefined. Sky and your creativity are the limits. I will spoil this party and say that I do not recommend doing it. Oh, maybe just a little. Well OK, one more:

const sassy: any = {netWorth: "billion dollars", popularityLevel: "celebrity"}
const sassyNull: null = sassy

const p: Person | null = sassyNull

A bottom that is not empty will cause the language to be unsound. Allowing all values in a bottom type, I would call it insane.
However, using an any type similar to TS’s seems to be a common practice in gradually typed languages (e.g. Python does it too).
Using any is like saying “hey, TS, please suspend type checking, I know what I am doing”. This is the antithesis of type safety, but what else can TS do and maintain JS compatibility?

Actually, TS has a very clever solution for this, it is described in the following sections.
I view any as a form of type coercion or casting.

Casting casting in a bad light

I will use the term casting and type coercion interchangeably. TypeScript documentation also uses the term type assertion. I view the any type to be in the same boat as well (an implicit type coercion).
TS uses the t as T or <T> t syntax to cast expression t into type T, e.g. iAmSureIsString as string.
(IMO, the second notation, <T> t, is somewhat unfortunate as it is very similar to type application and generic function declaration e.g. const f = <T>():T declares, <T>f() casts, f<T>() applies. I recommend the v as T syntax to make casting more explicit and searchable in your code.)

Let’s beat on office.js some more. Here is a piece office.js documentation about (you guessed it, this post is so very predictable) the Office.context.mailbox.item:

If you want to see IntelliSense for only a specific type or mode, cast this item to one of the following:
AppointmentCompose
AppointmentRead …

TS offers a neat alternative to casting. I will explain it by not following the office.js documentation ;)

As I indicated already, I can interact with outlook email using Office.context.mailbox.item. However, item property is overloaded into several types discussed in the previous section (I called them facets):

The legacy code I am currently re-implementing at work is retrieving the email subject using item.subject and checking what kind of item.subject it is (a string, has asyc methods, etc) and using it accordingly. It does a similar “check before you use” game to retrieve to, from, cc and other email information.
Such an approach is typical, almost idiomatic to JS. It is also hard to maintain as making changes directed at one facet can easily break the other facets. And you can test your heart out on all emails you can think about and your app will still crash and burn if used with an office calendar appointment.

So what is the new TS-idiomatic way to do it?  TS has the is types.

Improving office.js with type predicates

export const isMessageRead = (item: any): item is Office.MessageRead => {
    return (item.itemType === Office.MailboxEnums.ItemType.Message) && item.getAttachmentsAsync === undefined
} 
  
export const isMessageCompose = (item: any): item is Office.MessageCompose => {
    return (item.itemType === Office.MailboxEnums.ItemType.Message) && item.getAttachmentsAsync !== undefined 
} 

declare function doSomethingWithViewedEmail(item: Office.MessageRead): void
declare function doSomethingWithComposedEmail(item: Office.MessageCompose): void
declare function onlyEmailEntriesAreSupported(): void

(OK, checking getAttachmentsAsync is ugly, office.js could provide some nicer and more stable way to identify the exact item type. This is still not bad. Let’s move on.)

doSomethingWithViewedEmail and doSomethingWithComposedEmail can now be coded with confidence (if I trust office.js types) following the corresponding MessageRead or MessageCompose types. IntelliSense makes writing these a breeze and the code is very clean. E.g., subject is just a string in MessageRead.

I can use these without any casting:

//'unknown' replaces incorrect office.js type (see previous section). 
const item: unknown = Office.context?.mailbox?.item

if(isMessageRead(item)) {
  //doSomethingWithComposedEmail(item) //this will not type check!
  doSomethingWithViewedEmail(item)    
} else if (isMessageCompose(item)) {
  //doSomethingWithViewedEmail(item) //this will not type check!
  doSomethingWithComposedEmail(item)  
} else {   
  calendarEntriesAreNotSupported()
}

This is a really nice, bravo TypeScript! Simple to use, yet very useful.

It is also IMO a very interesting case of TS making a bigger impact on how we actually code. “Check before you use” game becomes type assisted and happens on a coarser scale of item types instead of single (e.g. the email subject, from, cc, etc.) properties.
This adds a lot of clarity to the code. TS types not just check my code, types change how I code!

t is T type is one of the TypeScript narrowing tools. The documentation refers to it as a type predicate or a type guard (a more general term).
IMO, the idea of a middle ground between type checked safety and unsafe type coercion is brilliant.
It is something that sits a half way between a cast and a type equality proof.
This will probably influence other languages (e.g. here is enhancement proposal for Python).

The syntax t is T is interesting, it clearly borrows from dependently typed languages. The value t appears next to the type T and comes from the earlier part of the declaration. This also somewhat justifies the existence of otherwise cumbersome parameter names in type definitions (something I complained about in my previous post).

I hope the TS community develops a healthy aversion to casting. Why would you use a type checker if you keep subverting it? I also hope that exporting functions returning type predicates will become a standard practice for APIs.

Note about the unknown type

This post started with a use of the unsafe JSON.parse. I am quite sure that if TypeScript could travel back in time JSON.parse would return unknown instead of any.

export const safeParseJSON : (_: string) => unknown = JSON.parse

const isPerson = (p: any): p is Person => 
        typeof p.firstNm === 'string' && typeof p.lastNm === 'string'

const possiblyPerson = safeParseJSON('"John Smith"') 

if (isPerson(possiblyPerson)) {
    console.log(possiblyPerson.firstNm)
} else {
    // console.log(possiblyPerson.firstNm) //does not compile
}

unknown is a newer and a safer alternative to any.

unknown type is (only) the top type (you can assign anything to it but you cannot assign it to anything else, maybe except for any). This is a much better safety than being both the top and the bottom. Compared to any it is more cumbersome to use but significantly safer.

Let’s criticize the unknown a bit. A rough view (IMO) of what type safety is: an ability to separate apples from oranges. If you can assign both an apple to unknown and an orange to unknown then they are no longer separated.
What makes this worse in TS, is its occasional tendency to widen return types to unknown. TS tends to do that if it cannot find a more precise return type, when it tries to apply subtying rules to things like functions, or when it gets confused. We saw two examples of this in the last post:

//compilation bug allows this incorrect code to compile with
// emailBody4: unknown
//this code will accutally work at runtime because 'crazyConfig' ends up not being used 
const crazyConfig : (_: Office.AsyncResult<string>) => void = x => ""
const emailBody4 = await officePromise (curry3(item.body.getAsync)(Office.CoercionType.Html)(crazyConfig)) 

//test: (a: unknown) => (b: unknown) => unknown
const test = curry({} as any)

Also, notice unknown in some of the blooper examples from the previous post:

//these should not compile but they do. Names are consitent with previous post and the linked github repo

//const nonsense2: <T1, T2, R>(a: (ax: T1, bx: T2) => R) => (b: unknown) => (a: T1) => (b: T2) => R
const nonsense2 = curry(curry) 
//const nonsense3: <T1, T2, T3, R>(a: (ax: T1, bx: T2, cx: T3) => R) => (b: unknown) => (a: T1) => (b: T2) => (c: T3) => R
const nonsense3 = curry(curry3)
//const nonsense4: <T1, T2, R>(a: (ax: T1, bx: T2) => R) => (b: unknown) => (b: unknown) => (a: T1) => (b: T2) => R
const nonsense4 = curry(curry(curry))

and we will encounter more examples of unknown widening in future notes. I would be happier if many of these examples resulted in a compilation error. Current status quo reduces safety of TS code.

Let’s look at how unknown makes things like === more complex. I really love the fact that this code (a contrived example but generalizes easily to real situations) does not compile:

//Compilation error:
//This condition will always return 'false' since the types 'string' and 'number' have no overlap.
"some email body" === 1 

However, this does compile:

("some email body" as unknown) === 1

and so does this:

emailBody4 === 1

Let’s bring in the type hole _<T>(): T from the last post. The type hole is a convenient way to ask the compiler type questions.

//hovering over res and _ allows me to see the typing of '===`
const res = _() === _()
(1 as 1) === _()
_() === (1 as 1)

So the “imaginary” type signature of === is:

declare function eqeqeq(a: unknown, b: unknown): boolean

Except

eqeqeq("some text", 1) //compiles

"some text" === 1 //does not compile

In fact, === does not have a type. It is a built-in JS operator. TS applies semantic narrowing rules to the code that uses it.
This complex approach is needed to provide type safety while maintaining compatibility with JS.
TS’s semantic rules prevent certain types like someText === someNumber from compiling, except, this safety is somewhat fragile and breaks when someText or someNumber are accidentally widened to unknown by the type inference. TS uses a similar approach for other built-in JS operators. (We will discuss the crazy === semantics in a deeper detail in the next post.)

General safety concerns about the top: Developers, like me, who had spent decades working in languages like Java and then switched to a typed FP language see immediate safety benefits just because there isn’t any top type. The concern about unknown is that it is used with many JS functions and operators. Such use is not type safe, similarly to how Java’s Object methods are not type safe.
From the type safety point of view, these JS functions and operators are not implemented well either. Consider for example JSON.stringify which accepts any. Does this expression (it returns undefined) make much sense to you: JSON.stringify(() => {})?
Generic functions lose safety too, generics are not generic if a generically typed function parameter can use a specific JS function (like the JSON.stringify function).

Something like unknown is probably the only way for TS to achieve JS compatibility, nonetheless unknown is not ideal.

I will come back to this discussion again, I plan to discuss the complexity of TS types. I will also return to the unknown type itself in the future in a more theoretical setting.

Honest typing conventions

These notes will be a little ranty (you’ll probably ask: “Did you read your other notes?”). Any coding convention is effectively a hand waving rant. That is why we use types, so we can rant less!

One of my former colleagues liked to use the phrase “gentlemen’s agreement”. It means an agreement between developers to self impose certain limitations on the code they write. These limitations are not enforced by the compiler, only by developers who agree to abide by the set rules. Coding guidelines, design patterns, you know what I am talking about.

There is a term in Programming Language Theory called parametricity. Roughly speaking, a language that supports parametricity can assure that a generic function cannot discover what is the type behind a type variable. Remove the top and the bottom from the language too. You are left with very precise types. As an example,

declare function someName<T>(t:T): T

could be only implemented as an identity. Incidentally, there are a few languages that support strict parametricity and a few that come very close, for mainstream languages, parametricity is an gentlemen’s agreement.

Can you write a whole single page app in TS and give it that signature? I bet you can.
We would probably not call it a type-lie. Calling it not descriptive would probably be more accurate. Or, maybe just not the best design?
If some type definitions are better than others, which of them are better? Apps are written so the decisions are being made, but based on what?

I will give you my very type centric view of programming:

1. Well written program means well typed. Well typed means the types express what is happening.
   
2. Types are more fundamental than a programming language.
3. Coding conventions supplement the language in implementing typing concepts.
4. TS (or any programming language) programming needs a balancing act. My approach for writing TS is to balance principled and safe with approachable and informative. That balance is subjective and project specific, my balance point may differ from yours.

Expanding on 2:
TS type checks my code, I type check TS (last post). A library (e.g. office.js) provides types, I type check these types and fix some of them (this post). Developer interventions are needed. Understanding of types does not change with a programming language environment. The cumbersomeness of their use does. TS is, comparatively speaking, not that bad.

Expanding on 3:
In TS, almost any program can have almost any type. I can implement

function program(): void {...}

and do almost anything I want in that code.
It would not be very clear if most of my types looked like this. There needs to be some coding convention that discourages such code.
Enforcing some level of parametricity when implementing generics is another example of a coding convention.

The goal is to move from designing programs to designing types.
This post suggests that types are used to define coding conventions.

So, besides guarding parametricity, what else can we do? Here are some bootstrapping ideas:

Referential Transparency

Referential Transparency is an FP topic but is also very relevant to types and crucially important to the discussion of “type honesty”.

A function is referentially transparent if it does the same thing every time it is called. Referential transparency comes with clear type signatures. The output needs to be a function of the inputs and of nothing else. You can do things like curry or partially apply, but you cannot say, retrieve the current time and act on it (that time parameter would need to be provided as input).
For program:() => void to be referentially transparent would mean that the implementation does not do anything, just returns.
IMO well written programs identify and separate the referentially transparent parts.

In TS, referential transparency is a coding convention. I will use React.js example to demonstrate this. Readers not familiar with React should think about creating a function from some model (Person in this example) to an actual part of the HTML DOM. Here is my example of a vanilla React component type (I like React to be vanilla as much as possible)

const PersonCard: ({ model, onChange }: {
    model: Person;
    onChange: (_: Person) => void;
}) => JSX.Element

Hopefully, the implementation does not use any hooks, it only uses the parameters (I call them setters¹ and getters) to create bits of HTML with event handlers. This would be an example of a referentially transparent React type. It also would be an example of a very explicit type that is very “honest”.

Many developers will very much disagree with me on this. E.g. many will prefer to encapsulate state handling inside components. I do not intend to argue which approach is better. I will just point out that encapsulation is secretive in the type definition and I am looking for transparency here. Many parts of React code will require some use of hooks, my approach is to do that only when I have to and to keep the hooks outside of my main components. It is not about not using hooks, it is about not having them all over the code base. The goal is to make things very type-explicit. It is an IMO.

Such type is also self documenting.

Expanding on my point 4: IMO, the best communication tools for developers and the best documenting tools for the code, in that order, are: types and tests. I will only focus on the first.

Types as documentation

When I write TS, I want my types to be very informative. For example, compare these two slightly modified versions of the above React component:

const PersonCard: React.FC<{
    model: Person;
    onChange: (_: Person) => void;
}>

vs:

const PersonCard: React.FC<Props> //Commonly used 'Props' type alias defined next to 'PersonCard'

I like the first one better.
And, I am not suggesting the names for the setters and getters here. I would be equally happy with this:

const PersonCard: React.FC<{
    get: Person;
    set: (_: Person) => void;
}>

There is no safety benefit in doing this. Communication, documentation and accessibility are the only goals.
I like to think about a modernized definition of the KISS principle: “Simple” is a lot of very transparent types.

Next Chapter

There are parts of TS that I absolutely adore and I will talk about them. The complexity of TS types is another big topic to discuss. Complexity causes compilation issues (we will encounter some new bloopers) and makes the language hard to use.

Here is the link: Part 3.

I am working on these notes during the 2021 holiday season. Merry Christmas, Happy New Year! Stay happy and healthy!

Please Leave Feedback in: git discussions

Disclaimers: (imagine this is a very small font, read it very fast in a half-whisper)
I assume strict compiler flags are on, something you get by default with scaffolding, e.g. using create-react-app my-project --template typescript is close enough.
The code examples have been tested with TypeScript v4.4.4, v4.5.2, and v4.6.3.
office.js examples are based on https://appsforoffice.microsoft.com/lib/1.1/hosted/office.js and @types/office-js@1.0.221 (these match the current scaffold for office.js/React).
This post is a pandoc conversion of markdown document and code examples are not interactive.
Most of the code examples are published in ts-notes folder in this github repo: ts-experiments.

Introduction to the series

“TypeScript began its life as an attempt to bring traditional object-oriented types to JavaScript so that the programmers at Microsoft could bring traditional object-oriented programs to the web. As it has developed, TypeScript’s type system has evolved to model code written by native JavaScripters. The resulting system is powerful, interesting and messy.”

From typescriptlang TypeScript for Functional Programmers

I wanted to write a short post about my experience with TS types, I ended up with a draft the size of a short book. I decided to split it into digestible installments and publish it as a series of shorter posts. The series will be about the powerful, interesting and messy types in TS. This post is the first in that series.

Here is my plan:

• Part 1 (this post). Is a warm-up. Part 1 has been motivated by a project at my work that uses TS. I will show code examples that are hard to compile. I will discuss strategies and methods for resolving compilation issues. I will present code examples that compile but really, really should not, and code examples that should compile but surprisingly don’t. I will also summarize my overall experience of working with TS.
  This series needed a JS library with TS bindings to draw examples from, I decided to use office.js and Part 1 introduces it.
• Part 2. Will be about keeping types honest. Are runtime values consistent with the types? We hope they always are but, especially in a gradually typed language like TS, types will sometimes lie. We will see concrete examples of type dishonesty from office.js. Part 2 will cover the notorious any and its safer cousin unknown, the type coercion (casting), and TS’s type guards. I will also discuss (or rather rant about) coding conventions for transparent, self documenting types.
• Part 3. Will cover some of the TS type safety features that I absolutely love. Throughout the series, we will encounter several examples where TS compiler does not work as expected. This part will discuss questionable (and arguably incorrect) semantics of subtyping variance and of narrowing. It will argue that what TS is and does it quite complex. Complexity is the likely cause of errors in TS programs and in the language itself.
• Part 4, Part 5. Will be more theoretical. Notes in Parts 4-5 will discuss topics such as TS’s structural, recursive types, subtyping, phantom types, type variable scoping, higher-rank polymorphism (TS supports a version of it!), and type level programming. I will show a trick to increase type safety that prevents widening to unknown or other supertypes.
  
• Part 6. Will be a wrap-up with some final thoughts.

Why am I writing these notes?
To be honest, it is because I am really impressed and excited about some of the type safety features in TS.

Despite being a superset of JavaScript, TS stands out among mainstream languages as one that supports some interesting types.
There exist a tiny but important feedback loop: the more developers play with types the more they will end up being used.
So, to be perfectly honest, the goal of these notes is to simply play with some interesting types and see how the compiler reacts.

IMO, to master something is to understand its limitations.
So, to be brutally honest, the goal of these notes is to explore the TS compiler limitations.

Target audience and prerequisites. I assume that the reader is interested in types and either uses or considers using TypeScript.
Types tend to be related to FP. There will not be much FP in these notes. However, I will use some basic functional programming concepts, like currying, without explaining them.
TypeScript is a superset of JavaScript with type syntax very similar to any other C-like language. These notes will probably be hard to read without some experience with JavaScript or ability to read C-like types.

About the author. I am spearheading a rewrite of a legacy frontend component at work, the goal is to rewrite it using the new React.js and TypeScript. In recent years I have been spending all of my time in the backend designing, writing, and maintaining Haskell programs. Haskell code has a lot of types. Thus, I use types a lot. Types allow me to code faster, safer, and with much more confidence.
I wear a hat with types on it when writing TS.
I love Programming Language Theory and have some experience and lots of interest in compiler and language design.
I wear a very thin headband embroidered with PLT symbols under my hat (should be mostly invisible in this series).
All of this gives me a different (compared to most typescripters) perspective and a reason to write these posts. For some readers, parts of these posts will feel strange. Established practices like overloading will be considered a bad thing, writing experimental code (that won’t even run) to answer type questions will be a good thing. Strange is a corollary of different.

What is TypeScript for? Is it just a JavaScript add-on used to prevent typos and trivial code errors?
Or, will TypeScript more fundamentally change the way the code is written?
Please have these questions in mind when reading these notes.

We will cover a lot of topics.

“And we never say anything unless it is worth taking a long time to say.”

J.R.R Tolkien and Treebeard about discussing types in TypeScript

TypeScript is great!

It literally took me less than one minute of playing with TS to get excited about it.
Just look at the union types (using a somewhat contrived example):

type Person = {firstNm: string, lastNm: string} 
type NullablePerson = Person | null

const getName = (p:NullablePerson): string => {
    //const tst1 = p.firstNm //does not compile
    if(p===null){
        //const tst2 = p.firstNm //does not compile
        return "John Smith"
    } else {
        return p.firstNm + " " + p.lastNm //compiles
    }
}

How cool!

Talking about my “literal” excitement, my next play example implements Either (I am not trying to implement my own Either type, only to play with the language):