Test Pappy

A topic that keeps coming back to me is pondering about simplification of processes. What is a simple process and what is a complicated or complex process? In which constraints does it make sense to have more or less details? Just because it looks simple doesn’t mean it is simple. And just because it looks complex doesn’t mean it is complex.

I’ll start again with a wood working example or two. Let’s look at the process to build a box. A rough picture of the result is provided.

The process instruction could be:

• Take four pieces of wood of size X by Y
• Joint the corners
• Insert bottom

Or the process instruction could look like this:

• prepare your stock by milling it to 15mm thickness
• cut 4 pieces of size X by Y each, grain direction is important
  • if possible cut them from one long piece, so that you have continuous grain
  • prepare stock for the bottom by milling it to 6mm thickness
  • in the 4 pieces cut a groove on the table saw or router table with 6mm width, 10mm from the bottom edge
  • <now comes a long instruction how to place and cut the dovetails>
  • and the list goes on and on

In the end of both processes you will get a box, don’t you?

Well, the boxes might look rather differently. Or they look exactly the same.

Now what when you have to make 10 boxes? People following instruction no. 1 will come up with a more or less implicit process. People following instruction no. 2 will just continue to do so. Will the process for all look the same after box 10? Maybe. Probably not.

Another example from TV. If you know the show Great British Bake Off on British television, you might know that the second round is the “Technical Challenge”. And the recipes and ingredients provided range from measured with detailed recipes about measurements, order of steps, temperature and length to bake. All the way to more than enough from each ingredient and the instruction: Make a Genoise sponge.

The result will be a Genoise sponge, right? If someone has baked dozens of Genoise sponges in the past, they know what to do. If someone roughly knows the recipe, they might land close enough. And if someone has no clue what a Genoise even is, you might be lucky to get something edible.

The consistency of the result is all about guardrails, experience and expectations in the process. If you have a picture of the box and detailed instructions or just the vague idea of building a box, the results will defer.

A simple looking process doesn’t mean it’s easy to follow. It means you need experience to fill the gaps that are not prescribed. Is that experience always the same? I doubt it. On the other hand, a complex looking process might be easy to follow and requires less experience. Does that mean anyone can follow it? Not necessarily. There might still be knowledge, skill and practice involved.

Now look at your processes at work. How detailed are the instructions and expectations in there? I have seen e.g. bug reporting processes that are extremely detailed. Questions to answer with examples, information to provide, guidelines for evaluating impact and severity.

And I have seen processes where nothing was written down. A bug ticket type was available in the work management tool – no not Jira, back then, and the implicit expectation to log bugs when they are found. The difference in quality of the bug tickets is accordingly. The more details are provided, even the rookiest rookie was able to fill out a bug ticket with as many information as possible. In the other system, the quality ranged from “X doesn’t work” to full reproduction steps, attached database dumps, log files, etc.

In the workshop that I still regret to not have lead differently – “Quality Eats Process for Breakfast” – I wanted to guide my participants to the insight, that the quality of your process decides about the quality of the product. And when it comes to the ultimate evaluation of quality – the user’s or customer’s evaluation – it is hard to pre-bake that into the process, if you don’t have enough information. What is it that my customer values in the product? How can I take care that this is included in the result?

In my favorite example to explain to me how to make coffee – I’m not a coffee drinker – I usually get a rough list of steps how to prepare a cup of coffee. I don’t drink coffee, but sometimes I want to make a cup of coffee for someone else. Most times, with the given steps I would have produced something that reminds of coffee – a brown-ish, hot-steaming liquid. If it tastes good or not, I don’t know. After chatting with one of my best friends, a coffee lover, I learned about different types of beans, coarseness of grind for different styles of coffee, different water temperatures and pressures. The amount of coffee powder to use for one cup was defined, the size of the cup, and so on. Can I make a good coffee with these instructions. Yes, I can make a coffee and with repeatable results. And the factor “good” is evaluated by my friend. If other coffee drinkers come to the same conclusion, I don’t know.

Now think e.g. of your software development process. How detailed are certain steps defined? Do you need consistent results across people and teams? What measures are in place to get consistent and good quality results? How much training and experience is necessary to follow your process? Do you have constraints in your context that need either explicit shared knowledge between everyone involved or detailed process steps? Where is the Goldilocks zone to provide enough detail to get consistent results and enough freedom to rely on the experience and solutions by the individuals to deliver what is asked of them?

The more vague the instructions are the wider the result space will be. The more detailed the instructions are the smaller the result space will be.

My advice here is to experiment in your context. Processes can be improved like software. Improve them bit by bit. Add something, remove something. But don’t forget to tell people about the changes!

And it also depends on how narrow your result space needs to be. If the results vary too much, add more guide rails.

I’m now going into my workshop to turn a pepper mill. The instructions I have are only the exact diameters and depth for the holes and the distances between certain points. e.g. the length of the lower element of the mill. That’s the parameters set by my pepper mill instruction manual that is available from the same source where I get the mill inserts. When following these instructions, I’m sure that the result will be a piece of wood that the inserts will fit in. In the end it will be a pepper mill. But it’s up to my experience, choice of design, availability of wood, to come up with something useful and nice.
In the past I also have bought a pepper mill kit with detailed instructions. Provided was the wood, the inserts, and a 20 page set of instructions to produce the exact same mill as printed on the title.

What is simpler? The sketch with hole depths and diameters or the detailed step by step instruction?

This quote is by American novelist and poet Jack Kerouac. And ever since I read this, I aim to achieve it. So far without success. Finding the right and simple words is hard work. The same is said with the quote “I wanted to write you a short letter, but I only had time for a long one”, which is assigned to Goethe, Lichtenberg, Pascal, Swift and others. Meaning, that boiling something down to the essential is hard work.

My problem is also my love for complexity. I enjoy complexity to a certain degree. Especially when I’m able to understand it. But that is also the point I usually trip over. When I want to explain some part of a complex system, I need too many words. I want to explain all the elements, and rules, and dependencies, and perspectives. Because in my mental model they matter. And as I don’t know if my “reader” is aware of all these wonderful connections, I have to mention them as well.

It’s hard to focus on explaining just one aspect, when there is a whole system around it.

There are many issues with long explanations. Writing them can lead you in the wrong direction, you loose focus of what you actually wanted to say. People don’t read or listen, because too many lack the attention span or the willingness to pay attention.

The issue with short explanations is that people might misunderstand it or don’t know where to put it in their mental model. You might simplify it too much, so that it looses what you tried to say.

Complex systems are made of simple rules and elements, just with too many of all of them. When picking some aspect of the system and describing it, there needs to be a middle ground. A Goldilocks zone between simplicity and complexity.

Drop everything that is not necessary. But not necessary for whom? You might not know your audience. You don’t know what they already know. For some it will be clear with just one explanation. Others need more context. Some might not have any foundation. So, where is the middle ground?

The answer is, I don’t know it. I try to reduce context. But then I find too many good examples not to share. What is better, have a short, precise statement with some examples, or write it more convoluted that can then be applied by the audience to their context?

One example is conference talks. Some of them. They describe the problem, describe a solution with usually three bullet points. Great! Love those talks. Usually these don’t take more then 5-10 minutes to come to this point. But then there are 20-30 more minutes of time to fill. And then they repeat the points over and over.
One of my favorite sessions of all time was a 2015 EuroStar session called “Lightning Strikes the Speakers”, where I think 8 people had 5 minutes each to present a topic. It was on point. Clear messages. No fluff. Maybe too many topics and ideas in a short time frame to digest, but they were all on point.

I hope to break down my future topics even further. Stay short. Stay meaningful. One example, and be done with it. Fingers crossed.

This weekend – when you read this, last weekend – I listened to my favorite woodworking podcast. And the topic was “Gratitude, Appreciation and the concept of Enough“.

“Enough” is a topic that lately rumors in my head all the time. When I look at politics, careers, economics, global warming, ecosystems, billionaires, and all topics along this line. Why do we always need more, more, more. This fundamental capitalist thought of always wanting more that is deep engraved into society. But that is not what I want to talk about today.

Perfection and completion are destinations at the end of a very long road. This road leads along a very long part called “enough”. If you are not making a jig-saw puzzle, then “enough” is likely to be found around 80-90% of perfect/complete. Sometimes it’s even less, sometimes maybe a bit more. When you look at the Pareto principle, also called the 80/20 rule, you realize that the further you are down the road the longer the rest is going to take. Pareto says that for 80% of the work you need 20% of the time, and for the last 20% you need 80% of the time. Or as we call it in IT, the 80/80 rule. For the first 80% you need 80% of the time, and for the last 20% you need the other 80% of the time.

Be it requirements engineering, programming or testing, in the world of IT we regularly have to be content with “enough”. We might strive for perfection, but we don’t have the time, or sometimes even the ability to reach perfection. There are opposing quality criteria, unknowns about the future, unclear requirements, missing feedback and most of all, not enough time. We have to settle with enough for now on a regular basis.

The old tale of “testing is never done” comes from the fact, that there is always something more to test. Especially as testers we have to find peace with the fact that sometimes we are “done”. Even when it is not enough. The time to cover the remaining scenarios is often multiples of what you need to do for a solid coverage. This is why we need a risk based approach. Cover the biggest risks first and then the next and the next, until time runs out.

It’s tough to defend decisions on why we tested certain criteria and others not. If you don’t do a full-fledged risk analysis first to determine what to test first, second and so on, then it will always be a topic of trust. Your project trusts you that you test the most important aspects first, and that you stop when you have a good feeling about the latest changes.

The more experience you get with your product/project and also with testing in general, the easier it will become for you to identify “enough”. But how do we teach “enough” to the next generation of testers? All testing courses that I’m aware of teach you how to define all kinds of test cases with path coverage, decision coverage, state transitions, decision tables, etc. This can produce a lot of test cases or experiments to cover. So you learn how to combine them to save time. But that still leaves too much. And we have not even spoken about testing of performance, resilience, accessibility, security, you name it.

Over time a tester will establish a mental model of where priorities lie or where tricky situations might be. A tester will develop a certain grasp for the system at hand. This mental model will go further than what the specification can provide. This will be more than what standard test case analysis methods can tell you.

Practice your systems thinking skills, so that you can extend your mental model beyond the specification of the software – in case you even have one. Company goals, clients, user bases, regulations, other teams or departments are all valuable elements to a model around the system that you have to test. You can do simple things and just speak with your colleagues what has been changed, what is important to the customer or users. When you are new to the project, ask them if there were problems in that area before? Ask business people what is most important to them. In regulated environments, speak with your regulatory folks to understand what is an absolute must have.

You will never have enough time to test everything, so every approach to understand quickly when enough is enough is useful.

And yes, you will miss issues. And this will be hard. But you are not alone. Others have missed the issue before you. The person writing the requirement has not put it explicitly in the specification. The developer has not thought about it as well. The code reviewer also didn’t have the case in mind. Maybe the case never happened before. Appreciate what you have done. Learn from the ones that you missed. But don’t go too hard on yourself.

I’m very hard with myself with every bug I missed that had a certain severity. See, I have gained already the experience to be fine with minor bugs. Some bugs are just not relevant enough to cover them beforehand.
Having missed a bug can also mean for me to conclude that I will miss bugs like that in the future. Some issues are just too time-intensive to mitigate.
So, if your software is not putting lives at risk, find a comfortable level of “enough”.

Many years ago I learned the basics about the Cynefin Framework. The very basics. To repeat these very basics, Cynefin differentiates events into five different domains. Simple or obvious, Complicated, Complex, Chaos and Unknown or disorder. This helps to understand how to perceive situations and make sense out of them. Cynefin is based among others on systems thinking, which makes it sometimes a bit tricky to distinguish.

If you don’t know where you currently are, you assume the unknown domain. You first need to observe and conduct some more analysis to find out where you are. Simple is, well that. Complicated needs a bit more information to understand. Complex is not trivial to understand on first sight. And Chaos, well. When shit hits the fan.

Introspectively I found out about myself, that I tend to begin my evaluations in the complex domain. I don’t trust simple things. Is it really that obvious? Can’t be. The world around me is chaotic and complex, and there is barely anything simple left that is not connected somehow with something else.

I remember two or three colleagues many moons ago, who seemed to treat everything as simple or maximum a bit complicated. Just do A to reach B, how hard can it be. Well it was harder than that. Context matters, and context means the rest of the bloody system. So why remove the context? I still don’t understand that approach.

I enjoy a good complex system, no matter how sad and disappointing the insight might be. Economics, ecosystems, machines, politics, companies, you name it. Even in wood-working I don’t treat most things in a simple way. Just do A to get B. It might work in many attempts. But sometimes things might go wrong, and I want to know beforehand why they could wrong. I want to make more informed decisions. And wood is a complex material. You have grain patterns and thickness, you need to know what you want to use it for, the forces it will experience. Sure, you can also glue two pieces together, and maybe it works. Just do it! is a valid approach in wood working.

Maybe it’s my ADHD brain that likes to overthink things, but so far I was rarely disappointed or surprised. Even when being in a work context and you treat an IT system as more complex as it might be for others. You start examining the thing, trying to understand the dependencies and relations. How does it work, when does it not work. If the thing is really simple and it is just do A to get B, then that’s fine. I fall in the better direction. Imagine it’s the other way around. You think it’s just do A to get B and 2 minutes after the code hits production your system goes down, because: reasons. Of course I have fallen into this trap. Not with such a bad impact though. But I remember a few occasions where I looked at a bug, found the code to fix, “just” fixed it and thought it was done. Funny what can wrong sometimes.

As a systems thinker I enjoy complex systems, so I will continue to “land” in the complex domain, no matter how simple the situation might look like on first sight. I have seen too many things go wrong to trust that there are really many “simple” situations.

Stay imaginative!