Tag Archives: regression

Takeaways from Alan Richardson’s “Dear Evil Tester”

Alan Richardson is the Evil Tester. He talks and writes about evil testing, about how to be the best software testers we desire. He gives talks at conferences, offers consultancy services and training over at Compendium Development, and writes about Selenium on Selenium Simplified. He’s been in the testing industry for many years and certainly knows its ins and outs, and he’s continuously trying to keep himself and other software testers in the field updated and valuable. His book, “Dear Evil Tester“, is his way of providing us with an approach to testing founded on responsibility and laughter.

Some lessons from the book:

  • The point is we don’t need permission. We should do whatever it takes.
  • If you short change yourself then that isn’t self-preservation. It is allowing your skills and integrity to slowly rot, wither and die. It is condemning yourself to victimhood as a response to other people’s actions. Don’t do that to yourself. Always work to the highest level that you can be proud of. That is an act of self-preservation.
  • I try very hard to get in the habit of evaluating myself. Not in terms of the actions of others, or in terms of their expectations of me, or in terms if a ‘generic’ tester. I try to evaluate myself in terms of my expectations of me. And I try to continually raise my expectations.
  • Stop doing the same things you did on the first day.
  • Test with intent, with minimal up-front planning, taking responsibility for your path and the communication of your journey.
  • Anyone can test. And you need to be able to test better than anyone off the street. And be capable of demonstrating that you can test better than anyone off the street.
  • I don’t believe that systems ‘regress’. I believe systems ‘are’. I believe systems can exhibit behavior that we don’t want. I believe systems can exhibit behavior that we don’t want and which we have seen the system before. But I don’t call that regression.
  • The best test tool to help me test is my brain. I then use that to help me find other tools that help me observe, interrogate, and manipulate the system at the different technological levels I’ve found the system to be composed of.
  • I don’t think I do test the quality of a product. I think I test ‘qualities’ of a product. I test observations of those ‘qualities’ against models that I have made of the ‘qualities’ independently of the product. And then I put the product into different states, allowing me to observe the product in different ways. And when I see a difference, I can then offer commentary upon the differences between my observation and its comparison to my model.
  • The hard part is not finding tools, and making lists of tools, and learning how to use tools. The hard part is figuring out what you need to do, to add value to your test approach.
  • I look at my model of the system and I see parts I’m not observing, or manipulating, or interrogating. I try and work out what risks I’m not testing for, because of that. Then I work out which of those risks I want to target. And I go looking for a tool to help.
  • Good and Evil are relative terms. If you accept that, then words hold less power over you. Choose your actions well. Stop labelling them ‘Good’ and ‘Evil’. Take responsibility. Build your own models. Do what you need to. Do what you have to, to be the change you want to see.
  • Not all testers are evil, just the good ones.

Data-Driven Testing

For new apps and software features, it’s perfectly normal to exhaustively test the matrix of all possible scenarios with equal importance before release – the happy path, the confused path, the forgetful or greedy paths, etcetera – whenever the tester can. After all, no one knows yet how exactly the new application will behave in Production, and it is always better (and cheaper) to find problems and fix them early instead of reacting as they occur.

For small software that are already in the wild and which do not change often, it is still a good rule of thumb to test everything with equal priority, if the budget allows for it. It’s still nice to catch bugs before the next version release rather than have a client surprise us at odd times with a system failure.

For complex legacy systems which frequently change and usually do not automatically test themselves, it is still a great idea to test features in full-scale, if we can, if it’s possible to perform all those complicated use cases within hours before the deadline. That’s a noble but a hugely difficult task for testers, if not impossible to accomplish, and something that might be a complete waste of time and creativity, or a foolish thing to do, even if it is very important for teams to have confidence in the stability of their applications. For such systems it might be better to re-evaluate what goals does our testing need to reach, re-think what sort of stuff do we want to find, re-learn what services do we want to keep providing to our customers as well as how effectively do we want to provide those services, and go from there. It might be better to spend some of our time on data monitoring and analysis, on understanding how our clients actually use our products and focus our testing those usage scenarios. It might be better for us to amplify testing and developing the features that our clients love. And it might be more worth it to regularly identify the riskiest parts of the system and improve their design and testability, rather than trying to keep up with the matrix we know we couldn’t possibly deal with everytime.

Apps evolve, and I think that the testing that we perform needs to change too according to our current contexts.

About Software Applications And Their Required Regression Testing Time After Each Update

About two years ago, when I was a new software tester in my current company, I started estimating the time I need in performing regression testing on each of the organization’s major products. I wanted to know how much time was enough in order for me to do my tests properly, without rushing and without dilly-dallying, because it helps me prioritize and pace tasks. If asked if I can deliver results after a specified period, I would be able to answer yes or no immediately. If I say I can’t deliver on the initial negotiated time, I would be able to tell my superiors how much more I require to be able to provide the information they want.

For one application, I found out that I have to commit a day, provided that tests are focused, not rushed, without much distraction from everyone else. I got my estimation where I then based all my decisions and made guided expectations. For that product, one day provided enough testing time such that every further project release for that software often went well. There were no troubles in the required testing time. I had no worries in pacing myself well.

That is, until recently. There was a scheduled major release and I told my boss I can finish testing before the deadline but I wasn’t able to, even after giving overtime work. Imagine my surprise: that day, the estimated testing time that I was so sure of looked like it wasn’t enough.

Several days later, I realized two obvious things that I never thought could lead to a disaster in decision-making:

  1. software applications grow in complexity as they continue to be updated, and, because of that,
  2. the regression testing time required for growing applications also needs to increase by a properly factored amount.

The Problem With Regression Testing

is that it can easily bore the person who is doing the tests.

Imagine yourself repeatedly checking each form, each button or field, each functionality for completeness, for compliance to system requirements and see if you don’t get tired of doing it over and over again, after a while. It is dragging, because you already know how things should work and there’s already a list of what particular things (in your mind) to look out for. The new feature is more fun to test just because it is new – the tester is almost always bound to stumble upon bugs of various sorts.

But, of course, the importance of regression testing can’t be stressed enough. It is there because maintaining the quality of existing system features is as important as the usefulness of the new requirements. It is then up to the creative mind of the tester to find ways to counter the boredom.

One suggestion: learn software test automation.