Category Archives: Uncategorized

A closer look at functional testing heuristics

Standard

I consider myself to be a tester of the functions of software. It means my expertise is in looking at how (or why, or when etc…) systems and applications function. It also means that I am less experienced in non-functional aspects of systems such as usability, performance or security. I believe that those aspects can be considered areas of expertise in themselves.

I think most testers start out as functional testers. In some introductionary course on software testing they encounter a number of software testing techniques that can be used to evaluate the functions that are contained in an application or a system. Most commonly these techniques aim at paths and decisions and come from a more or less mathemical examination of the function. Combinations and coverage are important aspects of these techniques.

Now profiency in functional testing is not a common trait. For a number of reasons the functional tester is thrown off the path of developing skills in functional testing. One reason is the testing career path. Gaining competence in functional testing is not rewarded as much as climbing the ladder and becoming a test coordinator, a test manager, a test coach, a quality consultant etc… Furthermore; if we define functional testing by a restrictive set of testing techniques then it seems that the only way forward in functional testing is gaining experience with those techniques, which is a dead-end street. Another reason could be that functional testing is not regarded as a valuable area of expertise such as performance testing or security testing. It is remarkable that the latter areas in software testing have gurus expanding the craft. I find it hard to come up with the name of a ‘functional testing guru’. Well, actually there are some in the context-driven school of testing.

Most of the time you see functional testers going off on a quest based on a mixture of techniques, instincts and past experiences. The latter two most of the time provide valuable insight into the functions of a system. They seem to find the better bugs because the functional tester experiments with the application in ways that had not occured to the programmer or the designer. So it seems that there is a lot more to functional testing than just techniques, but there is little explicit knowledge of what more there is.

I stumbled across our common lack of profiency in functional testing quite a number of times. In functional testing most of the time there is that nagging feeling that you’re missing something important but usually it takes a lot of effort to find out how to get to that important thing. For example; I may have covered the paths of a function using some technique and found no bugs. Yet despite that the function could still cause the application to go wrong in a fascinating way. In one project I developed a test strategy based on scenarios (see, for example, Soap Opera Testing [PDF]), which was quite a valuable and very useful approach in itself. Yet I had the feeling that this approach covered a minor part of the ways the functionality of the system could be used. I was looking for other perspectives I could use to research the system and experiment with it.

Most of the time I use James Bach’s SFDPOT heuristic to change my perspective. As a side note, I find that many of the functional testers I meet are awkwardly ignorant of this heuristic. On the particular project I mentioned the heuristic did not include some of the perspectives I was looking for. So through the analysis of a couple of sessions with users on risk assessment, I came up with a couple more. Yesterday I sat down and extended my list to what you see below. This list should probably be turned into a heuristic. Each of the items could use some clarification. I will leave that to later posts. Additions, of course, are welcome!

  • Patterns
  • Sequence
  • States
  • Concurrency
  • Confluence
  • Synchronization
  • Sharing
  • Interactions
  • Repetition
  • Hierarchy
  • Dependencies
  • Parameters
  • Rules
  • Configuration
  • Constraints
  • Resources

A final note to close this topic. Yesterday I was reading up on qualitative research (Qualitative Research: An Introduction to Methods and Designs). I got to the part on Grounded Theory and coding. This theory, in short, is a research method in which a theory is formed based on observation. Coding is the abstraction of theory parallel to the observation (for example; the text of an interview with a person). If we, as testers, consider, for example, our bug repositories as raw data and are able to abstract a theory from that by coding, we should be able to come up with numerous functional testing heuristics (such as the list above). One fact is that there is probably no shortage of bug descriptions in repositories world wide. The other fact is that we hardly ever use them to do valuable research. Such failure to scientifically use the data we generate must be one of the reasons for the immaturity of our craft.

Disconnected

Standard
In the Netherlands the testing practice is largely dominated by the TMap test management approach and by other methodologies such as TestFrame and ISTQB. Throughout the years I’ve spent as a software tester I’ve become concerned that the testing methodologies we value and apply are moving us further away from reality.

It could be argued that the Dutch have a natural tendency towards methodologies, standards, rules and processes to govern complexity. Although I cannot offer scientific proof for this statement it feels like we think that by putting a tremendous amount of intellectual effort into complex problem we can abstract from that problem a generic framework, a mechanized pattern, to govern and master the problem’s domain. I think our Delta Works – the effort to defend our country against the onslaught of the North Sea – counts as such a framework.

In software testing the seven hundred plus pages of TMap Next outline in very specific and careful detail the steps we should follow to tackle software reality. While I believe that this work is the result of a huge intellectual effort concerning the amalgamation and sublimation of an extensive body of historic knowledge in software development, I also believe that it has little to do with software testing. I would even go so far as to say that software testing, as defined in our methodologies, does not exist. There! I officially upped Alberto Savoia.

The disconnect is the gap between the finely tuned parts of the mechanism described in a methodology and the supreme insight that is a prerequisite for the establishing of the quality of a solution specified in software. The parts are the scripted test cases, the test plans, the risk matrices, the product risk assessments, the requirements analysis, the test environment intake checklist, the test case prioritization and the test effort estimations.The insight comes from a myriad of sources that are often specific scientific fields in themselves, such as mathematics, systems theory, epistemology, experimentation, analysis, design, language, critical thinking, decision making, communication and learning. While competence in some of these areas is acknowledged to be a prerequisite for testing even in the TMap approach, the methodologies are quick to assume that these elements, which are at the core of dealing with complex problems, are inherently present in the software tester.

As soon as we define testing solely by the machine we use to effect it – if we set out by thinking about testing in terms of constituents driving a process – we willingly refuse to notice any of the relevant aspects of software such as complexity, communication, knowledge, change, uncertainty, design, analysis and reasoning at large. From this principal exclusion, which either supposes that the tester inherently possesses all competences to deal with the relevant aspects of software or supposes that such competences are not needed, there is no turning back.

If, for example, we use a test case to test something, then ideally we use the right test case, at the right time, under the right conditions, executed by the right person. If we define testing by specifying what a test case is but omit all competences that are needed to adequately assess what is ‘right’ then what remains is an empty shell. And it is not safe to assume that the competences needed for the adequate assessment (the “judgment and skill” as mentioned in the principles of context-driven testing) grow on trees bountifully in La La Land.

If we want to define testing let’s start by explicating the competences that are required for making an adequate assessment.

My four favourite articles

Standard

I would like to mention my four favourite articles of the moment. Most of the time when I read something on software testing or software development it is interesting or worth finding out more about. Sometimes the subject matter is difficult and it takes a while to figure out the value of what’s written down.

The four articles I like to mention are nothing like that. They convey a message so powerful that reading them has changed the way I think about software and therefore about software testing almost instantly.

Richard Feymman – Cargo Cult Science [click here for a pdf of the article]

Richard Feynman

Richard Feynman

This is not truly an article; it is Richard Feynman‘s 1974 commencement speech for the California Institute of Technology. Here Feynman, perhaps one of the greatest scientists of the 20th century, speaks about scientific integrity in relation to experimentation. When I apply this to software testing, which is – after all – a form of experimentation (though not always a very scientific one), I feel that we need to be true to ourselves and to others about the experiments we conduct. That, in software testing, we need to be unrelenting in our experiments and in our quest for knowledge.

Here’s how mr. Feynman concludes his speech.

“So I have just one wish for you – the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a special need to maintain your position in the organisation, or financial support, or so on, to lose your integrity. May you have that freedom.“

Tom DeMarco – Software Engineering An Idea Whose Time Has Come and Gone [click here for a pdf of the article]

Tom DeMarco

Tom DeMarco

Tom DeMarco is one of the great thinkers in software. Through the book Peopleware he became a personal hero of mine. In this short article he refutes his early work on software metrics and that alone, in my opinion, shows the man’s true spirit and bravery. It is a change of heart that should have a significant impact on the way we view our craft. In essence DeMarco criticizes the way we scrupulously and against better judgment hold on to predictability in software development.

DeMarco’s closing words shed a good deal of light on the true nature of software.

“Consistency and predictability are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.”

Phillip Glen Armour – The Five Orders of Ignorance [click here for a pdf of the article]

Phillip Glen Armour

Phillip Glen Armour

The most important step I think we still have to take in software testing is to learn what software really is and use that knowledge for better testing. One way of looking at software is presented by Phillip Armour and I sincerely like how his definition radically throws the mind off the ‘software is a verified and validated factory product’ track and opens up a landscape of possibilities.

Philip Armour states in his article that software is not a product but a medium for the storage of knowledge. Developing software is an attempt to reduce ignorance, to gain knowledge about what the software should do. In my opinion testing is a craft that can be used to gain knowledge about the software. Armour mentions that in software development we need tools that generate questions in order to find out that there are things we do not know about the software. What I learned from my experience is that testing throws up these immensely valuable questions frequently. They are, at the very least, a byproduct of testing. As long as we think of testing as a crude ‘pass of fail’ tool we are unable to extend our competencies into such areas.

Jack W. Reeves – What is Software Design? [click here for a pdf of the article]

Jack W. Reeves

Jack W. Reeves

Jack Reeves’ 1992 article What is Software Design is perhaps the keenest article I’ve read that refutes the waterfall way of creating software. Reeves neatly points out why we should not compare the making of software to, for example, the creation of a hardware or any factory product. His main point is that it costs next to nothing to produce software; it is the design (the code) that is the focal point of creating software.

I look at it this way; by chopping up the development of software into phases, placing testing at the end of the manufacturing line, we suggest that testing is meant to verify and validate a finished end-product. Often testing finds errors that send us back to the drawing table. From this perspective testing leads to costly rework on a product that was supposed to be finished. It is like sending a car back to the factory because we found that the engine stalls when a speed of 30 km/h is reached, the seats are mounted backwards, there are only three wheels and the steering wheel is triangular. Yet such a car would never be built because the design would never go to production.

Testing, in Reeves’ perception, is a part of design; its purpose to validate and refine the design. I think this distinction places testing where it belongs. If we look at testing as a quality assurance (policing) activity – a process control methodology – we willingly impair the ability of testers to aid software design by constructing multiple layers of abstraction on top of the software ‘manufacturing’ process. The overhead involved takes the testers’ time and mental focus away from the actual testing and from the design.

Some words on design and testing by Jack Reeves.

Am I crazy? I don’t think so! Compiling and linking 50,000 lines of C++ code on your 486 may seem to take forever, but how would you like to assemble a circuit card with 50,000 discreet components, or build a bridge with 50,000 structural elements? We don’t construct mathematical proofs of software correctness or run our code through symbolic executors because it takes less time and effort to just build it and test it. We probably would get better software if we did more of the former, but we don’t. Why not?

There are probably lots of reasons, but I would like to suggest that many of them derive from our failure to consider testing and debugging as part of the software design process. We would like for it to go away completely. Since it will not, we try to treat it as some sort of “quality assurance” function and spend as little time, effort, and money on it as we can get away with. We consider it a shame of the software industry that testing and debugging take up half the typical software development lifecycle. Is it really so bad? I suspect that most engineers in other disciplines haven’t a clue about what percentage of their time is spent actually creating a design and what is spent on testing and debugging the result. Some industries are probably better than software. I am pretty sure that other industries are actually much worse. Consider what it must take to “design” a new airliner.