Not a very long time ago, software development companies could afford to engage developer who could not test and testers who could not develop. For most the industry, it was not necessary for either group to know the technical principles behind software testing or even software development. As the software engineering profession matures, and as software becomes more pervasive in everyday life, there are increasingly important requirements for software reliability, maintainability and security. Industry mush respond to theses changes by improving the way software is tested.
The good news is that the knowledge and the technology are available and based on over 30 years or research and practice. Any good software project will include testing. So what are the ways to do it?
All software testing breaks down into four broad categories.
The first broad category is Self-Reported. As the name implies, the user runs the software and reports on its use. But this category can be further broken down into Direct and Hidden.
With Self-Reported/Direct, an interviewer asks the user to tell them what they liked/did not like about the software. Seems straight forward, but it’s not. While simple, users often act differently or edit what they say, because someone is there asking them questions.
That is, users often tell us what they think, we want to hear. Some say, “users lie”! I don’t go that fair. But user do tell interviewers what they think is helpful. And paradoxically, you don’t want helpful answers, you want the truth. Sometimes that requires being a little sneaky.
To minimise these problems, the self-reported/hidden model is useful. The simplest example of this model is to let the users go at the end of the day. Then sneak into their offices and see the remnants and artefacts of software use. For example, if their monitor has a biz-zillion sticky notes with keyboard short cuts, maybe the software is not that intuitive. If there are dozens of report reprints in the trash, maybe the onscreen reports aren’t giving the user data in a simple way. Or maybe there is no on screen report for the info the user seeks. Reports with lots of notes may indicate that users need more details in the report. And so on.
The main disadvantage with this method is that it’s hard to get interaction data. With brings us to the observed category of testing.
The broad category of observed tries to watch the user without interference. observed/direct is where a tester observes a user, using the software. The observer sits behind the user or shadows them all day. They don’t ask questions under the assumption that, “people lie.” (thanks you Dr. House). Or put more politely, people want to influence the person collecting the data. To combat this, the observer takes notes only of the actions of the user as they interact with the software. Also taking note of any environmental interactions. For example, if the user goes to get a cup of coffee after initiating a report, it may be an indication that the report takes a long time to generate. And the user has learned to do something else while waiting.
There are people who argue that this still has an observer effect, where people change their behaviour when being observed. So there are a number of software products that test as observed/hidden. The most famous are all the mouse tracking packages that track where, on screen, the mouse goes and is clicked.
Industry mush respond to theses changes by improving the way software is tested.
Eye tracking has been shown to follow mouse movement (makes sense). So many software packages allow desktop and web mouse tracking. While this misses many environmental factors, it’s very useful to get unbiased data about a software package or website. And because it can be deployed via a network, thousands of users can be tested at once, and with little cost per each additional user.
Just a few example in this video of the cost impact of not doing proper testing. Wether the cost is purely financial, time or life, the industry mush respond to theses changes by improving the way software is tested.
In addition to the four above, here are three bonus testing systems.
Skinning. Today, changing the user interface is easy. Object Oriented coding standards and a design pattern known as Model-View-Controller, allow the guts of a program to be uncoupled with it’s user interface. So literally, you can give your software a facelift. This lets testers try various version of the same core software. Obviously there is some development cost with this option. But this allows split testing of user interface options or even competing GUI designs – often with surprising results.
Super-Users. There is a forum where someone loves to try every sales automation program on the market. This is his/her hobby. Users like this exists and know every kind of software in whatever industry your application competes in. Not only do these super-users have all the contenders, they’ll try yours for free – and give you great free feedback. And their advice will be couched in terms of how other competing products handle the same use-case.
Embedding. If you are releasing software to run a pizza delivery, go work at a pizza restaurant for a few months. Using your own software in the real job often gives the best insights. The two keys to doing this are: 1) you’ve got to have someone work at the same kind of job the end users are doing. And it should be a programmer or someone on the development team. Someone one how will gain insider insights into living with the software being developed. 2) the tester often needs a few months to really become an end user of the software. And that’s the problem with this. It costs a lot in terms of time. Yet, for expensive software or extensive sites, the insights gained are invaluable, this is also a Lean technic used to bring innovation in your company. I personally highly value this life skill.
There are a lot of testing books out there, some better than others. Most are narrow in the topics they cover and the level of detail they present. In stark contrast, Ammann and Offutt’s book has the advantage of presenting concepts and techniques that cover the broad range of languages and platforms used in practice by industry and academia. Theirs is one of the most thorough and practical testing books ever published I believe.