racheljoi.com

Teams at my com­pa­ny are mov­ing to agile, and all of us will go even­tu­al­ly. I’ve been learn­ing about how test­ing works in agile, and I’ve been talk­ing a lot with peo­ple about it, both inside my com­pa­ny and out­side. One thing that has come up is the per­cep­tion that man­u­al test­ing is not a big piece in agile, and that all of the testers will need to learn automa­tion to con­tin­ue to suc­ceed. While I agree that testers should know enough to at least iden­ti­fy areas for automa­tion, or bet­ter yet, do it, I think there will always be a place in agile for man­u­al test­ing. Some things just can­not be auto­mat­ed in the first place. Also, man­u­al test­ing is quick and robust, and the cost can be about the same as automa­tion. Let’s break those down.

Quick: Test­ing in agile needs to be quick. If some­thing is bro­ken, it needs to be dis­cov­ered and fixed quick­ly or put the sprint at risk. Man­u­al test­ing is an effi­cient way to make sure that some­thing is func­tion­ing. Sto­ries can be very small to quite large, and, espe­cial­ly for the small­er stuff and GUI items, being able to quick­ly look at some­thing, play around with it, and see whether it is work­ing the way it’s sup­posed to is best done with man­u­al test­ing. This is espe­cial­ly true for a small­er piece of a work in progress, when not every­thing is con­nect­ed, but the devel­op­ers want to make sure they’re on the right track.

Robust: Man­u­al test­ing is robust. Devi­a­tions from a script are expect­ed. If a URL or menu changes, or if a but­ton moves around, a man­u­al test can go for­ward. An auto­mat­ed test stops and throws an error and has to be inspect­ed for what went wrong. If an appli­ca­tion is very sta­ble and unlike­ly to change any­thing, automa­tion may be help­ful, but if fea­tures are being added or moved around, as hap­pens fre­quent­ly in agile, man­u­al test­ing will be able to con­tin­ue more eas­i­ly than an auto­mat­ed test.

Cost: Automa­tion is a large up-front expense. What takes five min­utes of man­u­al test­ing may take thir­ty min­utes to code an auto­mat­ed test. If the same test is going to be run over and over again over mul­ti­ple sprint, automa­tion may be more cost-effec­tive. But man­u­al test­ing is more cost-effec­tive to deter­mine func­tion­al­i­ty for ini­tial tests and one-off tests.

Automa­tion does have its uses. Automa­tion takes some human error out of test­ing (though pos­si­bly intro­duces human error in the cod­ing for test­ing itself). If testers want to be involved with writ­ing unit tests or help­ing with TDD, automa­tion is a good way to go. If it’s a long-term project with core func­tion­al­i­ty built at the begin­ning, regres­sion tests can be auto­mat­ed to ensure that future fea­tures don’t break the main stuff. Inte­gra­tion test­ing of ear­li­er fea­tures can be built too. But, espe­cial­ly for the first sprint of a fea­ture, before it’s ful­ly real­ized and inte­grat­ed, man­u­al test­ing is the way to go. I know of some com­pa­nies who do entire­ly man­u­al test­ing in agile, which pos­es its own set of chal­lenges.

For a dis­cus­sion of how agile changes the tester’s role, and for briefly touch­ing on dif­fer­ent ben­e­fits of man­u­al vs auto­mat­ed test­ing in agile, see Ulf Eriksson’s blog post here. From the same site, here is a mind map about test­ing in agile, and sec­tion 6 deals with man­u­al test­ing and why it’s nec­es­sary.

I ran this past peo­ple who know what they’re talk­ing about to make sure I’m not say­ing any­thing hereti­cal, but I under­stand there may be dif­fer­ences of opin­ion. I’d wel­come a dis­cus­sion about this!