Tags
Automation
automation thoughts - an update
Last year, I wrote some thoughts about the benefits of manual testing. The posts are here and here . Now that I’ve been on an agile team for over a year, and doing some automation, I think (hope) I have some more sophisticated thoughts about automation. Our team has two “manual testers” and one “automation engineer”. The automation engineer does all of the UI automation (for now), but I’ve written a lot of service layer tests using SOAP UI, and the other tester has been learning to write them too. The idea is to have the two of us start running (and eventually writing) UI automated tests as well, and we’re taking steps towards that end. The benefits of the API testing through SOAP have become clear, but so have the challenges. It has cut down on our need to test all flows in regression testing manually. I can write service tests while the UI is still being coded, and it’s a good way to start our monthly testing cycle. After all, if something is broken at the service level, there’s no point in testing manually, and we can pinpoint the error more quickly and precisely. We have everything in our app that can be tested at the service level automated, which I feel really good about. Or, at least, we have all the calls exercised. There’s always more to test, isn’t there? A big weakness of service testing is that it is more brittle than manual testing, because if something changes in the call (for instance, a value changes from being taken from a detail to being taken from a cookie), it breaks the test, though you wouldn’t notice a difference at the UI level. You also can’t run all the negative tests (like lockout tests) you might want to run, particularly if you think you’ll run the suite repeatedly through the day. We have two test cases that can only be run once every half hour, and one test case that requires a person to reset the user every time. Those test cases are fairly essential, so we make do with them. I still very strongly believe that there is no substitute for getting in the features and playing around, or, to be more formal about it, doing exploratory testing. That’s how we find all the interesting bugs. Automated testing is checking; it ensures certain things are stable, not that a product is “good”. However, automated testing is (can be?) fast, and it can be an efficient use of time, particularly for regression testing. Automated testing can make sure that bugs that are found and fixed don’t break again, but would a person writing automated test think to try refreshing a page without first finding an issue in exploratory testing? Perhaps I’ll have more thoughts about this later, but this is what I’ve been thinking about recently. Thoughts from y’all? Criticism? Questions?
Jul 2017SOAP testing and finding the right assertions
I’ve been delving into the more technical side of testing, that of using tools to exercise code precisely rather than galumphing through UI, which really is my preferred method. Our mobile app uses a lightweight service to connect to the other moving pieces at the company and our vendors, so we can test those calls directly through SOAP, which, for you non-software people out there, is also called API testing. The developers of our software took kind of a blanket approach to the envelopes for the service calls, so there are tons of superfluous fields for each of the calls, and it’s kind of a guessing game as to which are required for any particular service call. My developers didn’t create all the service calls, so I’ve actually been able to figure things out and share my envelopes with them (so, you know, I feel pretty cool). Once I figure out which fields are necessary to get a call working, then it’s time to figure out an assertion. These are what the testing software checks against to make sure things are working appropriately. In our testing, they’re usually related to the content of a field, whether it is a “success” or “failure” message, or, in a case I was working on recently, whether the name of a document started with a specific year or not. With this case, we modified the call to pull documents one year at a time, so the call for each year needed to return only documents for that year and none others. I ended up doing a check that the count of docs that started with something other than the year I was looking for was 0. This may sound simple, but it took quite awhile to get the xpath right, and I had some (a lot of?) help from a developer. The SOAP tests are nothing without the correct assertions. You can go ahead and get all the information you want, but without an automated check on the information, the tests have very little advantage over UI testing in terms of speed, though they do have the ability to cut down on some of the noise that is inevitable with UI testing. API testing is another tool in the box. It cannot be the only thing, and it should not be ignored. It’s great for getting precise information, for making sure you exercise calls at a lower level, and it can be faster than UI testing, but it is not a substitute for a full complement of testing, including exploratory testing (galumphing), scripted tests run manually, and scripted tests run through automation. And, of course, if your developers unit test their code, that helps too. :)