Categories
Software Testing
a transition to agile
I joined a team at work that has moved to agile from waterfall. It was a high-performing team in waterfall, and now the team is figuring out how to get that same level of performance while doing agile. It’s been a struggle. I wasn’t with the team before, so I’ve only seen them in this transition phase. There’s lots of talk like, “This is how we’ve always done it, and we’ve been just fine,” and “We don’t need to be told how to do our jobs.” People are frustrated, and those of us who are trying to be cheerleaders and evangelists get shouted down frequently. One of the principles that has fallen by the wayside is the idea of face-to-face communication being the best way to get things done. We are theoretically co-located. One guy works out of a different office, but the other 14 of us are in the same room. However, we all work from home on Fridays, and most people take a second day at home too, on different days, and then when the weather is bad, people work from home, and if their kid is sick, they work from home, and if their elbow hurts, they work from home, and on and on. Only for first and last days of the sprint is everyone in the same room. When the idea of video calls has been raised, the response has been swift and negative. Emails, IMs, and the occasional Skype call are said to be sufficient, and the suggestion that more information can be transmitted by being able to see facial cues has been deemed irrelevant. They say, “Our communication has always been good, why do we need to change it?” like I mentioned above. (note: voice deliberately passive in this paragraph to obfuscate my own embarrassment) We’re struggling to figure out how to make our testing the most effective. On the one side, there are a couple who are all about automation, who see it as a point of pride that they haven’t touched a mobile device (which is what we test) for a long time. On the other side, there are people who don’t want anything to do with automation, who don’t want to do any kind of programming. And in the middle, there I am. I think automation is a means to an end. That if you have good enough automation, you can spend more time touching devices and doing session-based exploratory testing, to try to find bugs that would be difficult or impossible to find with automation. I think our coach basically believes this too, but the automation enthusiasts only hear that we need more automation, not that it is meant to facilitate better manual testing. It’s almost a culture war. I guess that one of the effects (benefits?) of agile is that you’re always going to be a little annoyed, a little frustrated, a little driven to (hopefully) improve. It comes from interacting with people so regularly, from needing to rely on working as a team. Differences in approach, in opinion, in work ethic, all come to the forefront, where they can theoretically be discussed. Perhaps one of the next phases of our agile development development (ha, see what I did there?) is to have those open and honest conversations without devolving into raised voices and accusations. That’s a difficult step that requires a lot of maturity from everyone involved, and even then, emotions and egos can still impair the productivity of those conversations.
Oct 2016hand-waving and subject matter expertise
From this recent project I’ve been discussing, I have one more point. Our customers can make transfers into other accounts. The vendor told us that it would “hard fail” if the customer tried to go over their balance. “Hard fail” sounded fancy, and we were assured that it would prevent penalties. However, I was able to show that customers could go over their balance once the fees were included, and it wouldn’t fail. There were actually a number of ways a person could transfer more than they had to give. Turns out “hard fail” wasn’t anything fancy, and it only happened at one point in the process, rather than it being a continuing thing. Business got involved, said the internal people had control over whether to assess the penalties and that they “probably” wouldn’t assess those penalties. Finally, a person who had done that kind of work spoke up to assure us that he had never seen a penalty not assessed. It’s another battle I lost. (I lost a lot of battles on this project, but there were also just lots of battles.) The financial institution assumes that people won’t be that foolish, or perhaps the institution just likes those penalties. I learned a lesson about thoroughly understanding processes. Before the true meaning of “hard fail” came to light, and before a subject matter expert spoke up, it was just hand-waving. We tested it with our assumptions that there were fail-safes involved, when there weren’t really. It took pushing the issue and not giving into the hand-waving that we were able to make an informed decision about the process. The project also taught me not to trust anyone, particularly vendors, when they talk about behaviors. When they say something is “as designed”, they often mean “no design” or “designed without thought”. The product was exhilarating to test, because it was so very broken, but it was also immensely frustrating to be stonewalled by the vendor at most steps.
Sep 2016why testers need to be involved from the beginning
On a recent project, the vendor’s software was going to be embedded in our website through an iframe. After a quick tutorial from my husband (a software architect who had done a lot of research on the security of iframes not too long before that), I thought it would be interesting to poke around and see what I could find in their system and see if I could break into it, though I’m not a security tester. I found some of what I expected to find - they were using security headers that are a little dated but the best that work on Safari (there’s more secure stuff for the other browsers - I don’t know why Apple is behind the times on that). But there was also a header that was misspelled when it should have been automatically generated by the system. This gave me pause. There was a possibility that something got a little funky with the server, but it was also possible that they were hard-coding things on a homegrown server rather than using a commercial server that is regularly patched. I filed a bug. And pushed it. A lot. I got the Information Security team involved. The response finally came back (after more than a month of asking) that they were not using Apache, but they couldn’t tell us more about it, that they were keeping up with security issues and regularly testing their server. This was another issue. The software we were given was so poorly written and tested, that I wasn’t sure if we could trust their word that they were adequately testing their own servers, much less keeping up with OWASP vulnerabilities or doing anything security-related. InfoSec was aware of this. But when I asked about the results from their penetration testing, the project manager told me that they weren’t doing any testing. It was in our contract with this vendor that we wouldn’t do any testing of their server. Seriously? My response to that in a meeting was, “Someone is touching their servers, and I’m concerned that it’s not us.” I lost the battle. I looked through the contract to see if there was any way we could say they breached the contract (hooray, law background!), but the vendor had written it so wishy-washy that it was basically a “best effort” contract without anything binding. So frustrating. In addition to the gaping potential security issues, it was a big problem to me that testers weren’t involved in the decision-making process. I’m not sure if anyone technical was involved, though I really hope so. But a tester could have raised the issue about not testing their servers instead of lawyers and business people just agreeing to it without considering the implications. It should have been a major red flag that they didn’t want us to try to break into their server. Testers are not just a checkmark. We have experience and specific knowledge that can assist with product decisions and can end up protecting the business’s interests.
Aug 2016the benefits of manual testing, episode 2 - agile
Teams at my company are moving to agile, and all of us will go eventually. I’ve been learning about how testing works in agile, and I’ve been talking a lot with people about it, both inside my company and outside. One thing that has come up is the perception that manual testing is not a big piece in agile, and that all of the testers will need to learn automation to continue to succeed. While I agree that testers should know enough to at least identify areas for automation, or better yet, do it, I think there will always be a place in agile for manual testing. Some things just cannot be automated in the first place. Also, manual testing is quick and robust, and the cost can be about the same as automation. Let’s break those down. Quick: Testing in agile needs to be quick. If something is broken, it needs to be discovered and fixed quickly or put the sprint at risk. Manual testing is an efficient way to make sure that something is functioning. Stories can be very small to quite large, and, especially for the smaller stuff and GUI items, being able to quickly look at something, play around with it, and see whether it is working the way it’s supposed to is best done with manual testing. This is especially true for a smaller piece of a work in progress, when not everything is connected, but the developers want to make sure they’re on the right track. Robust: Manual testing is robust. Deviations from a script are expected. If a URL or menu changes, or if a button moves around, a manual test can go forward. An automated test stops and throws an error and has to be inspected for what went wrong. If an application is very stable and unlikely to change anything, automation may be helpful, but if features are being added or moved around, as happens frequently in agile, manual testing will be able to continue more easily than an automated test. Cost: Automation is a large up-front expense. What takes five minutes of manual testing may take thirty minutes to code an automated test. If the same test is going to be run over and over again over multiple sprint, automation may be more cost-effective. But manual testing is more cost-effective to determine functionality for initial tests and one-off tests. Automation does have its uses. Automation takes some human error out of testing (though possibly introduces human error in the coding for testing itself). If testers want to be involved with writing unit tests or helping with TDD, automation is a good way to go. If it’s a long-term project with core functionality built at the beginning, regression tests can be automated to ensure that future features don’t break the main stuff. Integration testing of earlier features can be built too. But, especially for the first sprint of a feature, before it’s fully realized and integrated, manual testing is the way to go. I know of some companies who do entirely manual testing in agile, which poses its own set of challenges. For a discussion of how agile changes the tester’s role, and for briefly touching on different benefits of manual vs automated testing in agile, see Ulf Eriksson’s blog post here . From the same site, here is a mind map about testing in agile, and section 6 deals with manual testing and why it’s necessary. I ran this past people who know what they’re talking about to make sure I’m not saying anything heretical, but I understand there may be differences of opinion. I’d welcome a discussion about this!
Jul 2016the benefits of manual testing, episode 1
I attended PyCon a few weeks ago, and it was a wonderful experience. I met lots of interesting people, heard great talks, and got inspired to get back to programming. One thing I encountered from multiple people is a lack of understanding of what testers do and why it’s necessary, particularly manual testing. Once I explained what I do and described the things I’ve found, I received a few job offers actually. I think it reflected more of an unhappiness with the rote checking that a lot of testers do instead of the targeted, intelligent testing I was describing. I was honest with people about my novice programming skills (solving math problems kind of counts, but I have more ambitious projects coming up). People immediately assumed that the only valid kind of testing was automated testing, though they tried to understand what I do. This post contains thoughts that came out of those conversations. I anticipate that this topic will be merit a few posts, hence “episode 1”. First, automated testing is almost always necessary for performance testing and stress testing, though sometimes the stress testing can be mimicked through the use of other tools. What manual testing offers is a more detailed look at how the software behaves at every step, because a tester can visually inspect pages, figure out alternate paths to what should be the same outcome, and test the foolish things and the unexpected things. I might be mistaken that not all of these things can be automated, but automating testing can be expensive and time-consuming to get off the ground and maintained, and it’s not a silver bullet. Manual testers function best when we can use creativity. “Checking”, the act of merely inspecting documents to make sure that something in program A matches something in program B, is a waste of talent and money. Manual testers use experience and “hunches” really to figure out where the weaknesses are, and they try to exploit weaknesses in multiple ways. Once that weakness is found and fully understood, a test can be written for automation that would make sure it doesn’t break in the future. Manual testers can cover the bulk of the testing by just exploring the software and getting creative with how they approach different scenarios. As a few examples, I’ve been working on a project that uses iframes. I knew a little bit about the security in iframes and how it used to be exploited (and how it can still be exploited in Safari, which is another matter). I thought I might be able to do something with that, so I went through some of the http headers from the iframe to see what was there. It wasn’t quite as bad as I had hoped, but I did find some suspicious tags that made me question the vulnerability of the server generally. The vendor promises they aren’t using a home grown server, but our information security team is preparing to go to town to try to break into the software. With the same software, the math wasn’t adding up. I tried a bunch of different combinations of things to make sure that people were consistently able to do things a financial institution would not want them to do. Basically, every thing that we customized or that the vendor bragged about had a high probability of being broken initially. I had a bit of a high from the first few weeks of testing. This is our first experience with this software, so until we know the weaknesses fully, manual testing is most appropriate. Manual testing can have more coverage in some instances, and it can be more effective in finding specific bugs that may not expose themselves until the tester plays a fantastic fool.
May 2016job and training
It’s Testing Tuesday! Let’s talk software testing! To start, software testing is finding weaknesses in software that, when fixed, make it a better product. Software testers don’t break software, they expose how it’s already broken. It is rather fun to say I break things for my career, but some people, particularly developers, respond poorly to that. Software testing is a productive practice in that it improves the end product, though it can feel a little destructive when the tester finds bug after bug. I’ve been at my new job since the end of March, ostensibly testing software, but mostly learning things. The first project I worked on was a small upgrade to existing software, and I didn’t have much to test, though I had plenty of time to test it. That was good, because I’m learning the formalities of testing and how this specific company does it, and I appreciated having time to get my bearings. On this project, the vendor supplied recommended test cases, and I was left to my own devices to create a suite of regression test cases. The people who had done the prior upgrades hadn’t left any test cases to run for regression, and the documentation of the tests they did run was virtually non-existent. The learning curve wasn’t too bad, though I did have to ask a lot of questions about some things, and the institutional knowledge of the product wasn’t great. A couple times, I got the response that something was working as designed when it was really a bug acting consistently across a subset of items, and when I asked why something was supposed to behave like that, I was told that the person didn’t know. This was frustrating, but the project could have been much more frustrating with less congenial people. I had a fairly high bug find rate, particularly in light of the number of test cases I ran and the amount of time I spent in ad hoc testing (which for me meant learning my way around and trying random and non-targeted things). As a novice tester, this has me very concerned that the software is really buggy (as opposed to me being very lucky or very good). Sometimes I wonder if the reassurances from my colleagues and managers are just false accolades, but that’s my own insecurity, not the topic of this post. All told, I really like the company I’m working for, I love the cooperative and collaborative environment, and I find the work to be fun and sometimes challenging. Plus, no one is going to sue me because of my work, so that’s a bonus. I was given training through SQE to prepare for the ISTQB exam. The certification is as a foundation level tester, and the exam is 40 questions long, with a passing grade of 65%. The training was fine. The teacher was engaging most of the time, and you could tell he had a real passion for testing. I didn’t find it at all useful in helping me to do my job though. It was about theory and vocabulary and forms. The only time we came to concrete techniques was a discussion about partitions and boundary analysis. The training did help fill in some gaps in the self-study I had been doing, but I think the usefulness of the training comes in giving the team common vernacular to use. I have kind of a big problem with 65% being a passing grade though. How should that reassure anyone that the person knows what they’re talking about? Another problem with the certification, aside from its low passing grade, it that it’s a one-time certification with no renewals necessary. There’s no requirement for continuing education, no need for production of work product to show competence. It seems to me to be a meaningless badge of legitimacy that isn’t needed once you have a real job behind you. I think a more valuable thing for a resume would be an online portfolio with a test plan and test cases. But I say all this as someone with a job now, and had I not been given a chance, I was planning on getting the certification on my own to show that I at least know something. I’ve been looking at a lot of resources to help make me a better tester quickly. These have included books, blogs, online resources, and streamed conference presentations. Of the resources I’ve consulted, one of my favorites is James Whittaker’s How to Break Software. Some of it isn’t applicable to what I do, but he gives real-world examples of how things can break. He talks about different kinds of tests to run through human interaction and manipulating file interaction as well. I just started reading Cem Kaner’s (et al.) Testing Computer Software. Just the first couple chapters are really useful so far. I’ve really enjoyed James Bach’s blog and Michael Bolton’s blog as well. They are both big into rapid software testing and rethinking the way exploratory testing is (and rechristening it simply “testing”). Their blogs are full of insights and good ideas for people who want to improve the way they think about software testing. As I finish or discover other resources, I’ll discuss them here. Until another Tuesday!