Tags
Software Testing
training a tester
I may have mentioned a few months ago that I’m training a tester who is new to mobile. We’re four months in, and I’m surprised to find that I’m still answering (many) questions daily. Language does play an important role, and sometimes I realize that concepts that are organized in a certain way in my head don’t translate well to words or the way other people think. But also, for a long time, I was just answering questions, imparting information. I’ve changed tactics, now that she’s been on our team for several months, to being more socratic. If she thinks she’s found a bug, I press her, ask her why she thinks it’s a bug, and how she can get additional information about the bug. Eventually, I want to know what her oracles are, what devices/OSes/app versions she’s tried on, what environments she’s been in, what the logs that we have access to say, and on and on. I know I don’t always follow all the steps myself, but I have a checklist published for our team that talks about all the different things to try. If she asks whether a feature is supposed to be in a specific app version, I push her to explain why it should or should not be before I give her an answer. Just this last week, I included her in writing SOAP tests, asking her questions about how we could modify certain things to get the right thing tested, instead of talking through my own thought process. I think this method is more effective. Instead of trying to describe how my own synapses fire, I’m making her form her own way of thinking about things. It’s frustrating sometimes, because it takes a lot more effort to work with someone for fifteen minutes so they come to their own answer instead of just providing it, but the goal is for it to save time in the future. In the couple months where I was the only tester on the team, I revamped how testing was documented, and how things in general were documented, to a way that made the most sense to me. I documented what I did rather than what I was going to do. My notes for things to test for were for my reference, not for anyone else to consume, and I made mind maps as artifacts. This worked great for me, and I think it’s working well for my partner, but I’m trying to be sensitive to the idea that not everyone thinks like me (nor should they), and to be open to doing things another way, should she come up with something better.
Jul 2017SOAP testing and finding the right assertions
I’ve been delving into the more technical side of testing, that of using tools to exercise code precisely rather than galumphing through UI, which really is my preferred method. Our mobile app uses a lightweight service to connect to the other moving pieces at the company and our vendors, so we can test those calls directly through SOAP, which, for you non-software people out there, is also called API testing. The developers of our software took kind of a blanket approach to the envelopes for the service calls, so there are tons of superfluous fields for each of the calls, and it’s kind of a guessing game as to which are required for any particular service call. My developers didn’t create all the service calls, so I’ve actually been able to figure things out and share my envelopes with them (so, you know, I feel pretty cool). Once I figure out which fields are necessary to get a call working, then it’s time to figure out an assertion. These are what the testing software checks against to make sure things are working appropriately. In our testing, they’re usually related to the content of a field, whether it is a “success” or “failure” message, or, in a case I was working on recently, whether the name of a document started with a specific year or not. With this case, we modified the call to pull documents one year at a time, so the call for each year needed to return only documents for that year and none others. I ended up doing a check that the count of docs that started with something other than the year I was looking for was 0. This may sound simple, but it took quite awhile to get the xpath right, and I had some (a lot of?) help from a developer. The SOAP tests are nothing without the correct assertions. You can go ahead and get all the information you want, but without an automated check on the information, the tests have very little advantage over UI testing in terms of speed, though they do have the ability to cut down on some of the noise that is inevitable with UI testing. API testing is another tool in the box. It cannot be the only thing, and it should not be ignored. It’s great for getting precise information, for making sure you exercise calls at a lower level, and it can be faster than UI testing, but it is not a substitute for a full complement of testing, including exploratory testing (galumphing), scripted tests run manually, and scripted tests run through automation. And, of course, if your developers unit test their code, that helps too. :)
Jul 2017language and learning
We have a new tester on my team. I recruited her, and I talked her up to my team. They liked her, and I was excited when it worked out. She’s not new to testing, but she’s new to mobile, and it’s a different beast and a different set of systems from what she was doing before. We’re now four weeks in. I’m exhausted. I think the fault is mine. I expected to train someone like me, and not to talk a big game, but I didn’t get tons of help when I started, and I made it through okay. But I also expected to be able to use my own language, with my own mental images and way of explaining things. That hasn’t worked so well. I’m being asked to show my work, like in math or law, for everything, and it takes so much effort to put things in terms that make sense to her rather than just to me. She’s a very visual learner, and though I make diagrams in my head of things, they don’t necessarily translate well. It’s gotten me thinking about how language is used to convey such complex ideas without many words. It’s like the Star Trek TNG episode Darmok, right? This society speaks entirely in metaphors, communicating deep ideas to each other but making it difficult for outsiders to come in. Carl and I have our shorthand, where one of us will mention a few words about a memory or an emotion and the other will instantly understand the meaning behind those words. We were at a conference once, and we ate with a couple who had been married for 50+ years and who were both deaf. Their translator explained that she didn’t know some of their signs, because deaf couples will make up signs that apply just to them. I found that fascinating, but it’s really no different from what we do with spoken word in our close relationships. I fear I’m just becoming lazy with language, and assuming that, if people don’t understand me, it’s on them and not me. I know that’s the wrong attitude, and I’m working on it. But it’s just… exhausting.
Jun 2017indispensability and value
I had a realization when I started my current job: I don’t want to be indispensable. For a very long time, I wanted to be indispensable, where things would come to a halt if I weren’t there or if I hadn’t left detailed instructions behind. I wanted to be so important to a company or a project that work absolutely required my expertise and my presence. I thought that was a sign of value, that it meant I mattered. But now, I don’t want to be indispensable. It seems to be a sign of arrogance to the point of irresponsibility and disregard for the well-being of an organization to be indispensable. I want to be valued and to train others to do my job, so that I have the freedom to take a vacation, or even (gasp) leave, somewhere down the road. I care about the team I’m on and want them to succeed, with or without me. What I want is to be valued, to be respected for the skills, knowledge, and ideas I bring, and to be regarded as a positive force on our team. However, we have a new tester on our team, and I’m training her. Though I know I really want her to succeed and be a partner with me, I still feel a little threatened, like… I’m no longer required. Even though I’ve been thinking these ideas, about the tension between indispensability and value, and coming to regard them as separate concepts, I still feel like they are the same thing, like the best way to be valued is to make myself indispensable. I actually talked with the new woman about this, so that she understands that if I start to sound a bit territorial or a bit fussy, it’s not about her, but about me, and she’s welcome to confront me about it. We’ll see how I react if she does confront me. :)
Feb 2017testing trainings - a comparison
In the past month, I’ve done two trainings on software testing: ASTQB’s Mobile Foundations course, and Satisfice’s Rapid Software Testing Applied course. The difference between them was marked. Like the SQE training I reviewed earlier and subsequent ISTQB test for the Certified Tester, Foundation Level, the mobile course was heavy on vocabulary and “best practices” and light on how to do a good job. It gave me things to think about, such as using simulators and emulators to increase coverage and getting cell data on some of our phones so our testing can be more real-life and more, well, mobile. But when it came down to it, a lot of the class was about the differences between web-based, native, and hybrid apps, and the risks involved in testing them. Looking at the risks that are inherent to the different types of apps was useful, but three days of vocabulary became a little wearisome. The test, which I took about a week and a half later, went just fine. It included a decision table, which took me by surprise, but the test wasn’t a big deal with a little bit of careful reading. I don’t have much more to say about the training or the test. I wasn’t planning on doing either, but then a spot was offered to me, so… it was fine. The training that I was really excited about, and that totally lived up to my expectations, was James Bach’s Rapid Software Testing Applied. We tested a vector graphics program called Inkscape, approaching it from some different angles. Each day was a combination of lecture, individual/team work, and review of that work. Some guys from another Utah company invited me to join their team, so I talked with them throughout the day and worked with them on the assignments. We talked about sanity testing, survey testing, risk analysis, coverage, deep testing, testing with tools, and how to report testing. It was a fascinating class, though I did receive criticism, both privately and then publicly the next morning, for some of my bug reports. (I still need to check with my developers to see if they’re annoyed by my reporting.) My ego was a little bruised, but I know he was trying to make me a better tester, and in the end, I appreciated (that might be too strong of a word) the criticism. This training brought out all my insecurities, particularly those surrounding tools, but I was also pleased to have some of my thoughts about testing affirmed. James Bach is something of an icon in software testing, and I really enjoyed learning from him. I’d like to take another RSTA class, as well as his lecture class of Rapid Software Testing. It’s possible I was just way more excited for RSTA, but I felt like I got more out of it as well. I really appreciated my company letting me do these trainings, particularly as they came so close together. I’m hoping to convince them to bring James Bach to Utah - that would just be fantastic.
Jan 2017a transition to agile
I joined a team at work that has moved to agile from waterfall. It was a high-performing team in waterfall, and now the team is figuring out how to get that same level of performance while doing agile. It’s been a struggle. I wasn’t with the team before, so I’ve only seen them in this transition phase. There’s lots of talk like, “This is how we’ve always done it, and we’ve been just fine,” and “We don’t need to be told how to do our jobs.” People are frustrated, and those of us who are trying to be cheerleaders and evangelists get shouted down frequently. One of the principles that has fallen by the wayside is the idea of face-to-face communication being the best way to get things done. We are theoretically co-located. One guy works out of a different office, but the other 14 of us are in the same room. However, we all work from home on Fridays, and most people take a second day at home too, on different days, and then when the weather is bad, people work from home, and if their kid is sick, they work from home, and if their elbow hurts, they work from home, and on and on. Only for first and last days of the sprint is everyone in the same room. When the idea of video calls has been raised, the response has been swift and negative. Emails, IMs, and the occasional Skype call are said to be sufficient, and the suggestion that more information can be transmitted by being able to see facial cues has been deemed irrelevant. They say, “Our communication has always been good, why do we need to change it?” like I mentioned above. (note: voice deliberately passive in this paragraph to obfuscate my own embarrassment) We’re struggling to figure out how to make our testing the most effective. On the one side, there are a couple who are all about automation, who see it as a point of pride that they haven’t touched a mobile device (which is what we test) for a long time. On the other side, there are people who don’t want anything to do with automation, who don’t want to do any kind of programming. And in the middle, there I am. I think automation is a means to an end. That if you have good enough automation, you can spend more time touching devices and doing session-based exploratory testing, to try to find bugs that would be difficult or impossible to find with automation. I think our coach basically believes this too, but the automation enthusiasts only hear that we need more automation, not that it is meant to facilitate better manual testing. It’s almost a culture war. I guess that one of the effects (benefits?) of agile is that you’re always going to be a little annoyed, a little frustrated, a little driven to (hopefully) improve. It comes from interacting with people so regularly, from needing to rely on working as a team. Differences in approach, in opinion, in work ethic, all come to the forefront, where they can theoretically be discussed. Perhaps one of the next phases of our agile development development (ha, see what I did there?) is to have those open and honest conversations without devolving into raised voices and accusations. That’s a difficult step that requires a lot of maturity from everyone involved, and even then, emotions and egos can still impair the productivity of those conversations.
Oct 2016hand-waving and subject matter expertise
From this recent project I’ve been discussing, I have one more point. Our customers can make transfers into other accounts. The vendor told us that it would “hard fail” if the customer tried to go over their balance. “Hard fail” sounded fancy, and we were assured that it would prevent penalties. However, I was able to show that customers could go over their balance once the fees were included, and it wouldn’t fail. There were actually a number of ways a person could transfer more than they had to give. Turns out “hard fail” wasn’t anything fancy, and it only happened at one point in the process, rather than it being a continuing thing. Business got involved, said the internal people had control over whether to assess the penalties and that they “probably” wouldn’t assess those penalties. Finally, a person who had done that kind of work spoke up to assure us that he had never seen a penalty not assessed. It’s another battle I lost. (I lost a lot of battles on this project, but there were also just lots of battles.) The financial institution assumes that people won’t be that foolish, or perhaps the institution just likes those penalties. I learned a lesson about thoroughly understanding processes. Before the true meaning of “hard fail” came to light, and before a subject matter expert spoke up, it was just hand-waving. We tested it with our assumptions that there were fail-safes involved, when there weren’t really. It took pushing the issue and not giving into the hand-waving that we were able to make an informed decision about the process. The project also taught me not to trust anyone, particularly vendors, when they talk about behaviors. When they say something is “as designed”, they often mean “no design” or “designed without thought”. The product was exhilarating to test, because it was so very broken, but it was also immensely frustrating to be stonewalled by the vendor at most steps.
Sep 2016why testers need to be involved from the beginning
On a recent project, the vendor’s software was going to be embedded in our website through an iframe. After a quick tutorial from my husband (a software architect who had done a lot of research on the security of iframes not too long before that), I thought it would be interesting to poke around and see what I could find in their system and see if I could break into it, though I’m not a security tester. I found some of what I expected to find - they were using security headers that are a little dated but the best that work on Safari (there’s more secure stuff for the other browsers - I don’t know why Apple is behind the times on that). But there was also a header that was misspelled when it should have been automatically generated by the system. This gave me pause. There was a possibility that something got a little funky with the server, but it was also possible that they were hard-coding things on a homegrown server rather than using a commercial server that is regularly patched. I filed a bug. And pushed it. A lot. I got the Information Security team involved. The response finally came back (after more than a month of asking) that they were not using Apache, but they couldn’t tell us more about it, that they were keeping up with security issues and regularly testing their server. This was another issue. The software we were given was so poorly written and tested, that I wasn’t sure if we could trust their word that they were adequately testing their own servers, much less keeping up with OWASP vulnerabilities or doing anything security-related. InfoSec was aware of this. But when I asked about the results from their penetration testing, the project manager told me that they weren’t doing any testing. It was in our contract with this vendor that we wouldn’t do any testing of their server. Seriously? My response to that in a meeting was, “Someone is touching their servers, and I’m concerned that it’s not us.” I lost the battle. I looked through the contract to see if there was any way we could say they breached the contract (hooray, law background!), but the vendor had written it so wishy-washy that it was basically a “best effort” contract without anything binding. So frustrating. In addition to the gaping potential security issues, it was a big problem to me that testers weren’t involved in the decision-making process. I’m not sure if anyone technical was involved, though I really hope so. But a tester could have raised the issue about not testing their servers instead of lawyers and business people just agreeing to it without considering the implications. It should have been a major red flag that they didn’t want us to try to break into their server. Testers are not just a checkmark. We have experience and specific knowledge that can assist with product decisions and can end up protecting the business’s interests.
Aug 2016the benefits of manual testing, episode 2 - agile
Teams at my company are moving to agile, and all of us will go eventually. I’ve been learning about how testing works in agile, and I’ve been talking a lot with people about it, both inside my company and outside. One thing that has come up is the perception that manual testing is not a big piece in agile, and that all of the testers will need to learn automation to continue to succeed. While I agree that testers should know enough to at least identify areas for automation, or better yet, do it, I think there will always be a place in agile for manual testing. Some things just cannot be automated in the first place. Also, manual testing is quick and robust, and the cost can be about the same as automation. Let’s break those down. Quick: Testing in agile needs to be quick. If something is broken, it needs to be discovered and fixed quickly or put the sprint at risk. Manual testing is an efficient way to make sure that something is functioning. Stories can be very small to quite large, and, especially for the smaller stuff and GUI items, being able to quickly look at something, play around with it, and see whether it is working the way it’s supposed to is best done with manual testing. This is especially true for a smaller piece of a work in progress, when not everything is connected, but the developers want to make sure they’re on the right track. Robust: Manual testing is robust. Deviations from a script are expected. If a URL or menu changes, or if a button moves around, a manual test can go forward. An automated test stops and throws an error and has to be inspected for what went wrong. If an application is very stable and unlikely to change anything, automation may be helpful, but if features are being added or moved around, as happens frequently in agile, manual testing will be able to continue more easily than an automated test. Cost: Automation is a large up-front expense. What takes five minutes of manual testing may take thirty minutes to code an automated test. If the same test is going to be run over and over again over multiple sprint, automation may be more cost-effective. But manual testing is more cost-effective to determine functionality for initial tests and one-off tests. Automation does have its uses. Automation takes some human error out of testing (though possibly introduces human error in the coding for testing itself). If testers want to be involved with writing unit tests or helping with TDD, automation is a good way to go. If it’s a long-term project with core functionality built at the beginning, regression tests can be automated to ensure that future features don’t break the main stuff. Integration testing of earlier features can be built too. But, especially for the first sprint of a feature, before it’s fully realized and integrated, manual testing is the way to go. I know of some companies who do entirely manual testing in agile, which poses its own set of challenges. For a discussion of how agile changes the tester’s role, and for briefly touching on different benefits of manual vs automated testing in agile, see Ulf Eriksson’s blog post here . From the same site, here is a mind map about testing in agile, and section 6 deals with manual testing and why it’s necessary. I ran this past people who know what they’re talking about to make sure I’m not saying anything heretical, but I understand there may be differences of opinion. I’d welcome a discussion about this!
Jul 2016the benefits of manual testing, episode 1
I attended PyCon a few weeks ago, and it was a wonderful experience. I met lots of interesting people, heard great talks, and got inspired to get back to programming. One thing I encountered from multiple people is a lack of understanding of what testers do and why it’s necessary, particularly manual testing. Once I explained what I do and described the things I’ve found, I received a few job offers actually. I think it reflected more of an unhappiness with the rote checking that a lot of testers do instead of the targeted, intelligent testing I was describing. I was honest with people about my novice programming skills (solving math problems kind of counts, but I have more ambitious projects coming up). People immediately assumed that the only valid kind of testing was automated testing, though they tried to understand what I do. This post contains thoughts that came out of those conversations. I anticipate that this topic will be merit a few posts, hence “episode 1”. First, automated testing is almost always necessary for performance testing and stress testing, though sometimes the stress testing can be mimicked through the use of other tools. What manual testing offers is a more detailed look at how the software behaves at every step, because a tester can visually inspect pages, figure out alternate paths to what should be the same outcome, and test the foolish things and the unexpected things. I might be mistaken that not all of these things can be automated, but automating testing can be expensive and time-consuming to get off the ground and maintained, and it’s not a silver bullet. Manual testers function best when we can use creativity. “Checking”, the act of merely inspecting documents to make sure that something in program A matches something in program B, is a waste of talent and money. Manual testers use experience and “hunches” really to figure out where the weaknesses are, and they try to exploit weaknesses in multiple ways. Once that weakness is found and fully understood, a test can be written for automation that would make sure it doesn’t break in the future. Manual testers can cover the bulk of the testing by just exploring the software and getting creative with how they approach different scenarios. As a few examples, I’ve been working on a project that uses iframes. I knew a little bit about the security in iframes and how it used to be exploited (and how it can still be exploited in Safari, which is another matter). I thought I might be able to do something with that, so I went through some of the http headers from the iframe to see what was there. It wasn’t quite as bad as I had hoped, but I did find some suspicious tags that made me question the vulnerability of the server generally. The vendor promises they aren’t using a home grown server, but our information security team is preparing to go to town to try to break into the software. With the same software, the math wasn’t adding up. I tried a bunch of different combinations of things to make sure that people were consistently able to do things a financial institution would not want them to do. Basically, every thing that we customized or that the vendor bragged about had a high probability of being broken initially. I had a bit of a high from the first few weeks of testing. This is our first experience with this software, so until we know the weaknesses fully, manual testing is most appropriate. Manual testing can have more coverage in some instances, and it can be more effective in finding specific bugs that may not expose themselves until the tester plays a fantastic fool.