Software Tasting --by Geordie Keitt » rapid testing

Testing In Principle: Root

Geordie Keitt — Tue, 25 Nov 2008 03:58:23 +0000

T’ai chi hurts to do. The benefits are enormous, and the cost is pain. The nice part is, the same deal applies to most things in life. If you want to be a great guitar player, your fingers will hurt. If you want to be a great athlete, your muscles will hurt. My teacher Chris Casey says, “There is no way out. But there is a way through.”

What hurts is letting go of the tension in my body and feeling that tension come into consciousness. When I work this while standing and imagining that I am connected with the earth through the soles of my feet, the pain is incredible in my feet and legs, the burn of chi blowing through parts of my body that I’ve denied it access to for many years. The gains of course are commensurate with the effort I put in. The key win while rooting is that I am connected to something much larger than myself, a thing to which everything and everyone else is connected as well: the earth. When I am participating consciously in that connection I have insight into the connections others have with the earth as well, and can analyze and manipulate it to my benefit in tai chi, and to their benefit too as a friend. The pain comes from losing the shell of ego I have built that separates me from my surroundings.

The testing parallel is obvious from my other blog entries: the bare facts of your context are the ground you stand on when choosing what to test and why. A great Rapid Tester does not arrive in a context already thinking she knows all she needs to know, fully formed, individuated and full of opinions about what is best. She learns. She crawls, then walks, then runs. Asks questions, appears clueless, and opens herself to learning about everything from everything. She molds herself into a teammate in order to be useful, takes on all the different viewpoints of the stakeholders, and also learns to “think like the code” (model the system) as well.

This is a painful process for those of us with egos that tell us to look like we have the answers. It is hard work to catalog my biases and weaknesses, and to systematically counteract them. One of the best methods for analyzing my tendencies is to participate in friendly competitions and debates with fellow testers. I learn that all of my favorite tricks have pitfalls, and there are a lot of great ways to test and manage testing that I have not yet thought of. I have become more comfortable with that fact. And I have become comfortable in my community, a vibrant and living aspect of my context that I am very happy to be able to tap when necessary.

One more point about Root. The tai chi player with more root always wins. That is because they can extend their root beneath their opponent and cut them off from the earth. A good player can literally bounce me like a basketball when they sink below me, test my connection to the earth, and find the flaws in it. We have the same job as testers, to probe the structure of our systems and the strength and depth of their logic and implementation, and to work on the flaws we see or suspect, until we break the illusion that the system “just works” and can bounce it around like a basketball. When I do good Rapid Testing, I am sinking deeply into the problem space, asking the hardest questions, getting at the heart of the matter, and feeling the burn.

Testing In Principle: Work Directly Toward the Center

Geordie Keitt — Sat, 18 Oct 2008 01:19:36 +0000

When I worked with James Bach on a Satisfice project he would ask me uncomfortable questions about what I was doing and why I was doing it. He’d ask, “Tell me what this bug is at its core.” I’d say, “When I click this button, I get an error message instead of the dialog I’m expecting.”

“Why do you expect a dialog?”

“Because that’s how the app is supposed to work.”

“How do you know?”

“It worked that way another time.”

“Ahh, so there is an inconsistency between how it worked before and the way it works now. What is different? How do you know that they are not both right?”

What he needed me to get at was the clearest possible understanding of the issue, and the clearest and simplest expression of it. He imposed a 75-character limit on bug report summaries.

A typical progression of a bug report might go like this:

Error message appears after user selects calendar icon on application page

Calendar sometimes fails to load on application page

Calendar sometimes fails to load on any page

Calendar sometimes fails to load on low-RAM system (256MB)

All Java calendars fail to load when less than 50MB RAM available

I have found that I know when I have hit the core of a bug (or at least close enough). My mind comes to rest. Robert Pirsig describes the feeling very nicely in “Zen and the Art of Motorcycle Maintenance”: “the material and the craftsman’s thoughts change together in a progression of smooth, even changes until his mind is at rest at the exact instant the material is right.”

In t’ai chi we focus on the center of everything. The center of your weight, the tan tien, is the source of all your power. If you can control your opponent’s center then you can control his balance in push hands and you will win. When you shift weight you will shift it to the center of your foot and the center of your knee will travel straight over the center of your foot.

I read a book about Abraham Lincoln that described his mental processes as slowly but inexorably penetrating to the heart of every matter. He paid attention to all the sides of an issue, peeling back the layers of emotion and baggage that people allow to accumulate on an issue such as the continuance of slavery in America, while aiming over and over and over for the heart of the matter. He didn’t mind if he didn’t fathom it all at once, he was patient and confident that when he arrived at the core it would be evident. It is this grasp of the issues that led him to be such a fierce debater. His stance was always more deeply rooted than his opponent, who inevitably would grasp at an argument thinking it was the core that Lincoln had already peeled back for himself and exposed as more mere baggage.

Rapid Testers directly approach the areas of greatest risk, greatest uncertainty, least clarity, greatest fear and greatest neglect. These are the leverage points where we can be the most use in the least time. The proper attitude is humble and fearless. Like a great martial artist we understand that we must dispatch these enemies of quality as quickly as possible because although only one thing happens at a time, there are no guarantees about when the next thing will happen. There are no shortcuts to the center though. You aim and release your best energy at it – tests, weight shift, tai chi push – and then adjust as your feedback returns to you.

Testing In Principle: Relaxation

Geordie Keitt — Thu, 15 May 2008 04:17:26 +0000

At the time I first took a class with James Bach on Rapid Software Testing (RST), I had been studying t’ai chi chuan for about 7 years. Throughout the class I kept hearing James say the same things my instructors told me: relax, be open to the energy of what’s around, listen and adapt, and keep your balance. Of course, he didn’t use those exact words. What he said was, “To do Rapid Testing, you don’t need permission. It is completely within your own control.” Keep your balance. “Rapid Testing is like team sports: quickly establishing the proper relationships with your team, handling the ball properly when it comes to you, passing it off to the right person at the right time, staying aware of the time and the score, and furthering the goals of the team.” Adapt to your situation. “Give up responsibility for impossible testing tasks like ‘complete coverage’ and ‘zero defects’, and instead do your best with what you have and be prepared to explain the choices you make.” Relax. “Do everything you can to improve your skill as a testing problem-solver and teller of the quality story.” Practice, practice, practice. T’ai chi will aid everything else you do.

Key principles taught in t’ai chi class are Relaxation, Straightness, Separation of Yin and Yang, Empty Steps, and Focus in the Tan T’ien. I’d like to share how I apply these to Rapid Software Testing.

Imagining Strength, the Relaxed Way

My first t’ai chi teacher, Dan Ogrydziak, made a point about strength. He said that strength is not just in how much we can do, but in how easily we can do it. Imagine, he said, a cartoonish strongman and a pencil-necked geek each lifting a 10 lb barbell. They each could do so, probably, but the stronger one will be much more relaxed about it. Now imagine the weaker man practicing lifting that barbell with as much relaxation as possible, with the goal to be as relaxed as the other man – what would his body look like when he was done? What would lifting that barbell feel like? What else could he lift, or do, more easily? T’ai chi is a path to explore the benefits of this re-definition of strength, strength through softness rather than through the usual method of over-work and damage and rebuilding the muscle tissue, which leads to hard, stiff, bulging muscles.

I had to make a similar transition between the goals I wanted to achieve when I began practicing rapid testing. I’d been told that I was “in charge of quality” and that “we should have bug-free software”, so I was pretty stressed: bugs were my responsibility, since I was in charge of quality, but I didn’t have any power over the bug-insertion process. I looked at the software and decided the UI was too complicated, so I tried to get the design team to change how it did its business, and promptly created significant ill-will with that team. I did not understand the central engine of the software, and did not want to appear dumb by asking too many questions, so I remained ignorant and covered my tail by finding low-hanging bugs. At the same time the code was being developed and morphed at a dizzying rate by a young, talented design team. I was in over my head, and tried to control the chaos by doing what I had been told: Requirements – Test Matrices (“Where are the requirements docs? How can I work like this?”), being the Gatekeeper (“We don’t release it until I say it’s fully tested!”), trying to nail down a Dev Process (“Which bug was this fix for? No bug, no fix!”). This is like being the weak guy, and coveting the way the strong guy looks, with his big bulging, inefficient muscles and his injury-prone joints.

The RST class gave me a new frame of reference for my success: actual value to the team, instead of control of the product. I began by throwing away the process documents and matrices, and asking the people who were in charge what they wanted from me right now. I kept lists of concerns raised by people who mattered, and followed up on them as well as I could. I swallowed my pride and got training in the innermost workings of the system, and ultimately decided that it was not worth my time to test it explicitly, since the developers were using TDD and xUnit – but at least I had done so rationally and with a clear conscience. And I involved the whole team in testing their product with several after-hours bug hunts where the Design team and I assigned user personae to the team and we all acted out scenarios – this built bridges back to the Design team, and also gave the developers a taste of testing, immediately improving my credibility when I asked them for testability suggestions and improvements. In other words, I gave up trying to control everything, and redefined my job into something that I could do where I was with what I had.

I resumed sleeping at night, another fabulous side benefit of practicing relaxation.

Letting Go of Preconceptions

When we relax under our current burdens, we become usefully strong in all the ways that can help us do what we are doing right now. The most important thing that relaxation does for us is to open the pathways of chi energy in our bodies. The feeling is immediate and clear: better blood flow, more energy, better coordination, more sureness and precision in movement. The same physical letting go I do when I relax can have immediate mental and emotional effects too: a clearer sense of the din, the energy right around me, less anxiety, a more appraising eye, more self-confidence and humility. This in turn leads to considered action which is more likely correct or correctable.

In the RST course, James spoke about this by saying, “Testing is applied epistemology.” If we want to know how a thing works, we have to open our eyes and ears and gather evidence, and open our analogic mind to create testable hypotheses which interpret that evidence. Then to test the hypothesis we need to separate what we want to keep stable from what we want to vary, and introduce those variations in a controlled, observable way. The enemy of this process is preconception. Tension, whether mental, emotional, or physical, is the result of anticipating something happening to us like something that happened in the past, and being on hair trigger alert to respond in the pre-planned fashion that our tension facilitates. We become predisposed to interpreting any stimuli as being whatever we are anticipating, whether it really is or not, and once we do that, we trip the trigger of whatever response we have already decided on and have prepared to do. Obviously, this is terrible testing practice. How many entire problem spaces have I ignored because I was not open to seeing any problems there?

James also addressed the idea of using the benefits of relaxation when he spoke about testing in a “flow state”, usually during exploratory testing. When we have granted ourselves the freedom to act as we see fit and to follow up on any leads we see, even within the bounds of a charter, we have chosen to test with our minds open. We can use our enhanced awareness of the energy of the team, of the project, and of the software itself to guide our tests to focus on the greatest risks or the most pressing needs. This is important when we are going to report our testing status and need to be accountable for our decision-making. It was easy when I had my matrix to be able to say, “I had to cover squares B5-C12 today. It isn’t my fault no one cares about the contents.” Now I need to be able to tell my team that I spent the week trying to calm down this particular customer, even at the expense of following up on that bug fix that the developer wanted feedback on. I find that I am in much closer touch with my leaders when I use this approach, because I have to make sure that they buy into my read of the situation. I find I’m usually right, and when I’m not, I am able to change course and accept the new direction with equanimity because my criterion for success is value to the team, not control of the product.

Avoiding Injury

Another important benefit of relaxation is that you avoid injury from failing at something that is too much for you. Tendon and ligament damage only happen when you cannot lift something with some relaxation. By working to do everything with relaxation, we start to notice when we can’t be relaxed, and perhaps ask ourselves what the problem is: too heavy a burden? Too hard an assignment? Too contrary a lover? Too high an expectation? Then we can work to shed the excess and recover our marginal capacity, and our relaxation, and our productivity, and our health.

The analog in the RST course is the Status Report game. “On a scale of 1 to 100”, asks your boss’s boss, “how done is testing?” The sort of answer James was looking for is, “0 to 100 what? I have a better idea, let me tell you about our product and our testing.” The usual answer, of course, was something like “85 – We’ve done most of our testing, there’s still a bunch to go, and 80 sounded too small.” People fall into the trap of playing up to the expectations that they believe others put on them, and then they paint themselves into a corner. Much better is to tell the story of quality, plainly and openly.

In terms of project work, the analog is to say, “I don’t know how to do that (ensure zero defects / program in PHP / make sure we are CMM-Level-2-compliant)” or “You might want to find someone else who can help you with that (gin up numbers for a sales brochure / close out defects to hit a quota / give the thumbs-up for a release)”. Otherwise I just take on stressful jobs that I can’t do.

Better Listening and Responding

One interactive t’ai chi activity we practice is called push hands or listening hands. We stand a few inches apart and place our hands lightly on one another’s arms. Then we relax and listen to each other’s bodies as we move our weight back and forth, with the goal to unbalance your partner while remaining balanced, and to use as little force as possible. It is a lot like trying to hack a system and keep from being hacked, and relaxation is your firewall and also your penetration strategy. With more relaxation you can hear your partner’s body, its limitations and its intentions, sometimes even before it physically moves. At the same time your body loses its tendencies and predictability, gains flexibility in the joints, and is able to sink lower and lower to be more stable, so you are harder to unbalance.

James and Michael Bolton have come up with a vast array of lists of options that are available to a tester: test techniques, analytical methods, planning methods, logistical issues, reporting techniques, and on and on, coalescing the hard experience of decades of testing and thinking into a diamond-dense pith of instantly-accessible heuristics to apply. Much of the RST course is built on these lists, and the goal is to open the student’s eyes to the incredible array of possibilities at their disposal without making it seem like to landscape is so limitless that they freak out. For some reason, my mind does not do well trying to memorize these lists (even though I can remember thousands of lines of song lyrics – strange). But I’ve developed a working familiarity with them, so that I understand when I’m in a situation where one might come in handy.

The most satisfying testing, though, comes when my interaction with the software takes on the characteristics of push hands, and I am working on its balance with variations of data and timing and environment, and it is challenging me with nuances of responses that require deep analysis to be sure what they are saying. Was that “success” message really valid, or was it given to me erroneously? Did it round down properly, or just truncate the response? One false assumption, and I miss a lead, and down I go on my butt when a customer calls with a a bug report. Or, when I write the exact right oracle utility (using perl, of course) for a new feature, and find the hidden showstopper – boom, down the app goes on its butt.

RST = Relaxed Software Testing

A Rapid Tester relaxes and remains useful while situations change around her. She has good work capacity and precision, dampens stress and anxiety, and inspires confidence. She appraises the energy around her and is able to respond appropriately: this rumor is a new risk item, this nagging flicker might be a bug, this email is a charter, this design tidbit is a model update. The Rapid Tester sheds what she cannot lift: she knows when to ask for guidance or state that she does not know how to accomplish an objective. She does her best and reports plainly on her progress. She knows her tendencies, her strengths and weaknesses, and exploits them and those of others in service to her mission. She can pinpoint the source and nature of the issues she encounters, and can imagine options to choose from in how to solve them.

Does this sound like you?

I’m Gonna Be a Rapid Tester Someday

Geordie Keitt — Wed, 02 Jan 2008 04:50:53 +0000

Welcome to my testing blog, offering an interesting, perhaps funny, perhaps entertaining, perhaps thoughtful view of my profession of software testing.

To kick this blog off right, I offer you Rapid Tester, a song about a tester at wit’s end about how to test all the stuff that he’s expected to test, and how the Rapid Testing course gave him hope that he could at least do a defensibly reasonable job given what he has to work with.

I this wrote on the occasion of attending my first RST class at James Bach’s Satisfice world headquarters in Front Royal, VA. I’ve edited it slightly since then as my understanding of the material has grown, but the essence remains the same. It borrows pretty much everything from Steve Earle’s beautiful song “Someday”.

Enjoy the song. Did you change your opinion of yourself or your job because of the RST course or others like it?