Software Tasting --by Geordie Keitt » james bach

WREST 2: Regulation Disrupts Sapience

Geordie Keitt — Fri, 05 Jun 2009 03:30:26 +0000

Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.

-From the Wikipedia entry for “Sapience”

At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.

I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.

Regulated Testing PNG

(Here is the PDF. Contact me via gmail for the original ODP, editable in OpenOffice, if you wish to modify or upgrade this document under the publishing rules of WREST.)

Note the small shaded section in the middle which depicts rapid cycling between test creation, test execution, and test reporting in the process of testing and learning about the system. This dynamic allows the tester’s brain to work. Disrupting this cycle disrupts the tester’s systemic awareness and renders testing infinitely less effective.

The term “sapience” has a special place in the heart of a context-driven-school tester like myself, given James Bach’s 2007 coinage of the term “sapient process” to mean “any process that relies on skilled humans” to draw a contrast between skilled testing by people vs. automated testing by computers. Many testers extend the contrast to include scripted testing of just about any kind, whether performed by people or computers. I would like to extend the contrast to include any testing that does not adapt to new information. As in the opening block-quote, a sapient tester displays good judgment in a complex, dynamic environment. To do this, a tester has to model the complexities of the system in her mind and keep that model alive as she interacts with the system. When information arrives that contradicts the model, she must adjust the model and question the validity of the adjustment – was that a bug I just saw, or was it a hint of a bug, or a problem with my understanding of the requirement, or…? To find out, she devises a new series of tests and runs them, and then adapts again to the new information. This is the essence of the learning cycle depicted in the diagram.

It doesn’t take much to interrupt the cognitive flow and bring the systems awareness of the tester tumbling down. Without a living system model in her mind she is comparatively non-sapient with respect to the system, and any testing activities she undertakes will be insensitive to new information, thus incapable of adapting to it, and thus (by my definition) her testing is not sapient. In other words, rote, mechanical, all the qualities we ascribe to poor testing.

In a regulated environment, sapient testing is not simply discouraged, it is extremely risky. Please refer to the diagram: every step in the testing cycle can initiate a process that threatens the existence of the testing organization. In a certification lab, such as for electronic voting systems, the following dynamics apply at a minimum:

Each new test that is created can potentially initiate a cycle of test method validation and re-accreditation of the regulated testing organization, which puts the viability of the company in jeopardy.

Every test report can potentially negatively influence the likelihood that the system will be certified, leading to less business with vendors who would prefer to work with a less picky test lab.

Every test executed can potentially fail an audit, resulting in expensive “process improvements” and possibly kicking off another round of test method validation and the risk of losing accreditation.

So the company risks its very existence whenever it engages in testing, even though testing is what the company was chartered to do, and the only method it has for minimizing risk is to minimize testing! It does so by imposing “process” on testing such that every act of test creation is followed by an expensive validation and record-keeping procedure, rather than by test execution. And of course vice versa.

Under this regime the testers have no means by which to develop system awareness and sapience, except to divide their testing into “official process” testing and “shadow process” testing. In the shadow process, the testers can test. But the “shadow tester” must be very judicious about how to bring what she learns back into the “official process” world without risking the existence of the company.

I would appreciate learning your thoughts on this.

Testing In Principle: Work Directly Toward the Center

Geordie Keitt — Sat, 18 Oct 2008 01:19:36 +0000

When I worked with James Bach on a Satisfice project he would ask me uncomfortable questions about what I was doing and why I was doing it. He’d ask, “Tell me what this bug is at its core.” I’d say, “When I click this button, I get an error message instead of the dialog I’m expecting.”

“Why do you expect a dialog?”

“Because that’s how the app is supposed to work.”

“How do you know?”

“It worked that way another time.”

“Ahh, so there is an inconsistency between how it worked before and the way it works now. What is different? How do you know that they are not both right?”

What he needed me to get at was the clearest possible understanding of the issue, and the clearest and simplest expression of it. He imposed a 75-character limit on bug report summaries.

A typical progression of a bug report might go like this:

Error message appears after user selects calendar icon on application page

Calendar sometimes fails to load on application page

Calendar sometimes fails to load on any page

Calendar sometimes fails to load on low-RAM system (256MB)

All Java calendars fail to load when less than 50MB RAM available

I have found that I know when I have hit the core of a bug (or at least close enough). My mind comes to rest. Robert Pirsig describes the feeling very nicely in “Zen and the Art of Motorcycle Maintenance”: “the material and the craftsman’s thoughts change together in a progression of smooth, even changes until his mind is at rest at the exact instant the material is right.”

In t’ai chi we focus on the center of everything. The center of your weight, the tan tien, is the source of all your power. If you can control your opponent’s center then you can control his balance in push hands and you will win. When you shift weight you will shift it to the center of your foot and the center of your knee will travel straight over the center of your foot.

I read a book about Abraham Lincoln that described his mental processes as slowly but inexorably penetrating to the heart of every matter. He paid attention to all the sides of an issue, peeling back the layers of emotion and baggage that people allow to accumulate on an issue such as the continuance of slavery in America, while aiming over and over and over for the heart of the matter. He didn’t mind if he didn’t fathom it all at once, he was patient and confident that when he arrived at the core it would be evident. It is this grasp of the issues that led him to be such a fierce debater. His stance was always more deeply rooted than his opponent, who inevitably would grasp at an argument thinking it was the core that Lincoln had already peeled back for himself and exposed as more mere baggage.

Rapid Testers directly approach the areas of greatest risk, greatest uncertainty, least clarity, greatest fear and greatest neglect. These are the leverage points where we can be the most use in the least time. The proper attitude is humble and fearless. Like a great martial artist we understand that we must dispatch these enemies of quality as quickly as possible because although only one thing happens at a time, there are no guarantees about when the next thing will happen. There are no shortcuts to the center though. You aim and release your best energy at it – tests, weight shift, tai chi push – and then adjust as your feedback returns to you.

Testing In Principle: Relaxation

Geordie Keitt — Thu, 15 May 2008 04:17:26 +0000

At the time I first took a class with James Bach on Rapid Software Testing (RST), I had been studying t’ai chi chuan for about 7 years. Throughout the class I kept hearing James say the same things my instructors told me: relax, be open to the energy of what’s around, listen and adapt, and keep your balance. Of course, he didn’t use those exact words. What he said was, “To do Rapid Testing, you don’t need permission. It is completely within your own control.” Keep your balance. “Rapid Testing is like team sports: quickly establishing the proper relationships with your team, handling the ball properly when it comes to you, passing it off to the right person at the right time, staying aware of the time and the score, and furthering the goals of the team.” Adapt to your situation. “Give up responsibility for impossible testing tasks like ‘complete coverage’ and ‘zero defects’, and instead do your best with what you have and be prepared to explain the choices you make.” Relax. “Do everything you can to improve your skill as a testing problem-solver and teller of the quality story.” Practice, practice, practice. T’ai chi will aid everything else you do.

Key principles taught in t’ai chi class are Relaxation, Straightness, Separation of Yin and Yang, Empty Steps, and Focus in the Tan T’ien. I’d like to share how I apply these to Rapid Software Testing.

Imagining Strength, the Relaxed Way

My first t’ai chi teacher, Dan Ogrydziak, made a point about strength. He said that strength is not just in how much we can do, but in how easily we can do it. Imagine, he said, a cartoonish strongman and a pencil-necked geek each lifting a 10 lb barbell. They each could do so, probably, but the stronger one will be much more relaxed about it. Now imagine the weaker man practicing lifting that barbell with as much relaxation as possible, with the goal to be as relaxed as the other man – what would his body look like when he was done? What would lifting that barbell feel like? What else could he lift, or do, more easily? T’ai chi is a path to explore the benefits of this re-definition of strength, strength through softness rather than through the usual method of over-work and damage and rebuilding the muscle tissue, which leads to hard, stiff, bulging muscles.

I had to make a similar transition between the goals I wanted to achieve when I began practicing rapid testing. I’d been told that I was “in charge of quality” and that “we should have bug-free software”, so I was pretty stressed: bugs were my responsibility, since I was in charge of quality, but I didn’t have any power over the bug-insertion process. I looked at the software and decided the UI was too complicated, so I tried to get the design team to change how it did its business, and promptly created significant ill-will with that team. I did not understand the central engine of the software, and did not want to appear dumb by asking too many questions, so I remained ignorant and covered my tail by finding low-hanging bugs. At the same time the code was being developed and morphed at a dizzying rate by a young, talented design team. I was in over my head, and tried to control the chaos by doing what I had been told: Requirements – Test Matrices (“Where are the requirements docs? How can I work like this?”), being the Gatekeeper (“We don’t release it until I say it’s fully tested!”), trying to nail down a Dev Process (“Which bug was this fix for? No bug, no fix!”). This is like being the weak guy, and coveting the way the strong guy looks, with his big bulging, inefficient muscles and his injury-prone joints.

The RST class gave me a new frame of reference for my success: actual value to the team, instead of control of the product. I began by throwing away the process documents and matrices, and asking the people who were in charge what they wanted from me right now. I kept lists of concerns raised by people who mattered, and followed up on them as well as I could. I swallowed my pride and got training in the innermost workings of the system, and ultimately decided that it was not worth my time to test it explicitly, since the developers were using TDD and xUnit – but at least I had done so rationally and with a clear conscience. And I involved the whole team in testing their product with several after-hours bug hunts where the Design team and I assigned user personae to the team and we all acted out scenarios – this built bridges back to the Design team, and also gave the developers a taste of testing, immediately improving my credibility when I asked them for testability suggestions and improvements. In other words, I gave up trying to control everything, and redefined my job into something that I could do where I was with what I had.

I resumed sleeping at night, another fabulous side benefit of practicing relaxation.

Letting Go of Preconceptions

When we relax under our current burdens, we become usefully strong in all the ways that can help us do what we are doing right now. The most important thing that relaxation does for us is to open the pathways of chi energy in our bodies. The feeling is immediate and clear: better blood flow, more energy, better coordination, more sureness and precision in movement. The same physical letting go I do when I relax can have immediate mental and emotional effects too: a clearer sense of the din, the energy right around me, less anxiety, a more appraising eye, more self-confidence and humility. This in turn leads to considered action which is more likely correct or correctable.

In the RST course, James spoke about this by saying, “Testing is applied epistemology.” If we want to know how a thing works, we have to open our eyes and ears and gather evidence, and open our analogic mind to create testable hypotheses which interpret that evidence. Then to test the hypothesis we need to separate what we want to keep stable from what we want to vary, and introduce those variations in a controlled, observable way. The enemy of this process is preconception. Tension, whether mental, emotional, or physical, is the result of anticipating something happening to us like something that happened in the past, and being on hair trigger alert to respond in the pre-planned fashion that our tension facilitates. We become predisposed to interpreting any stimuli as being whatever we are anticipating, whether it really is or not, and once we do that, we trip the trigger of whatever response we have already decided on and have prepared to do. Obviously, this is terrible testing practice. How many entire problem spaces have I ignored because I was not open to seeing any problems there?

James also addressed the idea of using the benefits of relaxation when he spoke about testing in a “flow state”, usually during exploratory testing. When we have granted ourselves the freedom to act as we see fit and to follow up on any leads we see, even within the bounds of a charter, we have chosen to test with our minds open. We can use our enhanced awareness of the energy of the team, of the project, and of the software itself to guide our tests to focus on the greatest risks or the most pressing needs. This is important when we are going to report our testing status and need to be accountable for our decision-making. It was easy when I had my matrix to be able to say, “I had to cover squares B5-C12 today. It isn’t my fault no one cares about the contents.” Now I need to be able to tell my team that I spent the week trying to calm down this particular customer, even at the expense of following up on that bug fix that the developer wanted feedback on. I find that I am in much closer touch with my leaders when I use this approach, because I have to make sure that they buy into my read of the situation. I find I’m usually right, and when I’m not, I am able to change course and accept the new direction with equanimity because my criterion for success is value to the team, not control of the product.

Avoiding Injury

Another important benefit of relaxation is that you avoid injury from failing at something that is too much for you. Tendon and ligament damage only happen when you cannot lift something with some relaxation. By working to do everything with relaxation, we start to notice when we can’t be relaxed, and perhaps ask ourselves what the problem is: too heavy a burden? Too hard an assignment? Too contrary a lover? Too high an expectation? Then we can work to shed the excess and recover our marginal capacity, and our relaxation, and our productivity, and our health.

The analog in the RST course is the Status Report game. “On a scale of 1 to 100”, asks your boss’s boss, “how done is testing?” The sort of answer James was looking for is, “0 to 100 what? I have a better idea, let me tell you about our product and our testing.” The usual answer, of course, was something like “85 – We’ve done most of our testing, there’s still a bunch to go, and 80 sounded too small.” People fall into the trap of playing up to the expectations that they believe others put on them, and then they paint themselves into a corner. Much better is to tell the story of quality, plainly and openly.

In terms of project work, the analog is to say, “I don’t know how to do that (ensure zero defects / program in PHP / make sure we are CMM-Level-2-compliant)” or “You might want to find someone else who can help you with that (gin up numbers for a sales brochure / close out defects to hit a quota / give the thumbs-up for a release)”. Otherwise I just take on stressful jobs that I can’t do.

Better Listening and Responding

One interactive t’ai chi activity we practice is called push hands or listening hands. We stand a few inches apart and place our hands lightly on one another’s arms. Then we relax and listen to each other’s bodies as we move our weight back and forth, with the goal to unbalance your partner while remaining balanced, and to use as little force as possible. It is a lot like trying to hack a system and keep from being hacked, and relaxation is your firewall and also your penetration strategy. With more relaxation you can hear your partner’s body, its limitations and its intentions, sometimes even before it physically moves. At the same time your body loses its tendencies and predictability, gains flexibility in the joints, and is able to sink lower and lower to be more stable, so you are harder to unbalance.

James and Michael Bolton have come up with a vast array of lists of options that are available to a tester: test techniques, analytical methods, planning methods, logistical issues, reporting techniques, and on and on, coalescing the hard experience of decades of testing and thinking into a diamond-dense pith of instantly-accessible heuristics to apply. Much of the RST course is built on these lists, and the goal is to open the student’s eyes to the incredible array of possibilities at their disposal without making it seem like to landscape is so limitless that they freak out. For some reason, my mind does not do well trying to memorize these lists (even though I can remember thousands of lines of song lyrics – strange). But I’ve developed a working familiarity with them, so that I understand when I’m in a situation where one might come in handy.

The most satisfying testing, though, comes when my interaction with the software takes on the characteristics of push hands, and I am working on its balance with variations of data and timing and environment, and it is challenging me with nuances of responses that require deep analysis to be sure what they are saying. Was that “success” message really valid, or was it given to me erroneously? Did it round down properly, or just truncate the response? One false assumption, and I miss a lead, and down I go on my butt when a customer calls with a a bug report. Or, when I write the exact right oracle utility (using perl, of course) for a new feature, and find the hidden showstopper – boom, down the app goes on its butt.

RST = Relaxed Software Testing

A Rapid Tester relaxes and remains useful while situations change around her. She has good work capacity and precision, dampens stress and anxiety, and inspires confidence. She appraises the energy around her and is able to respond appropriately: this rumor is a new risk item, this nagging flicker might be a bug, this email is a charter, this design tidbit is a model update. The Rapid Tester sheds what she cannot lift: she knows when to ask for guidance or state that she does not know how to accomplish an objective. She does her best and reports plainly on her progress. She knows her tendencies, her strengths and weaknesses, and exploits them and those of others in service to her mission. She can pinpoint the source and nature of the issues she encounters, and can imagine options to choose from in how to solve them.

Does this sound like you?

The Black Box Testing Song

Geordie Keitt — Tue, 25 Mar 2008 03:26:10 +0000

Cem Kaner and James Bach presented a course on Black Box Software Testing that I took circa 2003. I performed this song, Black Box, to wrap up the training. The reference to “underwear and dirty socks”, besides making a pleasant rhyme with “black box”, refers to Cem’s comment that once you get to know a particular developer’s personal developmental hygiene, you can start rummaging around their code looking for their characteristic errors – their dirty laundry, as it were. We spent a little time learning to make test data that is easy to grow to gigantic size and then shrink as needed, and how to recognize when this type of attack might be useful. We also discussed how to turn any statement about the software into a testable requirement, including marketing claims, hence the comment about the Marketing Department, and spent quite a bit of time going over good ways to review specification documents for ambiguity and raise questions that may save confusion later on. Especially if the developers are as frazzled as the ones in the song.

The most important lessons that I learned, though were these: when in doubt, start testing. And notice when you are running across the same bugs as before, stop retracing your own footsteps and try something new. I tried to capture these ideas in the song as well.

I had a lot of fun writing and recording this, and I hope you enjoy it. In case you couldn’t quite tell, it was ripped off from “Smooth” by Rob Thomas and Santana. See the page about it on the Audible Stuff blog for more music notes.

I’m Gonna Be a Rapid Tester Someday

Geordie Keitt — Wed, 02 Jan 2008 04:50:53 +0000

Welcome to my testing blog, offering an interesting, perhaps funny, perhaps entertaining, perhaps thoughtful view of my profession of software testing.

To kick this blog off right, I offer you Rapid Tester, a song about a tester at wit’s end about how to test all the stuff that he’s expected to test, and how the Rapid Testing course gave him hope that he could at least do a defensibly reasonable job given what he has to work with.

I this wrote on the occasion of attending my first RST class at James Bach’s Satisfice world headquarters in Front Royal, VA. I’ve edited it slightly since then as my understanding of the material has grown, but the essence remains the same. It borrows pretty much everything from Steve Earle’s beautiful song “Someday”.

Enjoy the song. Did you change your opinion of yourself or your job because of the RST course or others like it?