Software Tasting --by Geordie Keitt

Pair testing Gmail Notes

Geordie Keitt — Fri, 08 Jan 2010 16:18:09 +0000

This week I spent an hour testing with a complete stranger. And delivered a wallop of value to something he cared a lot about.

Let me back up a bit and offer some context…. I use Gmail for a lot of things: tracking discussions, keeping threaded conversations, creating and working on to-do lists, project management. I chose Gmail over some other GTD-type solutions because a) it has email built in (duh) and b) has a handy labeling feature for placing any particular conversation or thread into many different buckets, so that I don’t have to orthogonally categorize everything.

One thing that is frustrating about using Gmail in this fashion is that there is no clean way to annotate a thread or conversation or group of conversations. My big use case: as the eVoting SIG lead for the AST, I needed a convenient way to track and comment on the various threads of conversation that the IEEE working group was doing, as well as the internal work of the SIG. Since I was using Gmail to read and respond to the various discussions, that seemed like the logical place to annotate the conversation. I could write an email and pop a label on it, but once I send it it’s not editable, and if I don’t send it it clutters up my Drafts folder. Enter Gmail Notes.

A small group of developers decided that they had the same problem, and wrote a Greasemonkey script that used the Gmail API and the Gmail Apps capabilities to offer a note-taking capability, linked to a conversation or a label. This was just super for me, since I could finally do what I’d been unable to before, write an updatable, private note linked to a Gmail thread that would never get lost in the shuffle of a huge conversation. I installed the script, began working on it, and found a few bugs. I reported them back to the lead developer, John, and offered my testing services as part of his team.

Then in December, Gmail pulled support for the part of the API that supported Gmail Notes. So John had to punt. He switched over to using Jetpack instead of Greasemonkey, and came up with another method of identifying the conversation or label. I was called in to give it a spin, and had a hard time configuring it on my system. That’s when I set up an evening Skype session with John.

We used the screen sharing utility on Skype so that John could see my desktop (though not my cursor! Hey Skype, we testers need to see each other’s cursors!), and worked on setting up the Jetpack script correctly on my system. Then I led him through an exploratory test session, working all the functionality through a wide variety of states, and adjusting on the fly to the risks and issues that came up. We noted six separate issues, one of which appeared to happen only on the Mac I was using.

When we were done, I asked him how he liked pair testing like this. He was very pleased with the thoroughness of the testing and the flexibility of the format. He also said he’d be open to testing with a meetup group.

Testaphor Improv

Geordie Keitt — Thu, 07 Jan 2010 16:21:10 +0000

I love analogies – as comedy.

Eddie Izzard (a brilliant British standup comic) does a bit about Anglican ministers who try so hard to be hip and edgy and bring real life into the church. A close paraphrase: “Today’s reading comes from this magazine I found in a hedge. Cosmopolitan, Volume 224, page 58: ‘Lipstick colors this spring will be in the dusky pink area, with eye shadow to match.’ And this rather reminds me … of our Lord Jesus. Because you know, before he rode into Jerusalem on the donkey, he must have got tarted up a bit. Now let us sing hymn 399, ‘O Lord, What is My Hairdo All About?’”

So I have in mind a series called “And this rather reminds me … of Testing.” It might really work as improv – have people toss out ideas and have a panel riff on how it reminds us of testing.

Moderator: “I need an idea for a food item. You sir.”
Audience member: “Mayonnaise!”
Panelist A: “Ah, yes, mayonnaise. One places eggs in a bowl with a little lemon juice, then whisks as one slowly drizzles oil into it. The result is creamy, delicious, and shockingly unhealthful. And this rather reminds me … of Testing. I mean, who besides another mayonnaise maker even knows what a mayonnaise maker does? As far as most consumers of mayonnaise are concerned, it just comes in a jar.”
Panelist B: “Yes – just like testing! And, there are many flavors of mayonnaise.”
Panelist A: “Just like testing! In fact, I would go so far as to say we have transcended analogy and arrived in the realm of metaphor. Testing Is Mayonnaise. Can I get a hell yeah?”

I would find that funny.

Playing the dice game with developers

Geordie Keitt — Thu, 03 Sep 2009 04:44:34 +0000

At CAST2009, I took James Bach’s “Teach Yourself Testing” course. Part of his course was to have the testers play a deduction game with a variety of dice. The testers present a set of dice to the instructor, who responds verbally. The object is to figure out what the instructor’s response is going to be, and write down the algorithm that you deduce is being used to generate the response. The rules are you have to present a set of 5 dice, which you have rolled so their values are random.

I was paired up with another tester. Paul Holland, acting as Jim’s assistant, gave us our pack of dice. There were regular old 6-sided dice with dots, and also 4-, 6-, 12-, and 20-sided dice with Arabic numerals, and 6-sided dice with symbols such as exclamation point, spiral, explosion, skull, flower. One likely glowed in the dark.

My partner and I grabbed a random handful of dice, and rolled them, and asked Paul for his response. “Red blinking zero,” he said. We rolled again. “Red blinking zero,” he said. We rolled another set of dice. “Red blinking zero,” he said. Slowly, I began to detect a pattern…

My partner and I spent the next 30 minutes or so trying all kinds of ways to get Paul to say something other than “Red blinking zero.” We knew it was possible because he would offer other responses to other teams, but we could not find a combination of dice that led us out of that particular leg of the algorithm into some other response. I got pretty peeved and cussed at Paul a couple of times (if you read this, Paul, I’m sorry about that).

Finally, after a couple of hints from Paul, we hit on a strategy that started to give us some traction – though to do so we had to break the rules of the game – and within another 5 minutes had worked out the algorithm to Paul’s satisfaction. My partner and I gave the others a little retrospective on the game and our path to the solution, and decided that though it had started out frustrating it had ended up fun, and so it was a good mirror of our normal experience with testing.

******************

I was pretty excited about the dice game. It exercises some important testing skills: generating hypotheses, surfacing assumptions, gathering data, carefully keeping records of variables and outcomes, deducing cause and effect, putting abstract concepts into words, and perhaps most important, questioning the premise. I thought it would be cool to try it out with some coworkers. First I approached my testing teammate Anoop and explained the rules and the object of the game. He spent about 20 minutes with me, going down some of the same deadends that I did, and finally hit on the correct algorithm. He thought it was fun.

Next I asked a developer to play. I explained the rules and the object of the game to him, and he started showing me sets of dice. Within 2 minutes he quit. “I’m a coder. I don’t guess what the code is, I read the code.” He was polite enough to stick around to find out what the algorithm was, and to hear why I thought it was a useful testing game, but I could tell his heart wasn’t in it. He simply didn’t care to guess.

Then I asked another coder to play. He told me he had a couple minutes, so I set up the dice and told him the rules and the object. In response to his first couple sets I told him “zero”. The third time he rolled the dice I said, “Red blinking zero.” He immediately quit, throwing up his hands into the air and exclaiming, “What kind of crazy game is this? What other responses are in this algorithm, purple… people-eating… zeroes? And what’s the point of having rules if all they do is make it impossible to win?” He was pretty worked up (I’m pretty sure he mentioned sunspots as a possible variable in the algorithm, a good indicator of his worked-up-ness). We talked it over for a while, and the gist of his argument was that the puzzle was just way too hard for the couple minutes he had to devote to it. (To recap: He quit the game because he could see that it was a pretty open-ended problem that he did not want to devote the time it would take since he only had a couple minutes, then we immediately had a twenty-minute discussion on why the game would have taken too long. People are fascinating.)

He compared the dice game to the children’s game “Aunt Bertie Fish”, the object of which is to figure out what category of things Aunt Bertie Fish loves by being told a litany of things she loves and things she hates. An example I just made up: “Aunt Bertie Fish loves apples and hates bananas. Aunt Bertie Fish loves stop signs and hates street signs. Aunt Bertie Fish loves Boston baseball and hates Chicago baseball.” I’m outlining a pair of mutually-exclusive sets, X and Not-X, using tropes and examples which are common in my cultural context. In this case, Aunt Bertie Fish loves things that are red and hates anything that is not red.

According to my colleague, the dice game is like Aunt Bertie Fish except without the X. All he is getting is not-X, in which case X could be anything. Or else he is getting X and Y and Z, rather than a simple X – Not-X universe. It is impossible to bound the game using logic a priori, and it is in fact kind of unfair that the instructor does not offer the complete set of possible responses. And don’t get him started on how wrong it is to set up rules that have to be broken to succeed at the game. If someone played Aunt Bertie Fish with him that way, it wouldn’t be any fun at all.

******************

Slowly, I began to detect a pattern. Tester plays dice game: works through it, thinks it is fun, and recognizes it is relevant to day-to-day testing work. Developer plays dice game: gets upset or bored, quits.

It’s tempting to write off the first developer as a bad tester, but actually he is a fabulous tester who writes code in which it is very difficult to find problems.

It’s tempting to feel insulted that the second developer thought the dice game wasn’t any fun, but he has obvious respect for the tester’s willingness to jump into the game, sink or swim. My theory: this developer has a very healthy respect for the vastness of the testing problem space, and he hopes that games like this do not teach testers to be glib in the face of it, and to jump to conclusions too quickly, just because you figured out /this/ algorithm doesn’t mean that the /next one/ won’t depend on sunspots, and can never be solved.

How do your colleagues react to the dice game?

******************

Epilogue: I also played the game with my wife Sarah (who evidently loves me very much). Her advice on using the game to introduce testing to non-testers, was to take it slow. Start with a super-simple algorithm and 2 dice, and no “rules”. Then play again and incrementally add complexity: harder algorithms, more dice, more types of dice, various “rules”. If anyone has an opportunity to try this tactic on a set of interested non-testers, I’d like to get your thoughts.

CAST2009: “Rapid Tester” video

Geordie Keitt — Sat, 29 Aug 2009 03:48:17 +0000

View the post to see the video…

Another of Becky Fiedler’s videos from CAST 2009 in CO Springs. When “Feature Creep” was over, I had enough time in my Lightning Talk slot for an encore. Jim Bach wanted to hear “Rapid Tester”, so I played it.

CAST2009: “Black Box” video

Geordie Keitt — Sat, 29 Aug 2009 03:36:29 +0000

View the post to see the video…

Becky Fiedler shot this performance of “Black Box” after dinner was all over and we were just hanging out in the hallway. It pretty quickly turned into a Jonathan Coulton love-fest after the testing songs were played…

CAST2009: “Feature Creep” song & video

Geordie Keitt — Fri, 28 Aug 2009 19:11:24 +0000

View the post to see the video…

I performed this a month or so ago at CAST2009 in Colorado Springs, where Becky Fiedler recorded it. The intro to this song went, “This song is written from the perspective of a piece of bloatware that used to be sleek and clean.” I did this during a Lightning Talks session, meaning I had to bring it in under 4 minutes. That’s why there’s little time for dramatic pauses…

Talkin’ eVoting at CAST 2009

Geordie Keitt — Thu, 02 Jul 2009 17:00:24 +0000

Jim Nilius and I will be taking our act on the road again, this time to CAST2009. Our discussion at WREST 2 in Indianapolis on why regulated testing does a poor job of testing software produced a lot of great ideas for how to clarify the problems with using 19th-Century test methods to evaluate computerized voting systems.

Jim and I are preparing a strong argument for the Election Assistance Commission to overhaul the NIST certification standards to help certification labs handle testing complex software systems much more congruently, and we are taking this argument for a test drive at CAST. So come on over and test it, test us, rip it to shreds. We will rebuild it. We have the technology, we have the capability to make it better than it was before. Better, stronger, faster…

WREST 2: Regulation Disrupts Sapience

Geordie Keitt — Fri, 05 Jun 2009 03:30:26 +0000

Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.

-From the Wikipedia entry for “Sapience”

At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.

I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.

Regulated Testing PNG

(Here is the PDF. Contact me via gmail for the original ODP, editable in OpenOffice, if you wish to modify or upgrade this document under the publishing rules of WREST.)

Note the small shaded section in the middle which depicts rapid cycling between test creation, test execution, and test reporting in the process of testing and learning about the system. This dynamic allows the tester’s brain to work. Disrupting this cycle disrupts the tester’s systemic awareness and renders testing infinitely less effective.

The term “sapience” has a special place in the heart of a context-driven-school tester like myself, given James Bach’s 2007 coinage of the term “sapient process” to mean “any process that relies on skilled humans” to draw a contrast between skilled testing by people vs. automated testing by computers. Many testers extend the contrast to include scripted testing of just about any kind, whether performed by people or computers. I would like to extend the contrast to include any testing that does not adapt to new information. As in the opening block-quote, a sapient tester displays good judgment in a complex, dynamic environment. To do this, a tester has to model the complexities of the system in her mind and keep that model alive as she interacts with the system. When information arrives that contradicts the model, she must adjust the model and question the validity of the adjustment – was that a bug I just saw, or was it a hint of a bug, or a problem with my understanding of the requirement, or…? To find out, she devises a new series of tests and runs them, and then adapts again to the new information. This is the essence of the learning cycle depicted in the diagram.

It doesn’t take much to interrupt the cognitive flow and bring the systems awareness of the tester tumbling down. Without a living system model in her mind she is comparatively non-sapient with respect to the system, and any testing activities she undertakes will be insensitive to new information, thus incapable of adapting to it, and thus (by my definition) her testing is not sapient. In other words, rote, mechanical, all the qualities we ascribe to poor testing.

In a regulated environment, sapient testing is not simply discouraged, it is extremely risky. Please refer to the diagram: every step in the testing cycle can initiate a process that threatens the existence of the testing organization. In a certification lab, such as for electronic voting systems, the following dynamics apply at a minimum:

Each new test that is created can potentially initiate a cycle of test method validation and re-accreditation of the regulated testing organization, which puts the viability of the company in jeopardy.

Every test report can potentially negatively influence the likelihood that the system will be certified, leading to less business with vendors who would prefer to work with a less picky test lab.

Every test executed can potentially fail an audit, resulting in expensive “process improvements” and possibly kicking off another round of test method validation and the risk of losing accreditation.

So the company risks its very existence whenever it engages in testing, even though testing is what the company was chartered to do, and the only method it has for minimizing risk is to minimize testing! It does so by imposing “process” on testing such that every act of test creation is followed by an expensive validation and record-keeping procedure, rather than by test execution. And of course vice versa.

Under this regime the testers have no means by which to develop system awareness and sapience, except to divide their testing into “official process” testing and “shadow process” testing. In the shadow process, the testers can test. But the “shadow tester” must be very judicious about how to bring what she learns back into the “official process” world without risking the existence of the company.

I would appreciate learning your thoughts on this.

The 8% Rule

Geordie Keitt — Fri, 29 May 2009 18:29:17 +0000

Last Saturday night I played guitar, both solo and accompanying my friend Rob Balder, at a gig at Balticon, a sci-fi/fantasy/gaming convention in Baltimore. It was not one of my best performances. Rob and I did a new arrangement of a tune we had done before, and it never hit its stride, it felt slow and square. Another tune I sang well but my guitar work was kind of iffy, it was the first time I’d done it in front of an audience. (I did a fine job on one song, where all I did was play guitar instead of having to sing as well, and I just executed the song.)

I can pick out greater and lesser reasons for my struggles. There was the strangeness of the venue: we performed in a hotel lobby between the front desk and the bar, so there was constant traffic through the audience, no backstage area, and a very cramped space. The sound system was very small with no available inputs for my guitar, and there were ony 2 microphones, which meant that I had to hold the guitar right up to one mic and sing into the other, and on songs where I was not the lead singer, I had to have the lead singer hold the mic up to my mouth – and if he didn’t, I may as well not be singing at all. The speakers were aimed solely out at the audience with no monitoring system to allow the musicians to hear ourselves. These factors can influence a musician’s comfort level and make mistakes much harder to pick up and correct.

I did one song in an entirely different, though equally stressful environment: I collaborated with Paul Fischer on a parody version of a song written by George Hrab, an extremely funny and talented singer / songwriter, and sang our version in front of him during his weekly podcast which was being recorded in a suite with about 100 people crammed into it. (The original is called “FAR“. By the way, George is my new favorite person.) Earlier in the evening George had performed a long set on the lobby stage, and had done “FAR”, and I’d watched him voice the chords on the guitar /entirely differently/ than the way I had picked it out. So I was very intimidated by standing 2 feet from him and playing my dinky version of his song, with all the wrong voicings.

Something funny happened then, though. I’d practiced that silly parody song enough to know how to play it even with all the other distractions. And I have a lot of confidence in my singing – I can hit notes, and my voice has a particular tonality that I trust even in a crowded room to cut through the hubbub and make people listen. My collaborator Paul was the host of the show,and he encouraged full-audience ownership of the song by holding up cue cards telling everyone when to sing along. And God bless George Hrab, he read the lyrics off the page of my cheat sheet with me and sang /backup/ on his /own song/. And that song went over like you wouldn’t believe. I even figured out about 3/4 of the way through that I had my guitar tuned wrong! I am ashamed to admit that I hadn’t retuned my low E back up from D after the previous song, but it ultimately didn’t matter, the song was so much fun no one cared. Link to the podcast and move along to about the two-thirds mark to hear it. (Caution: colorful language …)

**************

What the hell is this gig report doing in a testing blog? I’m getting to that, please return to your seat until the captain turns off the fasten seatbelt sign.

**************

I ran back into George later in the evening as he was packing up the lobby stage. He asked me how the rest of my playing went, and I told him about the troubles I had and the issues with the weird venue and the gear. He nodded sympathetically while he loaded everything up on a cart, and said, “Music only takes up 8% of your thought process while you are playing. The rest is on-the-fly problem solving, and unresolved angst.” (Then he pulled out his guitar and very kindly showed me the proper voicings for his song, and made me play them back for him to be sure I had them under my fingers. A prince among men, I tell you.)

So, to recap – George and I had played and sung under identical conditions, on the same stage and through the same gear. George had sounded fantastic and looked comfortable and had done a brilliant job. Me – not so much. Why? The answer was in that 8% of the thought process that was available for music. I rolled out a lot of new stuff, and in my basement and even in rehearsal it sounded fine, but it took a lot more than 8% of my thought process to get it to sound that way. If I’d asked myself how much it did take I probably would have said more than 8%, but I never did – I didn’t fully understand the budget constraints I was under. And on stage, that 8% was aggressively enforced.

I needed more practice. I needed /a lot/ more practice, orders of magnitude more practice. I needed to practice so much that I could do a bang up job on any song in the set with less than 10% of my operational musical capacity engaged. That’s what George had done, which is why he sounded so great.

*************

Good for George Hrab, but what’s in this for me? Glad you asked. Riddle me this: How is testing any different than this experience report? When you are under a deadline, and you roll out a test tool or technique that you are new at, don’t you find that the testing itself takes up only a small fraction of your thought process, and suffers for it? Your audience is still looking for that polished professional job though, so practice, and plan for the day of the performance so that you can squeeze the best possible testing out of the 8% of your brain that will actually be available for it.

And I truly hope you have as graceful an experience with your testing as I did on that podcast. I received stakeholder ownership, support from the top, and respectful listening behavior modeled towards me by the alpha dog on the team. Anytime you get all that, your testing is sure to be a great success.

Testing In Principle: Root

Geordie Keitt — Tue, 25 Nov 2008 03:58:23 +0000

T’ai chi hurts to do. The benefits are enormous, and the cost is pain. The nice part is, the same deal applies to most things in life. If you want to be a great guitar player, your fingers will hurt. If you want to be a great athlete, your muscles will hurt. My teacher Chris Casey says, “There is no way out. But there is a way through.”

What hurts is letting go of the tension in my body and feeling that tension come into consciousness. When I work this while standing and imagining that I am connected with the earth through the soles of my feet, the pain is incredible in my feet and legs, the burn of chi blowing through parts of my body that I’ve denied it access to for many years. The gains of course are commensurate with the effort I put in. The key win while rooting is that I am connected to something much larger than myself, a thing to which everything and everyone else is connected as well: the earth. When I am participating consciously in that connection I have insight into the connections others have with the earth as well, and can analyze and manipulate it to my benefit in tai chi, and to their benefit too as a friend. The pain comes from losing the shell of ego I have built that separates me from my surroundings.

The testing parallel is obvious from my other blog entries: the bare facts of your context are the ground you stand on when choosing what to test and why. A great Rapid Tester does not arrive in a context already thinking she knows all she needs to know, fully formed, individuated and full of opinions about what is best. She learns. She crawls, then walks, then runs. Asks questions, appears clueless, and opens herself to learning about everything from everything. She molds herself into a teammate in order to be useful, takes on all the different viewpoints of the stakeholders, and also learns to “think like the code” (model the system) as well.

This is a painful process for those of us with egos that tell us to look like we have the answers. It is hard work to catalog my biases and weaknesses, and to systematically counteract them. One of the best methods for analyzing my tendencies is to participate in friendly competitions and debates with fellow testers. I learn that all of my favorite tricks have pitfalls, and there are a lot of great ways to test and manage testing that I have not yet thought of. I have become more comfortable with that fact. And I have become comfortable in my community, a vibrant and living aspect of my context that I am very happy to be able to tap when necessary.

One more point about Root. The tai chi player with more root always wins. That is because they can extend their root beneath their opponent and cut them off from the earth. A good player can literally bounce me like a basketball when they sink below me, test my connection to the earth, and find the flaws in it. We have the same job as testers, to probe the structure of our systems and the strength and depth of their logic and implementation, and to work on the flaws we see or suspect, until we break the illusion that the system “just works” and can bounce it around like a basketball. When I do good Rapid Testing, I am sinking deeply into the problem space, asking the hardest questions, getting at the heart of the matter, and feeling the burn.