Software Tasting

Rapid Tester lyrics

Geordie Keitt — Fri, 04 May 2012 22:10:40 +0000

I wrote up the lyrics for “Rapid Tester” on another fallow blog of mine. So for those who are interested in the actual rhetorical content of the tune, click to see.

Sadly I have a couple of other blogs lying around, neglected for years now. Sigh. But I like to think my family and I have benefited from all the attention I’m not lavishing on them.

What is smart?

Geordie Keitt — Fri, 20 Apr 2012 23:35:01 +0000

Smart can’t be just what you know. Prehistoric smart people didn’t have all the knowledge we have today, and what knowledge they did have, of weather and edible food and hunting and snow conditions and tribal differences, we do not have. Yet they were smart, as are we.

Smart can’t be just how much you know. Smart children may have very little knowledge.

Smart can’t just be how quickly you learn. I know many smart people who are stubborn and set in their ways (cough *Dad* cough).

So what is it?

I’m going to define “smart” for the purposes of this blog as “able to connect concepts effectively in unexpected ways.”

The simplest kind of concept connection is to call to mind a concept at an appropriate time and bring it forth. This is connecting a concept to the situation. If someone calls forth a concept inappropriately it will likely be ineffective in the situation and we will downgrade our opinion of how smart they are. This is a “Everything looks like a nail” situation.

If a person calls to mind a particularly effective concept, and if the connection is particularly unexpected, regardless of validity, then our estimation of their smartness will go up. The quality of unexpectedness also gives us a clue as to their sense of humor.

If a person calls to mind a range of concepts and arranges them around the situation, then our estimation goes up again. This kind of simple diagram is what this level of processing feels like:

courtesy irunabiz.com, found via google image search for "cloud diagram"

If they arrange them using causal connections one to the other, our estimation goes up again, and we can begin to imagine that they can be effective working on tasks and problems over longer time scales. This kind of thinking is what creates Weinberg system diagrams like this one:

courtesy the Thinking For a Change blog, found by Google images search for "Weinberg diagram"

If a person calls to mind a broad range of concepts which are related in parallel chains of causation that intersect eventually, our estimation goes up again.

Why am I bringing up expectations? Since “smart” is a qualitative notion it exists only in relation to something. In this case it is the observer’s idea about what is normal for the class of person whom they are evaluating. The level of facility with concepts that we will be astonished by in a toddler is of course much lower than in an adolescent, which will be lower than in a 20-year pro. And our own ability to connect concepts at these different levels limits our ability to expect that others will be able to do so, meaning that we have a tough time not counting anyone as smart who surpasses our level.

Counting on Friends

Geordie Keitt — Wed, 31 Mar 2010 03:54:18 +0000

At ProChain we are developing the capability for our software to be installed on operating systems using languages other than English. Naturally, I am called upon to test this capability. I chose to test in several languages: French, for the latin alphabet, Russian for Cyrillic, Hebrew for the right-to-left orientation, and Chinese for the massive character set. Unfortunately I am a monolingual guy, which poses some issues. For instance, I have no idea how to run a Hebrew language Windows XP installer.

Which does not daunt me, because I have confidence in my network. I contacted a tester with whom I’ve corresponded on a software testing Google group who works in Israel, and asked him to help. My teammate Anoop emailed him screen shots when it was time to click the next button and he responded with a note on which to click. It seems to have worked fine. Now I owe him a solid, and will gladly help him when he needs me.

I the same vein I have several cousins who are fluent Russian speakers, and a teammate is French. I will need a lead on someone to help with the Chinese setup, but I’ll cross that bridge when I get there. I count on my friends to be there for me when I need them, and just so you know, I’m here for y’all too.

Pair testing Gmail Notes

Geordie Keitt — Fri, 08 Jan 2010 16:18:09 +0000

This week I spent an hour testing with a complete stranger. And delivered a wallop of value to something he cared a lot about.

Let me back up a bit and offer some context…. I use Gmail for a lot of things: tracking discussions, keeping threaded conversations, creating and working on to-do lists, project management. I chose Gmail over some other GTD-type solutions because a) it has email built in (duh) and b) has a handy labeling feature for placing any particular conversation or thread into many different buckets, so that I don’t have to orthogonally categorize everything.

One thing that is frustrating about using Gmail in this fashion is that there is no clean way to annotate a thread or conversation or group of conversations. My big use case: as the eVoting SIG lead for the AST, I needed a convenient way to track and comment on the various threads of conversation that the IEEE working group was doing, as well as the internal work of the SIG. Since I was using Gmail to read and respond to the various discussions, that seemed like the logical place to annotate the conversation. I could write an email and pop a label on it, but once I send it it’s not editable, and if I don’t send it it clutters up my Drafts folder. Enter Gmail Notes.

A small group of developers decided that they had the same problem, and wrote a Greasemonkey script that used the Gmail API and the Gmail Apps capabilities to offer a note-taking capability, linked to a conversation or a label. This was just super for me, since I could finally do what I’d been unable to before, write an updatable, private note linked to a Gmail thread that would never get lost in the shuffle of a huge conversation. I installed the script, began working on it, and found a few bugs. I reported them back to the lead developer, John, and offered my testing services as part of his team.

Then in December, Gmail pulled support for the part of the API that supported Gmail Notes. So John had to punt. He switched over to using Jetpack instead of Greasemonkey, and came up with another method of identifying the conversation or label. I was called in to give it a spin, and had a hard time configuring it on my system. That’s when I set up an evening Skype session with John.

We used the screen sharing utility on Skype so that John could see my desktop (though not my cursor! Hey Skype, we testers need to see each other’s cursors!), and worked on setting up the Jetpack script correctly on my system. Then I led him through an exploratory test session, working all the functionality through a wide variety of states, and adjusting on the fly to the risks and issues that came up. We noted six separate issues, one of which appeared to happen only on the Mac I was using.

When we were done, I asked him how he liked pair testing like this. He was very pleased with the thoroughness of the testing and the flexibility of the format. He also said he’d be open to testing with a meetup group.

Testaphor Improv

Geordie Keitt — Thu, 07 Jan 2010 16:21:10 +0000

I love analogies – as comedy.

Eddie Izzard (a brilliant British standup comic) does a bit about Anglican ministers who try so hard to be hip and edgy and bring real life into the church. A close paraphrase: “Today’s reading comes from this magazine I found in a hedge. Cosmopolitan, Volume 224, page 58: ‘Lipstick colors this spring will be in the dusky pink area, with eye shadow to match.’ And this rather reminds me … of our Lord Jesus. Because you know, before he rode into Jerusalem on the donkey, he must have got tarted up a bit. Now let us sing hymn 399, ‘O Lord, What is My Hairdo All About?’”

So I have in mind a series called “And this rather reminds me … of Testing.” It might really work as improv – have people toss out ideas and have a panel riff on how it reminds us of testing.

Moderator: “I need an idea for a food item. You sir.”
Audience member: “Mayonnaise!”
Panelist A: “Ah, yes, mayonnaise. One places eggs in a bowl with a little lemon juice, then whisks as one slowly drizzles oil into it. The result is creamy, delicious, and shockingly unhealthful. And this rather reminds me … of Testing. I mean, who besides another mayonnaise maker even knows what a mayonnaise maker does? As far as most consumers of mayonnaise are concerned, it just comes in a jar.”
Panelist B: “Yes – just like testing! And, there are many flavors of mayonnaise.”
Panelist A: “Just like testing! In fact, I would go so far as to say we have transcended analogy and arrived in the realm of metaphor. Testing Is Mayonnaise. Can I get a hell yeah?”

I would find that funny.

Playing the dice game with developers

Geordie Keitt — Thu, 03 Sep 2009 04:44:34 +0000

At CAST2009, I took James Bach’s “Teach Yourself Testing” course. Part of his course was to have the testers play a deduction game with a variety of dice. The testers present a set of dice to the instructor, who responds verbally. The object is to figure out what the instructor’s response is going to be, and write down the algorithm that you deduce is being used to generate the response. The rules are you have to present a set of 5 dice, which you have rolled so their values are random.

I was paired up with another tester. Paul Holland, acting as Jim’s assistant, gave us our pack of dice. There were regular old 6-sided dice with dots, and also 4-, 6-, 12-, and 20-sided dice with Arabic numerals, and 6-sided dice with symbols such as exclamation point, spiral, explosion, skull, flower. One likely glowed in the dark.

My partner and I grabbed a random handful of dice, and rolled them, and asked Paul for his response. “Red blinking zero,” he said. We rolled again. “Red blinking zero,” he said. We rolled another set of dice. “Red blinking zero,” he said. Slowly, I began to detect a pattern…

My partner and I spent the next 30 minutes or so trying all kinds of ways to get Paul to say something other than “Red blinking zero.” We knew it was possible because he would offer other responses to other teams, but we could not find a combination of dice that led us out of that particular leg of the algorithm into some other response. I got pretty peeved and cussed at Paul a couple of times (if you read this, Paul, I’m sorry about that).

Finally, after a couple of hints from Paul, we hit on a strategy that started to give us some traction – though to do so we had to break the rules of the game – and within another 5 minutes had worked out the algorithm to Paul’s satisfaction. My partner and I gave the others a little retrospective on the game and our path to the solution, and decided that though it had started out frustrating it had ended up fun, and so it was a good mirror of our normal experience with testing.

******************

I was pretty excited about the dice game. It exercises some important testing skills: generating hypotheses, surfacing assumptions, gathering data, carefully keeping records of variables and outcomes, deducing cause and effect, putting abstract concepts into words, and perhaps most important, questioning the premise. I thought it would be cool to try it out with some coworkers. First I approached my testing teammate Anoop and explained the rules and the object of the game. He spent about 20 minutes with me, going down some of the same deadends that I did, and finally hit on the correct algorithm. He thought it was fun.

Next I asked a developer to play. I explained the rules and the object of the game to him, and he started showing me sets of dice. Within 2 minutes he quit. “I’m a coder. I don’t guess what the code is, I read the code.” He was polite enough to stick around to find out what the algorithm was, and to hear why I thought it was a useful testing game, but I could tell his heart wasn’t in it. He simply didn’t care to guess.

Then I asked another coder to play. He told me he had a couple minutes, so I set up the dice and told him the rules and the object. In response to his first couple sets I told him “zero”. The third time he rolled the dice I said, “Red blinking zero.” He immediately quit, throwing up his hands into the air and exclaiming, “What kind of crazy game is this? What other responses are in this algorithm, purple… people-eating… zeroes? And what’s the point of having rules if all they do is make it impossible to win?” He was pretty worked up (I’m pretty sure he mentioned sunspots as a possible variable in the algorithm, a good indicator of his worked-up-ness). We talked it over for a while, and the gist of his argument was that the puzzle was just way too hard for the couple minutes he had to devote to it. (To recap: He quit the game because he could see that it was a pretty open-ended problem that he did not want to devote the time it would take since he only had a couple minutes, then we immediately had a twenty-minute discussion on why the game would have taken too long. People are fascinating.)

He compared the dice game to the children’s game “Aunt Bertie Fish”, the object of which is to figure out what category of things Aunt Bertie Fish loves by being told a litany of things she loves and things she hates. An example I just made up: “Aunt Bertie Fish loves apples and hates bananas. Aunt Bertie Fish loves stop signs and hates street signs. Aunt Bertie Fish loves Boston baseball and hates Chicago baseball.” I’m outlining a pair of mutually-exclusive sets, X and Not-X, using tropes and examples which are common in my cultural context. In this case, Aunt Bertie Fish loves things that are red and hates anything that is not red.

According to my colleague, the dice game is like Aunt Bertie Fish except without the X. All he is getting is not-X, in which case X could be anything. Or else he is getting X and Y and Z, rather than a simple X – Not-X universe. It is impossible to bound the game using logic a priori, and it is in fact kind of unfair that the instructor does not offer the complete set of possible responses. And don’t get him started on how wrong it is to set up rules that have to be broken to succeed at the game. If someone played Aunt Bertie Fish with him that way, it wouldn’t be any fun at all.

******************

Slowly, I began to detect a pattern. Tester plays dice game: works through it, thinks it is fun, and recognizes it is relevant to day-to-day testing work. Developer plays dice game: gets upset or bored, quits.

It’s tempting to write off the first developer as a bad tester, but actually he is a fabulous tester who writes code in which it is very difficult to find problems.

It’s tempting to feel insulted that the second developer thought the dice game wasn’t any fun, but he has obvious respect for the tester’s willingness to jump into the game, sink or swim. My theory: this developer has a very healthy respect for the vastness of the testing problem space, and he hopes that games like this do not teach testers to be glib in the face of it, and to jump to conclusions too quickly, just because you figured out /this/ algorithm doesn’t mean that the /next one/ won’t depend on sunspots, and can never be solved.

How do your colleagues react to the dice game?

******************

Epilogue: I also played the game with my wife Sarah (who evidently loves me very much). Her advice on using the game to introduce testing to non-testers, was to take it slow. Start with a super-simple algorithm and 2 dice, and no “rules”. Then play again and incrementally add complexity: harder algorithms, more dice, more types of dice, various “rules”. If anyone has an opportunity to try this tactic on a set of interested non-testers, I’d like to get your thoughts.

CAST2009: “Rapid Tester” video

Geordie Keitt — Sat, 29 Aug 2009 03:48:17 +0000

View the post to see the video…

Another of Becky Fiedler’s videos from CAST 2009 in CO Springs. When “Feature Creep” was over, I had enough time in my Lightning Talk slot for an encore. Jim Bach wanted to hear “Rapid Tester”, so I played it.

CAST2009: “Black Box” video

Geordie Keitt — Sat, 29 Aug 2009 03:36:29 +0000

View the post to see the video…

Becky Fiedler shot this performance of “Black Box” after dinner was all over and we were just hanging out in the hallway. It pretty quickly turned into a Jonathan Coulton love-fest after the testing songs were played…

CAST2009: “Feature Creep” song & video

Geordie Keitt — Fri, 28 Aug 2009 19:11:24 +0000

View the post to see the video…

I performed this a month or so ago at CAST2009 in Colorado Springs, where Becky Fiedler recorded it. The intro to this song went, “This song is written from the perspective of a piece of bloatware that used to be sleek and clean.” I did this during a Lightning Talks session, meaning I had to bring it in under 4 minutes. That’s why there’s little time for dramatic pauses…

Talkin’ eVoting at CAST 2009

Geordie Keitt — Thu, 02 Jul 2009 17:00:24 +0000

Jim Nilius and I will be taking our act on the road again, this time to CAST2009. Our discussion at WREST 2 in Indianapolis on why regulated testing does a poor job of testing software produced a lot of great ideas for how to clarify the problems with using 19th-Century test methods to evaluate computerized voting systems.

Jim and I are preparing a strong argument for the Election Assistance Commission to overhaul the NIST certification standards to help certification labs handle testing complex software systems much more congruently, and we are taking this argument for a test drive at CAST. So come on over and test it, test us, rip it to shreds. We will rebuild it. We have the technology, we have the capability to make it better than it was before. Better, stronger, faster…