Software Tasting --by Geordie Keitt » Uncategorized

I like Thinklinkr

Geordie Keitt — Fri, 21 May 2010 13:24:16 +0000

When I journal in the morning I use Thinklinkr. When I write a conference proposal I use Thinklinkr. When I compose a blog post I use Thinklinkr. It’s a simple online outliner, and I really love it. I can collaborate on an outline with another Thinklinkr user. I can keep multiple outlines, and copy one outline and use it as the basis for another. I wish I could write something in one outline and include a reference to it in another. But I’ve told them that, and maybe they’ll have that feature next week. It’s a young tool and growing fast.

Full disclosure: I am giving Thinklinkr the link to this post and in return I expect to get a free upgrade to a Pro account. But that only affects the timing of this post, not the content. I’m at the Writing About Testing conference in Durango today and I’ll likely demo Thinklinkr to the other writers. I’m happy to be a known user and I recommend at least giving it a whirl.

Check it out.

Pair testing Gmail Notes

Geordie Keitt — Fri, 08 Jan 2010 16:18:09 +0000

This week I spent an hour testing with a complete stranger. And delivered a wallop of value to something he cared a lot about.

Let me back up a bit and offer some context…. I use Gmail for a lot of things: tracking discussions, keeping threaded conversations, creating and working on to-do lists, project management. I chose Gmail over some other GTD-type solutions because a) it has email built in (duh) and b) has a handy labeling feature for placing any particular conversation or thread into many different buckets, so that I don’t have to orthogonally categorize everything.

One thing that is frustrating about using Gmail in this fashion is that there is no clean way to annotate a thread or conversation or group of conversations. My big use case: as the eVoting SIG lead for the AST, I needed a convenient way to track and comment on the various threads of conversation that the IEEE working group was doing, as well as the internal work of the SIG. Since I was using Gmail to read and respond to the various discussions, that seemed like the logical place to annotate the conversation. I could write an email and pop a label on it, but once I send it it’s not editable, and if I don’t send it it clutters up my Drafts folder. Enter Gmail Notes.

A small group of developers decided that they had the same problem, and wrote a Greasemonkey script that used the Gmail API and the Gmail Apps capabilities to offer a note-taking capability, linked to a conversation or a label. This was just super for me, since I could finally do what I’d been unable to before, write an updatable, private note linked to a Gmail thread that would never get lost in the shuffle of a huge conversation. I installed the script, began working on it, and found a few bugs. I reported them back to the lead developer, John, and offered my testing services as part of his team.

Then in December, Gmail pulled support for the part of the API that supported Gmail Notes. So John had to punt. He switched over to using Jetpack instead of Greasemonkey, and came up with another method of identifying the conversation or label. I was called in to give it a spin, and had a hard time configuring it on my system. That’s when I set up an evening Skype session with John.

We used the screen sharing utility on Skype so that John could see my desktop (though not my cursor! Hey Skype, we testers need to see each other’s cursors!), and worked on setting up the Jetpack script correctly on my system. Then I led him through an exploratory test session, working all the functionality through a wide variety of states, and adjusting on the fly to the risks and issues that came up. We noted six separate issues, one of which appeared to happen only on the Mac I was using.

When we were done, I asked him how he liked pair testing like this. He was very pleased with the thoroughness of the testing and the flexibility of the format. He also said he’d be open to testing with a meetup group.

Playing the dice game with developers

Geordie Keitt — Thu, 03 Sep 2009 04:44:34 +0000

At CAST2009, I took James Bach’s “Teach Yourself Testing” course. Part of his course was to have the testers play a deduction game with a variety of dice. The testers present a set of dice to the instructor, who responds verbally. The object is to figure out what the instructor’s response is going to be, and write down the algorithm that you deduce is being used to generate the response. The rules are you have to present a set of 5 dice, which you have rolled so their values are random.

I was paired up with another tester. Paul Holland, acting as Jim’s assistant, gave us our pack of dice. There were regular old 6-sided dice with dots, and also 4-, 6-, 12-, and 20-sided dice with Arabic numerals, and 6-sided dice with symbols such as exclamation point, spiral, explosion, skull, flower. One likely glowed in the dark.

My partner and I grabbed a random handful of dice, and rolled them, and asked Paul for his response. “Red blinking zero,” he said. We rolled again. “Red blinking zero,” he said. We rolled another set of dice. “Red blinking zero,” he said. Slowly, I began to detect a pattern…

My partner and I spent the next 30 minutes or so trying all kinds of ways to get Paul to say something other than “Red blinking zero.” We knew it was possible because he would offer other responses to other teams, but we could not find a combination of dice that led us out of that particular leg of the algorithm into some other response. I got pretty peeved and cussed at Paul a couple of times (if you read this, Paul, I’m sorry about that).

Finally, after a couple of hints from Paul, we hit on a strategy that started to give us some traction – though to do so we had to break the rules of the game – and within another 5 minutes had worked out the algorithm to Paul’s satisfaction. My partner and I gave the others a little retrospective on the game and our path to the solution, and decided that though it had started out frustrating it had ended up fun, and so it was a good mirror of our normal experience with testing.

******************

I was pretty excited about the dice game. It exercises some important testing skills: generating hypotheses, surfacing assumptions, gathering data, carefully keeping records of variables and outcomes, deducing cause and effect, putting abstract concepts into words, and perhaps most important, questioning the premise. I thought it would be cool to try it out with some coworkers. First I approached my testing teammate Anoop and explained the rules and the object of the game. He spent about 20 minutes with me, going down some of the same deadends that I did, and finally hit on the correct algorithm. He thought it was fun.

Next I asked a developer to play. I explained the rules and the object of the game to him, and he started showing me sets of dice. Within 2 minutes he quit. “I’m a coder. I don’t guess what the code is, I read the code.” He was polite enough to stick around to find out what the algorithm was, and to hear why I thought it was a useful testing game, but I could tell his heart wasn’t in it. He simply didn’t care to guess.

Then I asked another coder to play. He told me he had a couple minutes, so I set up the dice and told him the rules and the object. In response to his first couple sets I told him “zero”. The third time he rolled the dice I said, “Red blinking zero.” He immediately quit, throwing up his hands into the air and exclaiming, “What kind of crazy game is this? What other responses are in this algorithm, purple… people-eating… zeroes? And what’s the point of having rules if all they do is make it impossible to win?” He was pretty worked up (I’m pretty sure he mentioned sunspots as a possible variable in the algorithm, a good indicator of his worked-up-ness). We talked it over for a while, and the gist of his argument was that the puzzle was just way too hard for the couple minutes he had to devote to it. (To recap: He quit the game because he could see that it was a pretty open-ended problem that he did not want to devote the time it would take since he only had a couple minutes, then we immediately had a twenty-minute discussion on why the game would have taken too long. People are fascinating.)

He compared the dice game to the children’s game “Aunt Bertie Fish”, the object of which is to figure out what category of things Aunt Bertie Fish loves by being told a litany of things she loves and things she hates. An example I just made up: “Aunt Bertie Fish loves apples and hates bananas. Aunt Bertie Fish loves stop signs and hates street signs. Aunt Bertie Fish loves Boston baseball and hates Chicago baseball.” I’m outlining a pair of mutually-exclusive sets, X and Not-X, using tropes and examples which are common in my cultural context. In this case, Aunt Bertie Fish loves things that are red and hates anything that is not red.

According to my colleague, the dice game is like Aunt Bertie Fish except without the X. All he is getting is not-X, in which case X could be anything. Or else he is getting X and Y and Z, rather than a simple X – Not-X universe. It is impossible to bound the game using logic a priori, and it is in fact kind of unfair that the instructor does not offer the complete set of possible responses. And don’t get him started on how wrong it is to set up rules that have to be broken to succeed at the game. If someone played Aunt Bertie Fish with him that way, it wouldn’t be any fun at all.

******************

Slowly, I began to detect a pattern. Tester plays dice game: works through it, thinks it is fun, and recognizes it is relevant to day-to-day testing work. Developer plays dice game: gets upset or bored, quits.

It’s tempting to write off the first developer as a bad tester, but actually he is a fabulous tester who writes code in which it is very difficult to find problems.

It’s tempting to feel insulted that the second developer thought the dice game wasn’t any fun, but he has obvious respect for the tester’s willingness to jump into the game, sink or swim. My theory: this developer has a very healthy respect for the vastness of the testing problem space, and he hopes that games like this do not teach testers to be glib in the face of it, and to jump to conclusions too quickly, just because you figured out /this/ algorithm doesn’t mean that the /next one/ won’t depend on sunspots, and can never be solved.

How do your colleagues react to the dice game?

******************

Epilogue: I also played the game with my wife Sarah (who evidently loves me very much). Her advice on using the game to introduce testing to non-testers, was to take it slow. Start with a super-simple algorithm and 2 dice, and no “rules”. Then play again and incrementally add complexity: harder algorithms, more dice, more types of dice, various “rules”. If anyone has an opportunity to try this tactic on a set of interested non-testers, I’d like to get your thoughts.

CAST2009: “Rapid Tester” video

Geordie Keitt — Sat, 29 Aug 2009 03:48:17 +0000

View the post to see the video…

Another of Becky Fiedler’s videos from CAST 2009 in CO Springs. When “Feature Creep” was over, I had enough time in my Lightning Talk slot for an encore. Jim Bach wanted to hear “Rapid Tester”, so I played it.

CAST2009: “Black Box” video

Geordie Keitt — Sat, 29 Aug 2009 03:36:29 +0000

View the post to see the video…

Becky Fiedler shot this performance of “Black Box” after dinner was all over and we were just hanging out in the hallway. It pretty quickly turned into a Jonathan Coulton love-fest after the testing songs were played…

Talkin’ eVoting at CAST 2009

Geordie Keitt — Thu, 02 Jul 2009 17:00:24 +0000

Jim Nilius and I will be taking our act on the road again, this time to CAST2009. Our discussion at WREST 2 in Indianapolis on why regulated testing does a poor job of testing software produced a lot of great ideas for how to clarify the problems with using 19th-Century test methods to evaluate computerized voting systems.

Jim and I are preparing a strong argument for the Election Assistance Commission to overhaul the NIST certification standards to help certification labs handle testing complex software systems much more congruently, and we are taking this argument for a test drive at CAST. So come on over and test it, test us, rip it to shreds. We will rebuild it. We have the technology, we have the capability to make it better than it was before. Better, stronger, faster…