Software Tasting --by Geordie Keitt » Geordie Keitt

I like Thinklinkr

Geordie Keitt — Fri, 21 May 2010 13:24:16 +0000

When I journal in the morning I use Thinklinkr. When I write a conference proposal I use Thinklinkr. When I compose a blog post I use Thinklinkr. It’s a simple online outliner, and I really love it. I can collaborate on an outline with another Thinklinkr user. I can keep multiple outlines, and copy one outline and use it as the basis for another. I wish I could write something in one outline and include a reference to it in another. But I’ve told them that, and maybe they’ll have that feature next week. It’s a young tool and growing fast.

Full disclosure: I am giving Thinklinkr the link to this post and in return I expect to get a free upgrade to a Pro account. But that only affects the timing of this post, not the content. I’m at the Writing About Testing conference in Durango today and I’ll likely demo Thinklinkr to the other writers. I’m happy to be a known user and I recommend at least giving it a whirl.

Check it out.

Counting on Friends

Geordie Keitt — Wed, 31 Mar 2010 03:54:18 +0000

At ProChain we are developing the capability for our software to be installed on operating systems using languages other than English. Naturally, I am called upon to test this capability. I chose to test in several languages: French, for the latin alphabet, Russian for Cyrillic, Hebrew for the right-to-left orientation, and Chinese for the massive character set. Unfortunately I am a monolingual guy, which poses some issues. For instance, I have no idea how to run a Hebrew language Windows XP installer.

Which does not daunt me, because I have confidence in my network. I contacted a tester with whom I’ve corresponded on a software testing Google group who works in Israel, and asked him to help. My teammate Anoop emailed him screen shots when it was time to click the next button and he responded with a note on which to click. It seems to have worked fine. Now I owe him a solid, and will gladly help him when he needs me.

I the same vein I have several cousins who are fluent Russian speakers, and a teammate is French. I will need a lead on someone to help with the Chinese setup, but I’ll cross that bridge when I get there. I count on my friends to be there for me when I need them, and just so you know, I’m here for y’all too.

Pair testing Gmail Notes

Geordie Keitt — Fri, 08 Jan 2010 16:18:09 +0000

This week I spent an hour testing with a complete stranger. And delivered a wallop of value to something he cared a lot about.

Let me back up a bit and offer some context…. I use Gmail for a lot of things: tracking discussions, keeping threaded conversations, creating and working on to-do lists, project management. I chose Gmail over some other GTD-type solutions because a) it has email built in (duh) and b) has a handy labeling feature for placing any particular conversation or thread into many different buckets, so that I don’t have to orthogonally categorize everything.

One thing that is frustrating about using Gmail in this fashion is that there is no clean way to annotate a thread or conversation or group of conversations. My big use case: as the eVoting SIG lead for the AST, I needed a convenient way to track and comment on the various threads of conversation that the IEEE working group was doing, as well as the internal work of the SIG. Since I was using Gmail to read and respond to the various discussions, that seemed like the logical place to annotate the conversation. I could write an email and pop a label on it, but once I send it it’s not editable, and if I don’t send it it clutters up my Drafts folder. Enter Gmail Notes.

A small group of developers decided that they had the same problem, and wrote a Greasemonkey script that used the Gmail API and the Gmail Apps capabilities to offer a note-taking capability, linked to a conversation or a label. This was just super for me, since I could finally do what I’d been unable to before, write an updatable, private note linked to a Gmail thread that would never get lost in the shuffle of a huge conversation. I installed the script, began working on it, and found a few bugs. I reported them back to the lead developer, John, and offered my testing services as part of his team.

Then in December, Gmail pulled support for the part of the API that supported Gmail Notes. So John had to punt. He switched over to using Jetpack instead of Greasemonkey, and came up with another method of identifying the conversation or label. I was called in to give it a spin, and had a hard time configuring it on my system. That’s when I set up an evening Skype session with John.

We used the screen sharing utility on Skype so that John could see my desktop (though not my cursor! Hey Skype, we testers need to see each other’s cursors!), and worked on setting up the Jetpack script correctly on my system. Then I led him through an exploratory test session, working all the functionality through a wide variety of states, and adjusting on the fly to the risks and issues that came up. We noted six separate issues, one of which appeared to happen only on the Mac I was using.

When we were done, I asked him how he liked pair testing like this. He was very pleased with the thoroughness of the testing and the flexibility of the format. He also said he’d be open to testing with a meetup group.

Testaphor Improv

Geordie Keitt — Thu, 07 Jan 2010 16:21:10 +0000

I love analogies – as comedy.

Eddie Izzard (a brilliant British standup comic) does a bit about Anglican ministers who try so hard to be hip and edgy and bring real life into the church. A close paraphrase: “Today’s reading comes from this magazine I found in a hedge. Cosmopolitan, Volume 224, page 58: ‘Lipstick colors this spring will be in the dusky pink area, with eye shadow to match.’ And this rather reminds me … of our Lord Jesus. Because you know, before he rode into Jerusalem on the donkey, he must have got tarted up a bit. Now let us sing hymn 399, ‘O Lord, What is My Hairdo All About?’”

So I have in mind a series called “And this rather reminds me … of Testing.” It might really work as improv – have people toss out ideas and have a panel riff on how it reminds us of testing.

Moderator: “I need an idea for a food item. You sir.”
Audience member: “Mayonnaise!”
Panelist A: “Ah, yes, mayonnaise. One places eggs in a bowl with a little lemon juice, then whisks as one slowly drizzles oil into it. The result is creamy, delicious, and shockingly unhealthful. And this rather reminds me … of Testing. I mean, who besides another mayonnaise maker even knows what a mayonnaise maker does? As far as most consumers of mayonnaise are concerned, it just comes in a jar.”
Panelist B: “Yes – just like testing! And, there are many flavors of mayonnaise.”
Panelist A: “Just like testing! In fact, I would go so far as to say we have transcended analogy and arrived in the realm of metaphor. Testing Is Mayonnaise. Can I get a hell yeah?”

I would find that funny.

Playing the dice game with developers

Geordie Keitt — Thu, 03 Sep 2009 04:44:34 +0000

At CAST2009, I took James Bach’s “Teach Yourself Testing” course. Part of his course was to have the testers play a deduction game with a variety of dice. The testers present a set of dice to the instructor, who responds verbally. The object is to figure out what the instructor’s response is going to be, and write down the algorithm that you deduce is being used to generate the response. The rules are you have to present a set of 5 dice, which you have rolled so their values are random.

I was paired up with another tester. Paul Holland, acting as Jim’s assistant, gave us our pack of dice. There were regular old 6-sided dice with dots, and also 4-, 6-, 12-, and 20-sided dice with Arabic numerals, and 6-sided dice with symbols such as exclamation point, spiral, explosion, skull, flower. One likely glowed in the dark.

My partner and I grabbed a random handful of dice, and rolled them, and asked Paul for his response. “Red blinking zero,” he said. We rolled again. “Red blinking zero,” he said. We rolled another set of dice. “Red blinking zero,” he said. Slowly, I began to detect a pattern…

My partner and I spent the next 30 minutes or so trying all kinds of ways to get Paul to say something other than “Red blinking zero.” We knew it was possible because he would offer other responses to other teams, but we could not find a combination of dice that led us out of that particular leg of the algorithm into some other response. I got pretty peeved and cussed at Paul a couple of times (if you read this, Paul, I’m sorry about that).

Finally, after a couple of hints from Paul, we hit on a strategy that started to give us some traction – though to do so we had to break the rules of the game – and within another 5 minutes had worked out the algorithm to Paul’s satisfaction. My partner and I gave the others a little retrospective on the game and our path to the solution, and decided that though it had started out frustrating it had ended up fun, and so it was a good mirror of our normal experience with testing.

******************

I was pretty excited about the dice game. It exercises some important testing skills: generating hypotheses, surfacing assumptions, gathering data, carefully keeping records of variables and outcomes, deducing cause and effect, putting abstract concepts into words, and perhaps most important, questioning the premise. I thought it would be cool to try it out with some coworkers. First I approached my testing teammate Anoop and explained the rules and the object of the game. He spent about 20 minutes with me, going down some of the same deadends that I did, and finally hit on the correct algorithm. He thought it was fun.

Next I asked a developer to play. I explained the rules and the object of the game to him, and he started showing me sets of dice. Within 2 minutes he quit. “I’m a coder. I don’t guess what the code is, I read the code.” He was polite enough to stick around to find out what the algorithm was, and to hear why I thought it was a useful testing game, but I could tell his heart wasn’t in it. He simply didn’t care to guess.

Then I asked another coder to play. He told me he had a couple minutes, so I set up the dice and told him the rules and the object. In response to his first couple sets I told him “zero”. The third time he rolled the dice I said, “Red blinking zero.” He immediately quit, throwing up his hands into the air and exclaiming, “What kind of crazy game is this? What other responses are in this algorithm, purple… people-eating… zeroes? And what’s the point of having rules if all they do is make it impossible to win?” He was pretty worked up (I’m pretty sure he mentioned sunspots as a possible variable in the algorithm, a good indicator of his worked-up-ness). We talked it over for a while, and the gist of his argument was that the puzzle was just way too hard for the couple minutes he had to devote to it. (To recap: He quit the game because he could see that it was a pretty open-ended problem that he did not want to devote the time it would take since he only had a couple minutes, then we immediately had a twenty-minute discussion on why the game would have taken too long. People are fascinating.)

He compared the dice game to the children’s game “Aunt Bertie Fish”, the object of which is to figure out what category of things Aunt Bertie Fish loves by being told a litany of things she loves and things she hates. An example I just made up: “Aunt Bertie Fish loves apples and hates bananas. Aunt Bertie Fish loves stop signs and hates street signs. Aunt Bertie Fish loves Boston baseball and hates Chicago baseball.” I’m outlining a pair of mutually-exclusive sets, X and Not-X, using tropes and examples which are common in my cultural context. In this case, Aunt Bertie Fish loves things that are red and hates anything that is not red.

According to my colleague, the dice game is like Aunt Bertie Fish except without the X. All he is getting is not-X, in which case X could be anything. Or else he is getting X and Y and Z, rather than a simple X – Not-X universe. It is impossible to bound the game using logic a priori, and it is in fact kind of unfair that the instructor does not offer the complete set of possible responses. And don’t get him started on how wrong it is to set up rules that have to be broken to succeed at the game. If someone played Aunt Bertie Fish with him that way, it wouldn’t be any fun at all.

******************

Slowly, I began to detect a pattern. Tester plays dice game: works through it, thinks it is fun, and recognizes it is relevant to day-to-day testing work. Developer plays dice game: gets upset or bored, quits.

It’s tempting to write off the first developer as a bad tester, but actually he is a fabulous tester who writes code in which it is very difficult to find problems.

It’s tempting to feel insulted that the second developer thought the dice game wasn’t any fun, but he has obvious respect for the tester’s willingness to jump into the game, sink or swim. My theory: this developer has a very healthy respect for the vastness of the testing problem space, and he hopes that games like this do not teach testers to be glib in the face of it, and to jump to conclusions too quickly, just because you figured out /this/ algorithm doesn’t mean that the /next one/ won’t depend on sunspots, and can never be solved.

How do your colleagues react to the dice game?

******************

Epilogue: I also played the game with my wife Sarah (who evidently loves me very much). Her advice on using the game to introduce testing to non-testers, was to take it slow. Start with a super-simple algorithm and 2 dice, and no “rules”. Then play again and incrementally add complexity: harder algorithms, more dice, more types of dice, various “rules”. If anyone has an opportunity to try this tactic on a set of interested non-testers, I’d like to get your thoughts.

CAST2009: “Rapid Tester” video

Geordie Keitt — Sat, 29 Aug 2009 03:48:17 +0000

View the post to see the video…

Another of Becky Fiedler’s videos from CAST 2009 in CO Springs. When “Feature Creep” was over, I had enough time in my Lightning Talk slot for an encore. Jim Bach wanted to hear “Rapid Tester”, so I played it.

CAST2009: “Black Box” video

Geordie Keitt — Sat, 29 Aug 2009 03:36:29 +0000

View the post to see the video…

Becky Fiedler shot this performance of “Black Box” after dinner was all over and we were just hanging out in the hallway. It pretty quickly turned into a Jonathan Coulton love-fest after the testing songs were played…

CAST2009: “Feature Creep” song & video

Geordie Keitt — Fri, 28 Aug 2009 19:11:24 +0000

View the post to see the video…

I performed this a month or so ago at CAST2009 in Colorado Springs, where Becky Fiedler recorded it. The intro to this song went, “This song is written from the perspective of a piece of bloatware that used to be sleek and clean.” I did this during a Lightning Talks session, meaning I had to bring it in under 4 minutes. That’s why there’s little time for dramatic pauses…

Talkin’ eVoting at CAST 2009

Geordie Keitt — Thu, 02 Jul 2009 17:00:24 +0000

Jim Nilius and I will be taking our act on the road again, this time to CAST2009. Our discussion at WREST 2 in Indianapolis on why regulated testing does a poor job of testing software produced a lot of great ideas for how to clarify the problems with using 19th-Century test methods to evaluate computerized voting systems.

Jim and I are preparing a strong argument for the Election Assistance Commission to overhaul the NIST certification standards to help certification labs handle testing complex software systems much more congruently, and we are taking this argument for a test drive at CAST. So come on over and test it, test us, rip it to shreds. We will rebuild it. We have the technology, we have the capability to make it better than it was before. Better, stronger, faster…

WREST 2: Regulation Disrupts Sapience

Geordie Keitt — Fri, 05 Jun 2009 03:30:26 +0000

Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.

-From the Wikipedia entry for “Sapience”

At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.

I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.

Regulated Testing PNG

(Here is the PDF. Contact me via gmail for the original ODP, editable in OpenOffice, if you wish to modify or upgrade this document under the publishing rules of WREST.)

Note the small shaded section in the middle which depicts rapid cycling between test creation, test execution, and test reporting in the process of testing and learning about the system. This dynamic allows the tester’s brain to work. Disrupting this cycle disrupts the tester’s systemic awareness and renders testing infinitely less effective.

The term “sapience” has a special place in the heart of a context-driven-school tester like myself, given James Bach’s 2007 coinage of the term “sapient process” to mean “any process that relies on skilled humans” to draw a contrast between skilled testing by people vs. automated testing by computers. Many testers extend the contrast to include scripted testing of just about any kind, whether performed by people or computers. I would like to extend the contrast to include any testing that does not adapt to new information. As in the opening block-quote, a sapient tester displays good judgment in a complex, dynamic environment. To do this, a tester has to model the complexities of the system in her mind and keep that model alive as she interacts with the system. When information arrives that contradicts the model, she must adjust the model and question the validity of the adjustment – was that a bug I just saw, or was it a hint of a bug, or a problem with my understanding of the requirement, or…? To find out, she devises a new series of tests and runs them, and then adapts again to the new information. This is the essence of the learning cycle depicted in the diagram.

It doesn’t take much to interrupt the cognitive flow and bring the systems awareness of the tester tumbling down. Without a living system model in her mind she is comparatively non-sapient with respect to the system, and any testing activities she undertakes will be insensitive to new information, thus incapable of adapting to it, and thus (by my definition) her testing is not sapient. In other words, rote, mechanical, all the qualities we ascribe to poor testing.

In a regulated environment, sapient testing is not simply discouraged, it is extremely risky. Please refer to the diagram: every step in the testing cycle can initiate a process that threatens the existence of the testing organization. In a certification lab, such as for electronic voting systems, the following dynamics apply at a minimum:

Each new test that is created can potentially initiate a cycle of test method validation and re-accreditation of the regulated testing organization, which puts the viability of the company in jeopardy.

Every test report can potentially negatively influence the likelihood that the system will be certified, leading to less business with vendors who would prefer to work with a less picky test lab.

Every test executed can potentially fail an audit, resulting in expensive “process improvements” and possibly kicking off another round of test method validation and the risk of losing accreditation.

So the company risks its very existence whenever it engages in testing, even though testing is what the company was chartered to do, and the only method it has for minimizing risk is to minimize testing! It does so by imposing “process” on testing such that every act of test creation is followed by an expensive validation and record-keeping procedure, rather than by test execution. And of course vice versa.

Under this regime the testers have no means by which to develop system awareness and sapience, except to divide their testing into “official process” testing and “shadow process” testing. In the shadow process, the testers can test. But the “shadow tester” must be very judicious about how to bring what she learns back into the “official process” world without risking the existence of the company.

I would appreciate learning your thoughts on this.