Software Tasting –by Geordie Keitt

Jim Nilius and I will be taking our act on the road again, this time to CAST2009. Our discussion at WREST 2 in Indianapolis on why regulated testing does a poor job of testing software produced a lot of great ideas for how to clarify the problems with using 19th-Century test methods to evaluate computerized voting systems.

Jim and I are preparing a strong argument for the Election Assistance Commission to overhaul the NIST certification standards to help certification labs handle testing complex software systems much more congruently, and we are taking this argument for a test drive at CAST. So come on over and test it, test us, rip it to shreds. We will rebuild it. We have the technology, we have the capability to make it better than it was before. Better, stronger, faster…

Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.

-From the Wikipedia entry for “Sapience”

At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.

I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.

Regulated Testing PNG

Last Saturday night I played guitar, both solo and accompanying my friend Rob Balder, at a gig at Balticon, a sci-fi/fantasy/gaming convention in Baltimore. It was not one of my best performances. Rob and I did a new arrangement of a tune we had done before, and it never hit its stride, it felt slow and square. Another tune I sang well but my guitar work was kind of iffy, it was the first time I’d done it in front of an audience. (I did a fine job on one song, where all I did was play guitar instead of having to sing as well, and I just executed the song.)

I can pick out greater and lesser reasons for my struggles.

T’ai chi hurts to do. The benefits are enormous, and the cost is pain. The nice part is, the same deal applies to most things in life. If you want to be a great guitar player, your fingers will hurt. If you want to be a great athlete, your muscles will hurt. My teacher Chris Casey says, “There is no way out. But there is a way through.”

When I worked with James Bach on a Satisfice project he would ask me uncomfortable questions about what I was doing and why I was doing it. He’d ask, “Tell me what this bug is at its core.” I’d say, “When I click this button, I get an error message instead of the dialog I’m expecting.”

“Why do you expect a dialog?”

“Because that’s how the app is supposed to work.”

“How do you know?”

“It worked that way another time.”

“Ahh, so there is an inconsistency between how it worked before and the way it works now. What is different? How do you know that they are not both right?”

Next Page »