WREST 2: Regulation Disrupts Sapience

Geordie Keitt — Fri, 05 Jun 2009 03:30:26 +0000

Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.

-From the Wikipedia entry for “Sapience”

At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.

I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.

Regulated Testing PNG

(Here is the PDF. Contact me via gmail for the original ODP, editable in OpenOffice, if you wish to modify or upgrade this document under the publishing rules of WREST.)

Note the small shaded section in the middle which depicts rapid cycling between test creation, test execution, and test reporting in the process of testing and learning about the system. This dynamic allows the tester’s brain to work. Disrupting this cycle disrupts the tester’s systemic awareness and renders testing infinitely less effective.

The term “sapience” has a special place in the heart of a context-driven-school tester like myself, given James Bach’s 2007 coinage of the term “sapient process” to mean “any process that relies on skilled humans” to draw a contrast between skilled testing by people vs. automated testing by computers. Many testers extend the contrast to include scripted testing of just about any kind, whether performed by people or computers. I would like to extend the contrast to include any testing that does not adapt to new information. As in the opening block-quote, a sapient tester displays good judgment in a complex, dynamic environment. To do this, a tester has to model the complexities of the system in her mind and keep that model alive as she interacts with the system. When information arrives that contradicts the model, she must adjust the model and question the validity of the adjustment – was that a bug I just saw, or was it a hint of a bug, or a problem with my understanding of the requirement, or…? To find out, she devises a new series of tests and runs them, and then adapts again to the new information. This is the essence of the learning cycle depicted in the diagram.

It doesn’t take much to interrupt the cognitive flow and bring the systems awareness of the tester tumbling down. Without a living system model in her mind she is comparatively non-sapient with respect to the system, and any testing activities she undertakes will be insensitive to new information, thus incapable of adapting to it, and thus (by my definition) her testing is not sapient. In other words, rote, mechanical, all the qualities we ascribe to poor testing.

In a regulated environment, sapient testing is not simply discouraged, it is extremely risky. Please refer to the diagram: every step in the testing cycle can initiate a process that threatens the existence of the testing organization. In a certification lab, such as for electronic voting systems, the following dynamics apply at a minimum:

Each new test that is created can potentially initiate a cycle of test method validation and re-accreditation of the regulated testing organization, which puts the viability of the company in jeopardy.

Every test report can potentially negatively influence the likelihood that the system will be certified, leading to less business with vendors who would prefer to work with a less picky test lab.

Every test executed can potentially fail an audit, resulting in expensive “process improvements” and possibly kicking off another round of test method validation and the risk of losing accreditation.

So the company risks its very existence whenever it engages in testing, even though testing is what the company was chartered to do, and the only method it has for minimizing risk is to minimize testing! It does so by imposing “process” on testing such that every act of test creation is followed by an expensive validation and record-keeping procedure, rather than by test execution. And of course vice versa.

Under this regime the testers have no means by which to develop system awareness and sapience, except to divide their testing into “official process” testing and “shadow process” testing. In the shadow process, the testers can test. But the “shadow tester” must be very judicious about how to bring what she learns back into the “official process” world without risking the existence of the company.

I would appreciate learning your thoughts on this.

Software Tasting --by Geordie Keitt » regulated testing

WREST 2: Regulation Disrupts Sapience