Displaying sound judgment in a complex, dynamic environment is a hallmark of wisdom.
-From the Wikipedia entry for “Sapience”
At WREST 2 in Indianapolis on May 14, I interviewed Jim Nilius, former Director of Testing at Systest Labs, a very tightly regulated shop in Denver CO that tests electronic voting systems. We discussed the applicability of the theme of the workshop, “Beyond Scripted Testing”, to the line of work his lab did. In his view, there is no space for unscripted testing in a regulated environment, simply because unless the tests are scripted they cannot be reviewed prior to being run, and if they cannot be reviewed then you cannot maintain your accreditation and your company will cease to exist.
I am not familiar with this line of thinking, but it got a lot of nods from around the table so it is probably a true dynamic. Which seems pretty scary and counterproductive, because it entirely stifles innovation and sapience in testing and drives it underground into what Jonathan Kohl refers to as a “shadow process“. I drew a system diagram that I believe represents the general state of a regulated testing shop.
Regulated Testing PNG
(Here is the PDF. Contact me via gmail for the original ODP, editable in OpenOffice, if you wish to modify or upgrade this document under the publishing rules of WREST.)
Note the small shaded section in the middle which depicts rapid cycling between test creation, test execution, and test reporting in the process of testing and learning about the system. This dynamic allows the tester’s brain to work. Disrupting this cycle disrupts the tester’s systemic awareness and renders testing infinitely less effective.
The term “sapience” has a special place in the heart of a context-driven-school tester like myself, given James Bach’s 2007 coinage of the term “sapient process” to mean “any process that relies on skilled humans” to draw a contrast between skilled testing by people vs. automated testing by computers. Many testers extend the contrast to include scripted testing of just about any kind, whether performed by people or computers. I would like to extend the contrast to include any testing that does not adapt to new information. As in the opening block-quote, a sapient tester displays good judgment in a complex, dynamic environment. To do this, a tester has to model the complexities of the system in her mind and keep that model alive as she interacts with the system. When information arrives that contradicts the model, she must adjust the model and question the validity of the adjustment – was that a bug I just saw, or was it a hint of a bug, or a problem with my understanding of the requirement, or…? To find out, she devises a new series of tests and runs them, and then adapts again to the new information. This is the essence of the learning cycle depicted in the diagram.
It doesn’t take much to interrupt the cognitive flow and bring the systems awareness of the tester tumbling down. Without a living system model in her mind she is comparatively non-sapient with respect to the system, and any testing activities she undertakes will be insensitive to new information, thus incapable of adapting to it, and thus (by my definition) her testing is not sapient. In other words, rote, mechanical, all the qualities we ascribe to poor testing.
In a regulated environment, sapient testing is not simply discouraged, it is extremely risky. Please refer to the diagram: every step in the testing cycle can initiate a process that threatens the existence of the testing organization. In a certification lab, such as for electronic voting systems, the following dynamics apply at a minimum:
So the company risks its very existence whenever it engages in testing, even though testing is what the company was chartered to do, and the only method it has for minimizing risk is to minimize testing! It does so by imposing “process” on testing such that every act of test creation is followed by an expensive validation and record-keeping procedure, rather than by test execution. And of course vice versa.
Under this regime the testers have no means by which to develop system awareness and sapience, except to divide their testing into “official process” testing and “shadow process” testing. In the shadow process, the testers can test. But the “shadow tester” must be very judicious about how to bring what she learns back into the “official process” world without risking the existence of the company.
I would appreciate learning your thoughts on this.
Great summary Geordie. I think the interview format of yours and Jim’s experience report was a microcosm of the dialog that needs to happen between the regulated and non-regulated testing world.
I like how the diagram turned out. I have a question though. Is the circle meant to show a “fence” around the sapient testing processes within, or is it meant delineate between activities that support existence vs. those that support the Accreditation process?
[Geordie: Let's get our figures straight from our ground. The big circle represents the boundaries of the test lab corporation, our first big "figure". The activities that take place outside the circle are external to the corporation, and are "ground" in this diagram. Part of the "ground" is the External Accreditation Process which enables the corporation to exist. The label "External Accreditation Process" should apply strictly to the upper-left-most processes of test method validation and accreditation. It appears as though it applies to the entire "ground", which isn't true. There is another "ground" segment on the right of the circle, which could be labeled "Vendor Contracting Process". Do you have a suggestion on how to better delineate this?
Inside the test lab circle there is a smaller shaded circle which represents the shadow realm where testing gets done in a flowing, sapient fashion.
I'm not sure what you mean by "activities that support existence." It seems to me that most activities put existence at risk!]
Okay, I think I’m getting the picture now. My first instinct with these type of diagrams is to assume it is a Venn type, so I’m looking for the overlapping circles, and didn’t see a circle for the External Accreditation process. The gray color of realtime/free was so light on my monitor I didn’t notice until you pointed it out, so I see that it is the sapient part of the diagram.
[Geordie: thanks for the feedback John. The diagram is meant to be interpreted as a Weinberg-style systems diagram. I'm not sure how to make that clear, but I'm updating the post to that effect. I'll post here again when I've updated the image with darker shading and some more figure-ground definition.]
[Geordie again: new diagram is up.]
Cool, I like the new diagram. I don’t see the link to the higer-res png anymore though.
[Geordie: Thanks, and the PDF is back up.]