Verification and Exploration

There’s a move towards separating what testers do into two categories, called by some “testing” and “checking”.  I don’t particularly like the terminology (or most attempts to redefine words), but I appreciate the distinction, as well as the reaching towards semantic clarity.

The distinction being that there are two unique activities that are called testing.

The first is exemplified by unit tests, but also includes the functional testing done to “assure” requirements are met — usually represented on spreadsheets with lots of steps and expected results, as well as regression testing (which those typically become.)

The second is exemplified by exploratory testing, but also includes performance testing, security penetration testing (though hopefully, a standard set of security tests falls under the first category), and usability tests.

This isn’t a clear separation, and some amount of each category can fall into any type of testing.

The distinguishing characteristic is whether the expected result of the test is known at the outset or not.  Or rather, whether the result is assumed.  The object of the result is the distinguishing factor — whether the object is to learn something new or verify something already known is correct.

This is where I find the appellations of “testing” and “checking” imprecise.  While you can clearly distinguish which category a test falls into, the labels are not obvious and must be learned – although it’s clear enough to categorize them when given the labels.

  • A test, for instance, at school is a verification of a student understanding requirements.
  • A test of strength or will is a challenge to find it’s capacity.
  • A scientific experiment can be considered a test, but isn’t typically called so
  • A check is what you do to the test results at school
  • A check is also a boundary limiting what can be done

I think better terms would be “verification” and “exploration”.  It is clear with verification that you are “checking” for expected behavior.  And it is also clear with exploration that you are “testing” unknown factors.

Both are important to QA, and being aware of the two categories, regardless of the terminology, helps you decide how the focus of testing should be balanced.  In different situations, more of one type or the other may be advantageous.  For instance, a startup with an unproven product might want to do more exploratory testing; but a mature product with clear requirements might want to do more verification.  The point being that neither should neglected.

Advertisements

11 thoughts on “Verification and Exploration

  1. Hi, Aaron…

    I appreciate you continuing the discussion.

    Checking is exemplified by jUnit tests that are no longer in figure, but have fallen into ground. While a programmer is in the process of interacting with product, those jUnit thingies are tests. When the programmer is no longer paying attention to them, letting the green bar reassure him that everything is still okay, they become checks.

    The separation between a test and a check is, to me, more clear than you might think. As I point out in http://www.developsense.com/2009/09/transpection-and-three-elements-of.html, a check has these three attributes: it involves an observation; the observation is linked to a decision rule; and the observation and evaluation of the rule can be made non-sapiently, by a machine or an unthinking human without exercise of judgement. The test (and it is a test, because it can’t be decided by a machine) of whether something is a check or a test lies in asking: 1) Is there an observation being made? 2) Is there a decision rule linked to that observation? 3) Could the determination be made by a machine? If the answer to all three questions is Yes, then you’re looking at a check; if not, it’s a test, or neither a test nor a check.

    The presence of an expected result is a distinguishing characteristic, but to me that’s not the most important thing. The most important thing is the presence or absence of sapience, human judgement, in the evaluation. The risk is in assuming that checks yield us sufficient information to proceed without further consideration.

    I’m bemused by the fact that people say “I don’t like attempts to redefine words,” and then immediately attempt to redefine mine. There’s nothing wrong with that; time and people will sort out the labels. I don’t see how “exploration” and “verification” add precision. Indeed, the apparently eternal debate over the meanings of verification and validation suggest a minefield.

    I agree wholeheartedly with your concluding paragraph. Again, thanks for your reflections on the subject.

    Cheers,

    —Michael B.

    • Michael-

      It’s a good checklist, but I’m not redefining your words. “Test” and “Check” are real words with existing meanings, unlike “transvestigate” or whatever it is, which you are free to define however you like — though most people will assume an amalgamation of the meaning of two root words, which two, you don’t control.

      I agree that junit tests can be written exploratorily (?), but that most aren’t. I was generalizing.

      I also agree that an expected result is distinguishing but not defining. My disagreement is that your terms are not intuitive.

      I would also disagree that “can’t be decided by a machine” is a distinguishing factor, because we can’t determine what can be decided by a machine. Not that I’m imbuing machines with sapience, but that the exercise of determining determinism is not deterministic, or at least not worth the effort. And “not worth the effort” is really the point at whether a test can be performed by a machine, not “sapience”.

  2. Aaron,

    Fabulous commentary! While our preferred “lables” are not the same, I read your post here and find that not to matter. What matters (to me) is that you understand the reason that there is a discussion on the topic in the first place, that you are able to apply the concept to your own experiences, and that you are willing to share your thoughts with the world.

    Personally, I don’t care if we (i.e. the entire, scattered, disorganized, so-called “software testing industry”) ever agree on terms. I do care that those who call themselves testers understand certain concepts. This is one of them.

    Cheers,
    Scott

    Scott Barber
    President & Chief Technologist, PerfTestPlus, Inc.
    http://www.perftestplus.com
    sbarber@perftestplus.com

    Vice President & Executive Director, Association for Software Testing
    http://www.associationforsoftwaretesting.org
    executive.director@associationforsoftwaretesting.org

    “If you can see it in your mind…
    you will find it in your life.”

    • Scott-

      Thanks for the comment. I look forward to the day when the software testing industry is a bit more organized, but not to when all terms become standardized. My relabeling was an exercise in understanding, and at least with the case of Michael’s comment above, discovered that my understanding of his concepts was incomplete.

      I look forward to participating in further discussions with you.

      -Aaron

  3. I added a comment to my own blog post on the topic pointing folks here (http://www.testingreflections.com/node/view/8282). I hope it gets you some “blog-love”.

    Cheers,

    Scott Barber
    President & Chief Technologist, PerfTestPlus, Inc.
    http://www.perftestplus.com
    sbarber@perftestplus.com

    Vice President & Executive Director, Association for Software Testing
    http://www.associationforsoftwaretesting.org
    executive.director@associationforsoftwaretesting.org

    “If you can see it in your mind…
    you will find it in your life.”

    • Thanks. I’ve gotten more “blog love” from this post than any other. Two people with multiple comments! Of course, it might be picking something easy to argue about and then twitter spamming real people.

  4. See, that’s what I like about this whole business: relabelling does afford us the opportunity to de-lump (that’s a neologism too, for “extract ourselves from assimilation bias) and understand something better. That’s the motivation behind argumentation (distinct from arguing).

    “Can’t be decided by a machine” is a problem that we’re famiilar with, through Godel and Turing, and to some degree, Popper. You’re absolutely correct that determining determinism isn’t deterministic, but deciding that something can be checked by a machine is; we make the decision heuristically; we declare that it can be checked by the machine.

    When you say “not worth the effort”, I want to make sure that you and I aren’t working at cross-purposes… so, which effort are you referring to? The effort of creating and programming a check, the act of performing a check, the effort to decide whether something should be a check or not… or something else?

    Cheers,

    —Michael B.

    • I meant “worth the effort” applying to both implementing a mechanical decision and determining whether a mechanistic implementation is feasible. “Feasible” being an analogy for “worth the effort” of course. Which of course, is itself a heuristic. I’m getting dizzy.

    • Language isn’t as fluid as people want it to be. The King James Bible is still perfectly clear (syntactically), and part of the problem with imprecise language is lack of education with people. By “education” I don’t mean pedantry, but understanding and experience.

      Rather, I’d say that language isn’t perfect, and precision, rather than fluidity, is the problem.

      English has been “fixed” for longer than most other languages, while continuing to grow. Due in no small part to the KJV Bible and Shakespeare, however archaic they may seem to most people now. A body of standard works, as well as the dynamic usage enabled by the standardization is what brings value to the language. You can make the exact same observation with programming languages. But I don’t want to attract that kind of “blog lovin'” here.

      Regarding your post, I particularly like this quote:

      “I encourage you to consider the distinction, and to make it explicit when you can”

      I think we can both agree that being aware of the distinction is the key factor, the naming of which enables us to think about it. (I like DHH’s talk here http://bit.ly/1aBbel) Although it sounds like we are make two distinctions which might have mostly overlapping sets of cases. Exploring the differences and similarities could be interesting, though I don’t know yet if it will be worthwhile.

  5. Pingback: Good programmers check, Great ones test as well « Happytesting

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s