Testing isn’t just for testers

Testing isn’t just for testers

Testers write tests all the time. But testing isn’t just for testers.

Actually, that would be like saying programming is for computers. Tests are written by testers, but it’s the value that other people get out of that most important.

NOTE: Testers are not computers. No matter how robotic they may seem to an outsider.

I can think of several groups that are involved in testing whose primary title is not “tester”

  • Developers
  • Operations
  • Product Owners
  • Designers
  • Project Managers
  • Executives
  • Customers

Developers, of course, write tests. At least they should. They write unit tests and component tests, and mock external systems to be able to write better isolated tests. Developers need to test their code, and even if they aren’t using a test framework to save their tests, they’re likely still testing, even if it’s just a using a debugger or print statements while writing the code initially. Those ad hoc tests can be easily be captured with a unit test framework (like JUnit, Mocha, RSpec, or PyTest). A couple of lines of boilerplate and changing that print(), console.log(), puts(), or System.out.println() to an assertion means that you can rerun that test whenever a change is made.

This ad hoc check using a print statement:

Contact contact = CRM.findContact("Aaron Evans");
System.out.println("contact: " + contact.email);

Becomes this unit test:

@Test
public void contactExists() {
  User user = CRM.findContact("Aaron Evans");
  assertThat(user.email).equals("aarone@one-shore.com");
}

But developers also use tests. They use tests written by other developers to make sure that their changes don’t affect other parts of the system. And they use tests written by testers to make sure that the functionality (or performance) of the application is what is expected.

System Administrators, DevOps, and Operations teams write tests too. They have to make sure the system as a whole stays working. And they write smoke tests to run after deployment to make sure the deployment succeeded and the system came back online.

Continuous Integration makes sure that the artifacts assembled by the operations team for deployment is consistent, persistent, idempotent, and integration tests make sure that those artifacts work together.

Operations teams can also leverage tests to make sure that production environments are up and running and monitor the system health in production.

Designers can use visual tests to see verify that functionality works on different displays, and usability tests can find out what features make sense or provide the most value.

A/B testing — apart from functional testing, can help find out which is the most effective way of presenting this.

Product Owners care about tests too. They define and execute acceptance tests to make sure that the software delivered is what they expected. This can be formal or informal — and like the unit tests written by developers, can be captured to facilitate repeated execution and avoid regressions.

Gherkin (or Cucumber) is one semi-formal syntax for capturing acceptance criteria as tests.

It can be useful either as documentation — for communication purposes with developers & testers, or mapped to executable code. A “user story” with acceptance criteria defined using Gherkin syntax might look like this:

As a salesperson
I want to keep track of my contacts
so that I can email them later

Given a contact is added to the CRM (e.g. Aaron Evans)
When I search for that contact
Then I should be able to see their email (e.g. aarone@one-shore.com)

The Product Owner or Project Manager (and similar roles) should care about what’s being tested and be able to tell that from it’s functional and unit tests. A burn-down of defects (or burn-up? of features) can show how close the project is to being ready to ship. And they should perform ad hoc testing to make sure that testers don’t lose sight of the end goal and that obvious issues that are visible from a high altitude aren’t missed by those who are down in the thick of things.

The tests should inform the stakeholders of the state of the system, as well as document which features are working and what defects have been found (every defect should have a corresponding failing test.)

Executives can gain useful insight into the quality and completeness of a system from its tests. A dashboard showing a high level summary (and which can be drilled down to see details) can give them peace of mind (or a warning flag) before they go in to do a demo with investors, or help arm them with information when they go into a board meeting and want to know when feature X will be done, or if they’re ready to make a product announcement.

Tests also help them to be able to merge different systems, cut cost and migrate to new platforms as needed. Without tests, you may not have a good answer to questions like “What if…” or “When can…”

Finally, the Customer benefits in the end from testing. It goes without saying (but I’ll say it anyway) that testing improves quality. Or at least allows you to know where quality can be improved. And quality is the key differentiator.

If your customers are delighted with your products because they are generally bug free, they’re willing to forgive the occasional mistake that slips through, and can even provide you with feedback — if you provide a mechanism for them to report it — and then actually listen to your users.

No one can anticipate every way real users will use your software in the real world, and their experience and insight is invaluable. And a customer who encounters an issue, is able to tell you about it, and then sees it get fixed, is likely to become your biggest supporter, and recommend you to other customers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s