The real value of test automation doesn’t come from finding bugs. At least not primarily. Automation is not good at that, except in one case which we’ll discuss later.
On a consultation with a client the other day I was talking about this.
Our conversation was interrupted when my daughter came running in, crying, and told me that my 5 year old son had been trampled by horses. Earlier that day, our neighbors horses had gotten into our yard and I’d held him up to pet a horse’s mane and show him that it wasn’t scary.
Later, while the two of them were playing in our yard, picking new spring leaves from an Aspen tree, something spooked the horses and they bolted. We were lucky (and blessed) that he only received scrapes and bruises and no serious injuries and he was fine after a couple days. My boy chafed at the restrictions placed on him, but my wife is still shaken.
It got me thinking about a couple things. First, that horses are big and powerful, and while not malicious, they can be dangerous. Second that fences make good neighbors. Perhaps not coincidentally, I’ve been meaning to put up a stronger fence (mainly to keep our goats and dog in.)
Although I welcomed the horses’ visit, I wasn’t prepared for consequences. And now I have to listen to my wife’s calls for caution more.
To trivialize the incident — and draw a tenuous parallel — tests are like fences. And like other fences, they need gates. Even with a strong fence, I would have let the horses in the yard. And even with good tests that catch bugs, you can still let bugs into production. You may do this intentionally, or not.
If you don’t trust your test results, if there are always “random” errors, if tests take too long to run, or are out of date, or don’t actually check what they purport to be checking, bugs can slip through, and the consequences can be severe.
But despite what you test, there are sometimes what you deem “acceptable risks” and there is always the unknown. You can’t test everything, especially what you don’t know. So you need to be able accept that testing isn’t the solution for preventing bugs.
So what is the value of testing? Is it just snake oil? Or is it good up to a point, and not beyond that.
My belief is that test automation does have value, but that finding bugs is not the primary value. My first posit is that test automation is good at preventing some bugs — often the easy obvious issues. Like a fence.
The act of writing a test, like planning, forces you to think about a problem — to think about problems in general in trying to anticipate problems, and that this itself is good at preventing — by avoiding bugs before they are even created.
Next, tests document features, and describe what they are intended to do, at least as understood by the author. But, like writing, writing tests forces you to articulate your understanding, and then allows you to communicate that understanding to others, who can then share in your understanding, correct you if it is mistaken, or expand upon it.
When you write a test, it should be written primarily to communicate — to communicate an assumption about the software that is being tested. A test forces you to describe *specifically* what that assumption is. If the assumption is wrong, it can be corrected, if it is correct, and the code is wrong, it can be fixed, and the tested again against that assumption.
By it’s existence, a test (especially an automated test) enables youo to repeat that process. And to objectively check if the software does what is expected. Or if it changes. That helps to prevent regressions.
A test verifies that requirements are met, or at least that features are exercised.
Knowing what features are being tested and which requirements (assumptions) are being made is the second powerful value of testing, and the first explicit benefit, after the implicit value (which should not be dismissed) of deliberately thinking about the problem.
Because tests are repeatable (even manual tests), it helps to prevent regressions. And that will speed up development velocity. Developers with good tests in place will need less time to spend reasoning about the impact of their changes on the system, and not have to worry about if a change here will affect a (seemingly) unrelated outcome over there.
Software systems very quickly become too large and complex for people to reason about all at once. Testing helps to break that down. And helps to make sure that you don’t forget a check.
Repetition of test execution, like anything else, means you should get better at it. Your tests will improve over time if they are exercised (and fixed) enough to become more robust, more precise, clearer, faster, etc. So the third key benefit of testing comes from repeatedly executing your tests.
The up front cost of creating a test is non-trivial. The benefit of executing it over time increases.
Getting back to my fence analogy, like building a fence, a lot of the initial work is in surveying the area, digging and setting the posts. This is most of the work in setting up a fence, and at the end of it, you just have a few posts in the ground, and it doesn’t prevent anything from getting through.
This is the situation I was in. In the real world, you have to spend time building infrastructure, defining architecture, and laying down the foundation upon which you will build your fence. This is also true in software, but a lot of the work can be done for you with a good test framework. But you don’t have to have a full framework in place to start writing tests that provide some value.
If I had the money, I could’ve rented a machine that pounds the posts in the ground instead of digging holes with a shovel and them filling them in by hand. A few seconds with a powerful hydraulic tool, and the post is set firmly in the ground.
A good foundational framework can also help with the setup of tests too. But it takes investment in knowing how to use it. But the best framework in the world won’t help you if you’re setting up your fence in the wrong place. I mean, if your tests aren’t covering the features you need to test most.
Test automation allows testers to focus on creative work, like actively finding bugs, instead of checking for regressions around the perimeter (or in the middle) of your system.
Writing a test that can execute again without you having to think about it or do anything, is a huge time saver. This is the third value of test automation. Once it’s written, it can provide residual value over and over again. As long as you keep executing it, and keep it maintained.
But test automation is notoriously hard to maintain. UI changes, for example, can break automation that makes assumptions about the user interface. Even if they were correct at one time.
So your goal when writing automation should be to make as few assumptions as necessary. Test only one thing. Don’t depend on the UI to validate something unless you have to. That way you spend less time fixing tests — or making sure that they are testing the right thing, and more time creating new tests, testing new features, and making sure that changes don’t negatively impact the desired (and expected) behavior of the system.
Good tests written this way will help when changes need to be made later. They can act as documentation for others when they need to understand the system to fix it or add to it. This could be a new developer coming onto the project, or it could be the same person, coming back to code they wrote — and knew was working correctly — days, months, or years later.
This is the fourth value of test automation, and possibly the biggest. Not only does it verify that the system continues to work as expected over time, but it allows you to be confident that when you make one change — it doesn’t change other parts of the system without your knowledge.
Test automation can help you find bugs that are introduced when changes are made to a working system that is well tested. Initially your tests helped you prevent bugs by thinking clearly and precisely about the system, anticipating bugs, and coding around the problem. Because you wrote tests, you documented the behavior and set up checks that you can perform repeatably, repeatedly, to make sure that changes do not contradict those assumptions you made when you initially wrote the system — or they alert you to the fact that those assumptions are wrong, or the circumstances around them have changed, so a previous assumption that may have been valid in the past is no longer valid.
Tests provide their greatest value over time. There is an up front cost, which is hopefully defrayed by the initial planning and checking that prevents bugs from being added in the first place.
Unless maintaining or adapting tests costs more than the benefits they give.
Which is why, although I am generally a test first advocate, when you are doing new development on a blue sky project, testing may actually slow you down, and end up not providing enough value to justify it’s expense.
A lot of the assumptions you make early on may not be valid. And by codifying those assumptions in tests, you may be making the system too rigid and resistant to change.
Often, the first version of software is written with the assumption it will be thrown away and rewritten from scratch once you have clearer understanding of the problem and how you are going to address it. In that case, tests will be a waste of time, and any residual value they provide will never be realized.
But just as often, the quick and dirty, one-off or proof of concept project becomes production code. And it often ends up being too brittle, not scalable, or too tightly coupled to expand. And then, you may be in a situation where you need to rewrite it, but you can’t, because real world business processes depend on it or external forces demand that it stay running.
In this case, tests can again provide a value. With reliable tests you can then dissect are migrate the system with confidence, even if the authors of the original software system (and their domain knowledge) are gone.
That’s where the fifth (and perhaps final) value of testing comes in. If you have tests in place on a working system, you can then make changes to subsystems, even rewriting or removing them, if your tests can verify that your replacements don’t negatively impact other parts of the system, or the system as a hole.
In this case the tests act as a fence (or railing) for safety. To prevent you from going over the precipice or out of bounds of the system. The tests can act as a crutch or scaffold that helps protect the system from falling to pieces while you work up fixing or updating it.
But in order to do so, you need to have good tests. And by “good tests” I mean tests that are clear what they are testing. They test only one thing, so you know what is broken if the test fails.
The tests need to be easy to modify, to adapt to a changing system, and to be able to be changed, eliminated, or replaced, when the assumptions about the system change, or the way they can be tested changes.
Good tests need to be flexible. They need to not be brittle, and they need to be able to work under different conditions. Not tied to the UI, environment, or specific data (except when needed to test that specific area of the UI, depend on some aspect of that environment), or when that specific data illustrates the conditions of that test.
Above all, good tests need to be reliable. They can’t break over every minor change to the system, or for random, indeterminate reasons. Flaky tests might be worse than no tests, because they reduce your confidence in testing. So they should be robust and adaptable to changes in the system.
A good test framework should help you focus on simplicity, help you to write reliable, robust, specific tests, and help you to keep them organized, and make reporting clear, meaningful, and concise.
You should be able to run your tests alt least every day, ideally for every change. And someone should care about the results. In order for people to care about tests, they need to pass reliably, no false positives or intermittent failures.
In summary, testing provide value 5 different time:
- The act of writing tests forces you to think about the system and plan for possible issues.
- Writing tests while developing software prevents bugs from appearing because you anticipate or catch them while writing tests. It also documents your assumptions about how the system should work.
- Executing tests exercises the system and informs you that the requirements are being met. Repeatedly executing tests and keeping them up to date will make tests more robust and adaptable.
- Testing provides value over time as the system grows, it prevents regressions and allows you to reason about the system and not be slowed down.
- Tests act as checks that allow you to refactor the system and make changes without breaking functionality. Even when part or the whole of the system has become a black box.