Can We Automate All the Things? Exploring the Limits of Testing in Software Systems

Can you automate all your testing? Well, it depends on how certain you are about your software system's behaviour.

If there is one thing I see most often across the software industry, it is the view that you can swap manual testing for automated testing while keeping all the upside of manual testing (feedback on how your product works) and removing all the downsides (slow, tedious, and requires people to do it).

Testing isn’t binary

As mentioned in The Testing Unknown Unknowns, I feel this perspective comes from how we frame testing which is often in a very binary view of either something works as intended (a pass), or it doesn't (a fail). I think this limits our thinking that testing only has one purpose, and that purpose can be automated away. But for me, testing is all about feedback that helps us lower the uncertainty we have about how our software systems behave.

And if we reframe testing into the unknown-unknowns matrix we can better understand how it helps us to lower our uncertainty about the behaviour of our software systems, see image below.

Can you automate all your testing?

But we’re still left with the question of whether you can automate all of your manual testing? Well, that depends on how certain you are about your software system's behaviour.

Where Does Uncertainty Lie?

As mentioned in The Testing Unknown Unknowns, one of the ways in which I like to use the model is in a linear three-line grid:

What you find when you think of testing as feedback is that each feedback approach always has some overlap with other areas, and so actually looks more like a Venn diagram:

So, if you only did testing in the Known-knowns. You would learn a little about the Known-unknowns and Unknown-unknowns. However, there would still be a lot left within each Known-unknowns and Unknown-unknowns. Therefore, if you had a perfect system with no unknowns, then you could automate all your testing.

Perfect systems don't exist

A perfect software system is one in which you can see and control all variables that affect its environment. In a system like this, there are no Known-unknowns or Unknown-unknowns. Essentially, you have complete certainty about the behaviour of the system. Visualised it would look something like this:

Therefore, as you know everything there is to know about the system and can control everything in its environment, you can technically automate everything. Now, this level of isolation is only realistically possible within a development environment.

And that’s the problem: Most software doesn’t exist in a vacuum—it’s interconnected with other systems. Therefore, perfect isolation doesn't exist, and some extent of uncertainty within the Known-unknowns and Unknown-unknowns quadrants will always exist.

So, while you can automate everything within the Known-knowns. You are still going to have some risks exist within the Known-unknowns and Unknowns-unknows, see image:

Therefore, if the only testing you did was automated, you wouldn't know how much risk could still be present and if what you have discovered is the highest risk within that area.

The more testing you do, the more certainty you have

To lower your uncertainty about how your system behaves, you need to get feedback (testing) from the Known-unknowns and Unknown-unknowns areas.

The more testing you do, the more certainty you have, slowly increasing the amount of overlap between all the known and unknown areas. Therefore, the more you automate the Known-knowns, the greater your coverage will be.

But change introduces more uncertainty

Every time you modify a system, you introduce new uncertainty, which reduces the overlap between what is known and what is unknown. As a result, you need more feedback (through testing) to decrease your uncertainty about the system's behaviour. This is why it is beneficial to implement small changes frequently. Small changes are easier to analyse, allowing you to better understand the uncertainty introduced and determine the amount of feedback (testing) required to reduce that uncertainty. In contrast, if you make large changes all at once, you will need significantly more feedback (testing) to manage the increased uncertainty.

So, can you automate all your testing?

Well, that depends on your uncertainty levels about the unknown behaviours of your software systems. If you have high levels of uncertainty, then no, you will not be able to automate all your feedback loops (testing). You will still need to use feedback techniques that help you understand your Known-unknowns and Unknown-unknowns. Now, some unknown feedback techniques might be partly or fully automatable, but that will depend on the techniques employed and tools used.

But if you have only low levels of uncertainty, then yes, you will be able to automate more of your feedback loops (testing) as you will have a much greater understanding of your knowns and unknowns. This leads me to the second question: Are some systems more automatable than others?

Are some systems more automatable?

Systems that talk to other software systems (think APIs) tend to be more applicable to automated testing as the environments they are deployed to are more deterministic. For example, you know who is likely to be calling your API, how they will call it, for what they are calling it for and even how often. Essentially, you are deploying to a controlled environment, so the number of unknowns is much smaller. Therefore, the Known-knowns are easier to automate even if there are a lot of use cases to cover.

Whereas systems that primarily interact with people (think web pages, mobile apps, etc.) can be automated, but you can not predicate every way in which your system can be used. This is even more of a problem for mobile apps that can be used in many different environments (varying screen sizes, low light conditions, intermittent connectivity, etc.) and different interaction methods (multitouch, keyboard or other accessibility tools, etc.).

Therefore, there is still too much uncertainty about your system's unknowns. Couple that with frequent software changes, and you're in a constant state of increasing uncertainty. This increasing uncertainty is why you need to deploy layers of testing and also think about the uncertainty those layers are helping to reduce.

Why exploratory testing matters

The unknown-unknowns Venn diagram also highlights why exploratory testing is so valuable. It's one of the few feedback techniques that can help you understand the Known-knowns, Known-unknowns, and Unknown-unknowns in just one testing session. While other techniques are very specific to the unknowns they help uncover, exploratory testing allows practitioners to adapt their approach as they explore the system.

But exploratory testing also has by far the biggest trade-offs. It's one of the slowest forms of feedback, it takes time to perform, it requires skill to do well, and it is often unrepeatable and, therefore, not automatable.

How does the testing unknown unknowns help quality engineers?

One way to think about testing or feedback is that it's about exploring uncharted territory. The feedback you obtain from testing is like a map - it helps engineering teams understand the lay of the land - the software system. The known and unknowns are regions on this map. Known-knowns are well-charted areas where we're confident in our understanding. The known-unknowns are areas we know exist but have yet to explore. Unknown-unknowns are the surprises—hidden cliffs or unmarked dangers we discover only by venturing into the unknown.

Automation is like surveying the land from afar using satellites - it works perfectly in the clear, open, controlled environments where the terrain is predictable - the known-knowns. However, we need scouts (exploratory testing) in dynamic, changing landscapes to uncover what the scanner misses.

Quality Engineers act as guides, helping teams chart the territory, decide where to send scouts and identify which areas are worth automating. The goal isn't just to map every corner but to focus on the regions that matter most for safe and efficient navigation.

The unknown unknown matrix helps quality engineers understand where their teams are uncertain and what they can do to improve. It also helps quality engineers understand what layers of testing they should be using, where they should be used, and how much of their software system is automatable. This approach doesn't just focus teams on inspecting for quality but using the information they obtain to build quality in by making their software systems more predictable and less uncertain in how they behave.

The future of exploratory testing

This brings me to another question about the benefits and trade-offs of exploratory testing. Are there instances where teams could reduce or even eliminate exploratory testing altogether?

I would love to hear others' thoughts on this topic, which I might explore in a future post.