The Key to Effective Test Automation: Start with Why, Not How
10 years of test automation: What should you do and why should you do it?
Key insight
Software engineering teams should approach automated UI testing with specific goals rather than consider it a solution for every testing challenge. It's essential to focus on the purpose of automation, such as improving observable behaviours and addressing actual problems, rather than simply automating for the sake of automation. There are many misconceptions and misuses of automated UI testing, which teams must recognise and proactively work towards reducing to extract as much value from UI automation as possible.
Three key takeaways
Start with the why and what: Instead of focusing on who and how to automate, it's important first to understand why you are automating and what you hope to achieve. The why and the what can help limit the impact of automated UI test bloat by allowing you to assess what should and shouldn't be automated.
Focus on observable behaviours and improvement: When deciding the purpose of automation, teams should focus on observable behaviours such as increases in the speed of delivery, tangible quality improvements, or decreased test cycle times. It's important to develop observable behaviours and benchmarks before writing any automation to avoid vanity metrics to make the automation look good.
Problem-first approach: Instead of treating UI automation as a solution looking for a problem, start with identifying the actual problem that needs to be addressed. Don't assume that testing is the biggest bottleneck to delivery speed without thoroughly evaluating the situation.
Over the years, I've been involved with quite a few user interface (UI) automation initiatives, from hands-on writing the automation to building tooling solutions so others could automate, and more recently, acting as a consultant feeding back to teams on their approaches.
I've also written about it a few times, from how we used to use three amigos, our mobile app automation approach, and whether you should retrofit tests around 2014. In 2018, I followed up with what I thought UI automation helped with and the unintended behaviours I have observed from our approaches. Then, in the last few years, I've gone deeper into Testability, which resulted in how we failed at testability talk (I should write this up) and how we can use automation to build confidence in our stakeholders.
I naively thought that by 2024, we would have learned everything there is to learn about UI automation and have some standard patterns for teams to follow. However, I still find myself having many of these conversations with developers and testers alike.
So, in this post, I will try to pull all those threads over the last ten years together and share how I think we should approach UI automation and some of the patterns to look out for when working with teams that are starting or are already using UI automation.
Automated UI Testing is probably not going to give you what you want
Automated UI testing is probably not going to give testers what they expected. Here are some patterns to look out for from UI Automation, what is it good for?
There are a lot of reasons that people give as to why they want to automate their testing, such as
Reduce test/regression testing 🤔
Regression testing is used to determine whether changes made to your code base have not broken anything existing. Unless you have automated all your UI checks/regression tests, automation will not help you as much as you think.
Spot issues/bugs faster 🤔
Automation doesn't find new bugs. It only tells you that the check you've scripted has somehow broken. You need to tell the script that if action A doesn't produce result B, it fails with an error message. What usually happens is the check fails in a way you didn't anticipate. Don't forget that if you had known beforehand how something would break, you would probably have put in a fix. That is why they are called defects, something behaving the way you didn't want/anticipate.
Free up Testers 🤔
Potentially, if they trust the automation (See Building Confidence with automation).
Consistently check a feature the same way 👍
This is one thing an automation check is perfect at.
Something laborious or difficult to set up and check 👍
Another good candidate for automation.
We're doing Behaviour Driven Development (BDD) 🤔
BDD is not about automating but more about collaborating to understand and create features. The automation is just one small part of it, and even then, it's not about testing the UI but the business logic, which could be tested at the Unit level. If you hear a development team saying, "The BDD tests are failing, " it's a good indicator that they are probably misusing BDD.
To release faster 🤔
Again, because you need to do less testing, see Reduce test/regression testing.
It's a part of continuous integration/delivery, so we have to 🤔
There is no thought in what you are automating other than what people say you must do.
The Test Manager or some other higher-up tells you to 🤔
Some think just telling a development team to automate their testing will help them. See above.
People within the development team or key stakeholders don't trust the developer's work 🤔
The test team are being used as a safety net to check the developer's work, which tends to have a self-fulfilling prophecy for the developers who start using the test team as their safety net.
I've heard these reasons repeatedly over the years. While automation could provide some of these benefits, if the teams don't proactively work towards these outcomes, they don't usually occur other than creating a lot of UI tests.
Automated UI testing is probably going to give you things you didn't want
If teams do start building out automated UI tests, then they risk introducing some unintended behaviours in their teams. Here are some patterns to keep an eye on from The unintended consequences of automated UI tests:
UI automation encourages:
Writing more integrated testing with the full stack rather than isolated tests.
Isolated behaviour tests (e.g. unit, integration, contract tests, etc.) run faster and help pinpoint where issues could be. An automated UI test will indicate that a specific user journey is not working. While beneficial from an end-user perspective, someone still needs to investigate why. A failed automated UI test can lead to re-running it to see if it's an intermittent error. This is only made worse by tests giving false negatives, and full-stack tests are more likely to do so because of having more moving parts.
Testing becomes someone else responsibility.
This is more apparent when somebody else in the team does the automated UI test, not the pair developing the code. Notice 'pair'. If you're not a one-person development army, then why are you working alone? Pairs tend to produce better, higher-quality code with instant feedback from a real person. It might be slower at first, but it's worth it to go faster later.
Pushing testing towards the end of the development life cycle.
Automated UI tests only work through a fully integrated system. Therefore, testing gets pushed later into the development life cycle. You could use test doubles for parts, but that is not an end-to-end test.
Slower feedback loops for development teams.
Due to testing being pushed to the later stages of development, developers go longer without feedback on their work progress. This problem is increased further when the automated UI tools are unfamiliar to the developers, who wait for the development pipeline to run their tests instead of doing it locally.
Duplication of testing.
As the automated UI test suites grow, it becomes complicated and more challenging to determine what is and isn't covered by automation. This leads teams to duplicate testing at other levels (code and, most likely, exploratory testing), adding to the development time.
UI automation discourages:
Building Testability into the design of the systems.
Why would you if you know you can "easily" test something end-to-end with an automation tool?
Maintainability of the code base.
By limiting the opportunities to build a more testable design, you decrease the maintainability of the code through tests. If you need to make a change, it's harder to see what the change in the code affects. By having more fine-grained tests, you can pinpoint where issues exist. Automated UI tests indicate that a journey has broken and how it could affect the end users, but not where the problem was introduced.
Building quality at the source.
You are deferring testing towards the end of the development pipeline when everything has been integrated. Instead of when you are actively developing the code. Are you going to go back and add in the tests, especially if you know an end-to-end test will cover it?
The responsibility to test your work.
With the "safety net" of the automated UI tests, you send the message that it's ok if something slips through development. If it affects anything, the automated UI tests will catch it. What we should be encouraging is that it's the developer's responsibility to build AND test their work. Developers should be confident that it can be shipped once they have finished that piece of code. The automated UI tests should act as another layer to build on your team's confidence that nothing catastrophic will impact the end users. Think of them as a canary in the coal mine. If it stops chirping, then something is really wrong.
More granular feedback loops.
By having automated UI tests, you're less likely to write unit and integration tests, which give you fast feedback on how that part of the code behaves. Remember, code-level tests should be testing behaviour, not implementation details.
While all these issues can be mitigated with different strategies, the system is constantly trying to regress towards these outcomes. Therefore, teams need to work to reduce these issues proactively. If you're unaware that these unintended behaviours could be problems then they are more likely to occur.
Why are we automating?
The typical approach to UI automation: The Who and the How
As you can see from above, many things people want from automation don't usually happen. This is because when teams start with UI automation, they spend more time on who and how they will do it. Which makes sense, automated UI testing isn't easy to do well. It often requires a lot of engineering skills, and most testers are not usually the best equipped. Therefore, they are learning on the job while still juggling their day jobs of testing. That's not to say testers can't automate UI tests; they can, but it's not going to be easy, and you're effectively asking them to become developers.
To help with the engineering skills, most teams will try to hire Software Engineers in Tests (SEiT) to do the heavy lifting (choosing tools, putting frameworks in place, and training others to write tests). This will enable the testers to write UI tests with support from the SEiT or more experienced automated engineers (usually testers who can write tests but maybe not independently set up frameworks from scratch like a SEiT).
When teams focus on the who and the how they usually end up with automated UI test bloat. This is when engineering teams have hundreds of UI tests, and no one knows what they cover. In addition, they often return false positives that require lots of effort (usually by SEiT and automation engineers) to maintain. Flakey tests frequently result in them not running as part of the main build pipeline but adjacent to it or, in some cases, completely detached on an overnight test run. Further limiting the fast feedback, the automated tests were there to provide the team.
We should be starting with the Why and What
By taking the time to understand why we are automating and what we think it will achieve, you can limit the impact of automated UI test bloat as you have a way to assess what you should and shouldn't automate. Rather than what typically happens is every use case the team can think of or couldn't figure out how to automate at lower levels within the code.
By looking at the why of automation, you can also begin pulling in the behaviours we still want to encourage and start thinking of strategies that help maintain and strengthen them.
What improvement will the automation bring?
When deciding the why of automation, teams should focus on observable behaviours. For example, increases in the speed of delivery would decrease cycle times (code commit to running in production) and increase release frequency (See What are the four key metrics to learn more). Or test cycle times could decrease from days to hours. If it's product quality improvements, then what tangible quality attributes can be measured?
Ideally, teams would develop observable behaviours and benchmarks before writing any automation. Otherwise, you get teams retrofitting behaviours they can see to make the automation look successful. For instance, you get vanity metrics like the number of tests run, hours of testing completed, or the number of devices tested that bear no relation to real-life usage or even benchmarked against what the team did before the automation as it wasn't recorded. So, any numbers they report look good.
UI automation: A solution looking for a problem?
When engineering teams start with automation, it's usually because someone has decided that it's what the teams need and will help improve things. It's often based on gut feeling or that it's something other successful teams are doing, so we should, too. Automation is frequently a solution looking for a problem, but what if you start with the problem first? Would automation be the solution?
One of the biggest benefits touted for automated UI testing is speed. Speed of feedback, which will hopefully lead to increases in speed of delivery. But that is assuming that testing is your team's biggest bottleneck to delivery. The thing is, your software development life cycle is only as fast as its slowest process. If anything before the testing phase is a constraint within the software pipeline, then it doesn't matter how fast you make your testing process; you'll still only be going as fast as that constraint. To realise the benefits of faster testing, you must first ensure all other constraints before testing have been alleviated.
Start with the constraints
Use techniques like the Theory of Constraints (ToC) to identify your biggest bottleneck to delivery within their software engineering process. Then, look at as many options to alleviate that constraint as the team can and choose the one they have the skills and resources to pursue. (See Speed Vs Quality, Section 9 Theory of Constraints for a walked-through example of how to use ToC). If the solution is automated UI testing, then congratulations. When you implement the solution, you not only get what you wanted (automated UI tests), but you've meaningfully improved your team's ability to deliver at speed. It also acts like a break on the amount of automated UI testing you implement because you'll naturally stop if your speed of delivery begins to decrease.
Now, when you show your stakeholders what you've accomplished, they will be much more pleased than if you just showed them a bunch of vanity metrics that don't connect to anything they value, e.g., new features faster. It also builds confidence with your stakeholders that you know how to improve your ways of working while delivering value. Which will likely lead them to be more willing to give you time to make other improvements in the future.
Sharing your achievements with your stakeholders is not about getting their approval but building a collaborative relationship through transparency. By showing them how and what you are doing, they will be more receptive to your suggestions in the future. By reaffirming how your work benefits them, they will know you have their interests at heart.
But we've already started automating. What should we do?
Suppose you've already started automating at the UI level, then what? In that case, you should focus on two things: 1. Core user journeys and 2. Testability. For the core use journey, you can apply the PUMA model (Proves core functionality, Understood by all, Mandatory to be part of the build pipeline and Automated) to help you decide what you should test, from UI Automation, what is it good for?:
We use the concept of PUMA to decide what our core journeys are and, ultimately, what we should and shouldn't automate. A general rule of thumb is that if it's not a core journey, then it should be covered by a unit/integration test that does not involve the UI.
We should be building confidence with our team about what is and isn't covered with those tests, from Building confidence with automation:
If you're a developer or automation specialist, you have two options for improving people's confidence in the automation output. Help them "see" the process or build their trust with you. Both of these will improve their confidence in the process.
Or go one step further with principles:
Better yet, help them understand the principles behind automation. If done with humility and compassion, this will naturally lead to those people trusting you as well. By helping them understand the principles behind automation, you enable them to work out for themselves what is and isn't being automated and what standard it is happening to. This lets them see the gaps in the process, which they can raise as risks or work to plug them up.
As for Testability, this is all about designing your systems as small testable chunks, from Building Quality in via Testability:
Testability is not about testing the system end-to-end but piece-by-piece. Each piece is a specific type of behaviour the system provides and is tested in isolation from the other pieces. The team should decide the scope and definition of the behaviours collaboratively.
Final thoughts
Automated UI testing is one of those approaches that will never go away. Most teams will have some form of it to a greater or lesser extent. But there still appears to be a lot of misconceptions about what it will and will not do, with most people assuming that more has to be good. From my experience, teams going from predominately manual exploratory testing to automated UI testing usually swap one set of problems (slow feedback loops, days of regression testing) to another set (flaky automated tests, test bloat, uncertainty in what is covered).
Don't get me wrong. I'm not saying we shouldn't use automated UI testing. It can be incredibly valuable (testing core user journeys, testing hard-to-setup tests, repeatable testing). However, the line between valuable and bloated is often quite blurred, and without thinking through the what and the why, teams jump straight into the who and the how.
Automated UI testing should be one of the many layers in an engineering team's approach to understanding how changes to their code base affect its behaviour. If it becomes the main approach or, worse, the only approach, then teams are opening themselves up to bigger problems down the line.
How should quality engineers help engineering teams?
As quality engineers, our role in engineering teams is to enable them to build quality in, so anything that is overly focused on inspecting for quality at the end of the development pipeline is leading away from this goal.
Look for the patterns described in what testers think automation will give them and the unintended behaviours of test automation to see if engineering teams are heading towards test bloat over improving their software delivery process.
We want teams to think about what is constraining their ability to deliver at speed based on actual measurable observations, not gut feelings. See my talk Speed Vs Quality on how to help teams start using the Theory of Constraints (ToC) to help them identify the bottleneck to delivery and whether testing is really their biggest constraint. I'd also recommend looking at Faster Delivering Teams? Kill the Test Column, as that also has a worked-through example of how to use ToC in engineering teams to improve their ways of working.
Then, help engineering teams build Testability into our software systems. Dave Farley's The Role of QA in Agile Software does an excellent job of explaining Testability and how testers can help.