The six layers of testing
The six layers of testing is a mental model representing the fundamental levels of testing and how they interrelate. It highlights the benefits and tradeoffs and encourages collaborative discussions.
The Layers of Testing
I was recently thinking about how I talk about testing when working with engineering teams and realised that I have this mental image. It is an almost ideal state of testing, which happens to be a mixture of the testing pyramid with a few other testing stages thrown in.
Why the Testing Pyramid Isn't Enough
Now, I don't visualise it (mentally or otherwise) as a pyramid, as while it's a good rule of thumb (have more code-level tests than end-to-end tests), it's not all that helpful. People tend to get stuck on what the layers are and how they look for them, e.g. trophies, ice cream cones, honeycombs, hourglass*
* Check out Nick Moore's post on the Qase blog, which has a good history of the test pyramid.
Mental scaffolding
Not only that, but the layers of testing I'm often thinking of can vary from team to team depending on their context, level of skill, or time required to implement tests. As we're discussing testing, I'm mentally visualising the different layers and using this as scaffolding to help guide the conversation with the team.
My approach is not to lead the conversation but to let it flow naturally. For instance, if the team discusses a specific layer, I'll nudge them to consider other aspects, such as the approach's benefits or trade-offs.
If their understanding at that level is good, then I may move up and down the layers (depending on what they are discussing).
My aim is to encourage them to investigate other approaches that help them build up layers of testing rather than focusing on just one.
The terminology varies from team to team, but typically, it has around six layers. I've described them below to give you an idea of how I think about them and the benefits and trade-offs I’d mention.
The six layers of testing
Layer 1: Testing in Production
Testing in production can involve a range of approaches, such as exploratory testing, monitoring using telemetry to see how the system behaves, observability tools to investigate behaviours, feature flags and phased rollout of systems and features.
The benefits are that it is the most realistic form of testing, with no stubbed data or test environments; this is how your end-users experience your system.
The trade-off is that it's live, which means some (all?) users see the issues with your system, which could damage your reputation and brand image.
Layer 2: Manual regression testing
Regression testing is usually done at the end of the development process when developers think the feature is ready. Here, testing is the traditional inspection for quality with Testers at the end of the process. Testers usually do this testing at the end of the sprint every two weeks (if they work in sprints). Feedback is however long it takes the testers to tell the dev team something works as expected, or something else has broken.
The benefits are that this is the last line of defence before the system reaches your end users and is closest to what users will likely experience.
The trade-off is that finding an issue here means going back into the dev cycle again, which can further delay the release, or some teams opt to ship it and see what happens. It can be tedious to run manually, often seen as a bottleneck to releasing.
Layer 3: Automated end-to-end UI tests
Automated UI tests typically run as soon as the developers think they are done, usually at the end of the build pipeline. Feedback time is however long the automated test takes to be created and run, usually 10-30 minutes (excluding engineering time). Developers, Testers, Software Engineers in Test (SEiT), or Automated Testers can write automated UI Tests.
The benefits are that the engineering teams can run them as often as their infrastructure allows them to, and, depending on the quality of the tests, they can build confidence that the product will perform as expected in the scripted scenarios.
The trade-offs are that they can be time-consuming to write and maintain, and once you have 50+, it can be hard to understand what is and isn't covered. They also tend only to be run at the end of the development cycle rather than during due to the time needed to set up, create, maintain, and interpret the results, slowing down the developer feedback loop.
Layer 4: Exploratory testing
Exploratory testing can occur at any stage during development and will likely occur throughout the life cycle. I've put it in layer four because I've always felt this testing should happen as close to the development process possible so the engineering teams can benefit most from this feedback.
Therefore, I encourage teams to do this testing during development, pairing Testers and Developers so that Testers can provide immediate feedback to Developers as they build the feature.
The benefit is that it's the most flexible and informative type of testing and one of the few ways to understand how your product performs as an end-to-end system. It can also allow for the testing of unique test cases in which the trade-off of automating would outweigh the value they would provide.
The other trade-off is that it requires skill and experience to be done well, which Testers tend to be the best at but are often in short supply. It is also time-consuming and challenging to repeat.
Layer 5: Automated code tests
Automated code tests are run every time the Developer writes code. These tests are sometimes called unit/integration testing or TTD (Test-Driven Development) tests, which Developers write before any code is written. TTD is an approach to using tests to drive the design of the code and helps engineering teams build in testability. This feedback is as often as the d
Developers execute the tests, which usually takes less than a second to run.
The benefits are they are fast and can be run on demand by Developers, giving them instant feedback on whether their system's behaviour has changed.
The trade-off is that it only tests the code written by the engineering team, with third parties replaced by test doubles. Due to the number of tests, it is also challenging to see what is and isn't covered, even if they have been written to a high standard. It also tends only to cover happy paths through the systems.
Layer 6: Requirements Testing
Requirements Testing is usually done before any code is written or even if the teams have decided on whether it will be built. Testers, Developers and Product Managers discuss what the feature is, what it would do, how it would do it, and the benefits it brings the team, organisation and users (some call these pre-flight checks, story kicks-offs, triads or three amigos). The discussion is not about testing or coding but understanding the feature, the what, and why to ensure we're building the right thing.
The benefit is that the engineering team will use the shared understanding developed here throughout all the other stages mentioned above.
Trade-offs are that nothing yet exists, so any unknown-unknowns will not be discovered until later in the development life cycle.
Tailoring Testing to Your Team
These layers of testing are by no means exhaustive in terms of the types of tests, their descriptions, and the benefits or trade-offs. They are also very likely to be called different names by different people, so tailor them to your situation, needs, and levels of understanding.
I've also found that many teams blur the lines between these layers or have them in different orders or even occur simultaneously. The scaffolding helps me think through what layers they are talking about and what they could be missing. I'll then use that to dig into why they are missing those layers and what benefits and trade-offs having them could bring to the team.
Share Your Testing Framework
I'm intrigued to hear how others approach these conversations. Do you use a framework or mental scaffolding to help with the conversation, or do you pull out the pyramid? What are you hoping to help the team achieve and why?