Testability: The tests aren’t enough
Why the learning system around a technical practice determines whether it lasts
We thought we had “solved” regression testing
Back in 2018, I was a Software Engineer in Test (SEIT) working in a mobile engineering team. My team was responsible for developing a media player used by other mobile apps across our organisation. Those apps passed the media player the URL of the media they wanted to play, and the player handled everything else, including decoding, playback controls, screen orientation and full-screen playback, etc.
Whenever we changed the media player, it could take between one and three days of manual regression testing to establish whether we had broken any existing behaviour.
So we embarked on a code-level test automation approach that let us make changes to the code base and quickly identify whether we had broken any existing behaviour. This brought regression testing down to under a day, and in some cases, only a few hours. It also made releasing the module much quicker. It appeared that we had solved the problem.
But by 2020, the approach had largely stopped evolving, and the team had returned to manual regression testing.
I’ve written up the full story of how we built and lost the approach in Why we failed at testability. That post goes into the technical and team journey in much more detail. In this post, I want to revisit one part of it through ideas I’ve developed since: feedback loops, uncertainty, psychological safety and how quality emerges from the conditions around the work.
The tests survived. The learning system didn’t
The tests and the architecture were the visible outputs of our work. The team could see what we had built, but what was less visible was how we had learned our way towards this approach. That learning system was essential for the practice to continue evolving.
The learning system was made up of feedback loops, collaboration and experimentation, supported by an environment in which we could make our uncertainty visible.
What I now understand is that our approach helped us reduce uncertainty in two ways. The tests helped us reduce uncertainty about the software. Experimentation and collaboration helped us reduce uncertainty about how to build a testing approach that worked for our context.
Two kinds of feedback
Feedback about the approach
We first needed to learn more about code-level testing. As a team, we watched talks about how others approached it and the traps they had fallen into. This helped us develop a shared understanding of why these tests mattered and how faster feedback could reduce the uncertainty created whenever we changed the code.
But that didn’t tell us which tests our system needed or how we should write them. We still had to apply that learning to our own context.
We also couldn’t stop delivering for weeks while we added tests to everything, so we identified a change where we could experiment with the approach. Our media playback module needed to be migrated to a new version of the third-party player, which gave us an ideal opportunity. So to allow the rest of the team to continue delivering, three of us formed a temporary trio: two developers and me, the tester. Our task was to work out how to test the new implementation effectively.
This gave us a bounded space to learn through trial and error without stopping the rest of the team from delivering.
As a trio, we had to learn as we went along. We needed to understand which types of code-level tests we needed, how often and where they would run, what feedback they provided and what it meant when they failed. The testing approach and the architecture that supported it evolved together as we developed the implementation.
As the approach evolved, the code-level tests gave us the confidence to refactor any architecture that did not fit our team’s needs.
We kept the wider team informed through regular demonstrations of what we had built. But those demos mostly showed the technical output. They didn’t expose all the experiments, conversations, failed ideas and adjustments that had helped us create it.
This learning feedback loop didn’t just validate the solution. It helped create it.
Feedback about the software
A key learning for our trio was that we needed multiple layers of testing to reduce different kinds of uncertainty about the changes we made. This resulted in what we called fast and slow code-level tests.
Our fast code-level tests ran hundreds of checks in a few seconds. They gave us almost immediate feedback about whether a code change had altered existing behaviour.
Our slow code-level tests ran through the build pipeline and deployed the module to actual phones, providing feedback within a few minutes. They gave us broader confidence that our integration with the third-party player worked within the wider module and that media would play when requested.
We used exploratory testing more selectively to investigate new behaviour, challenge our assumptions and look for risks we had not yet captured in code.
When exploratory testing revealed an important risk or behaviour, we added it to the fast test suite wherever possible. We added slower tests only when the behaviour could not be covered effectively at the faster level. Once we were confident we had covered the new behaviour through fast and slow tests, we trusted those tests to tell us if it changed in the future.
The fast and slow tests helped us identify when known behaviour had changed. Exploratory testing helped us investigate new behaviour, challenge what we thought we knew and discover risks that were not yet represented in the automated tests.
When we first started, we still defaulted to exploratory testing because we did not yet trust the automated feedback. Over time, exploratory testing revealed fewer issues with existing behaviour. As the automated suite grew and proved reliable, we increasingly trusted it to identify behavioural changes in the code.
This allowed us to focus more of our exploratory testing on new behaviour and risks rather than repeatedly checking whether existing behaviour still worked.
Psychological safety
Both feedback loops depended on information being made visible. Automated tests made changes in the software visible, but our trio also needed to make confusion, disagreement and gaps in our understanding visible.
This was an aspect that I’d overlooked at the time. For our trio to share this feedback and develop the automation approach, we needed to be willing to work across our discipline boundaries and be willing to make ourselves vulnerable.
Vulnerable in the sense that we could make visible what we didn’t know or understand, and where we disagreed. We needed to be able to say things such as:
“I don’t understand unit testing.”
“I’m not sure what these tests prove.”
“This architecture is difficult to test.”
“Our current approach is not working.”
“Let’s try something different.”
Some of these things were not easy to say. Each of us had to take a small leap of faith and trust that the others would not judge or embarrass the person taking the risk. Each time someone admitted uncertainty and was met with curiosity and understanding rather than judgement, it became a little easier for the rest of us to do the same.
Without this honesty and candour within the trio, I don’t think we would have been as successful. This feedback allowed us to make the gaps in our understanding visible and work together to fill them.
At first, this could feel as though it was slowing us down because we took time to understand the tests and approach together. But as our shared understanding grew, we were able to work more quickly and make decisions with greater confidence.
Quality is emergent
When we started, our intention wasn’t to improve testability or even to improve the quality of our product. What we wanted was rapid feedback on whether changes to the codebase had broken any existing behaviour. Essentially, we were asking whether there was a better way to do regression testing that didn’t take days.
But what emerged was much more than that. We developed:
A shared understanding of what code-level testing meant for our team.
Cross-disciplinary learning in which we could expose gaps in our understanding.
Small experiments that allowed us to reflect and adapt as we learned.
Layered tests and an architecture that gave us faster, more focused feedback.
All these things created better conditions for how we worked. None of them guaranteed quality on their own. Together, however, they made it easier for us to understand changes, expose gaps, correct mistakes and adapt our approach.
Better quality emerged from our ability to learn and respond, but it wasn’t accidental. It emerged from the conditions we had created around the work. Testability was part of the approach we developed, while many of the wider quality improvements were outcomes we had not originally set out to create.
Why did it fail?
So why had the team returned to manual regression testing by 2020? When our trio moved into different roles, the technical tests remained, but many of the conditions that had produced them did not. Without those conditions, the practices gradually atrophied.
With the approach no longer being evolved, the existing tests remained, but no new tests were created because it wasn’t clear from the tests alone what benefit they brought. The fast and slow automated tests began to seem overly complicated. Why did we need two types of tests and multiple layers of abstraction?
Confidence in the tests slowly declined. Instead of being maintained when they failed, they were gradually disabled and replaced by manual regression testing.
For the approach to become a lasting capability, it needed more than just the technical tests. People needed to understand its purpose. They also needed to be able to adapt it as circumstances changed and continue learning without the original trio being present.
What needed to survive was the learning system that had led to the approach. The tests were its visible output, but they were not the capability itself.
Some of the test code survived, but the team’s ability to understand, adapt and extend the approach didn’t.
If this helped you think differently about testability and technical practices, subscribe to receive future Quality Engineering posts.
Further reading
This post brings together multiple ideas I’ve explored in more depth:
Why we failed at testability tells the full technical and team story behind this post. It looks at how we developed the approach, what enabled it to succeed and why we eventually returned to manual regression testing.
The six layers of testing introduces a broader mental model for thinking about different layers of testing. It can help you consider what feedback each layer provides, the uncertainty it reduces and the trade-offs involved.
Why is psychological safety important to software engineering teams? explores why people need to feel able to ask questions, admit uncertainty, challenge ideas and expose mistakes. It provides more detail on the conditions that allowed our trio to learn effectively together.
Nudging and boosting complex systems looks at how we can influence complex systems without assuming we can directly control their outcomes. It helps explain how changes to feedback, collaboration, experimentation and architecture can influence the conditions around the work.
Quality is Emergent develops the broader argument that quality cannot simply be added through a process or individual practice. Instead, it emerges through the interactions between people, technology, decisions, constraints and the wider environment.