Test coverage does not tell you whether a change is safe enough
Five factors that help teams reason about change safety
Imagine two mobile app teams. They’ve both made changes to their mobile apps.
Team 1 is rolling out to 1% of its users and has telemetry built in. The change sits behind a feature flag and can be disabled instantly.
Team 2 has a solid testing strategy, with testers feeding back their experience. They plan to deploy to all their users, and if there is an issue, they would push an emergency patch.
Both teams have the same amount of automated test coverage, yet the conditions around the change are very different. Test coverage reduces uncertainty, but it cannot tell you on its own whether a change is safe enough to release.
Things like monitoring, observability, staged roll-outs, feature flags and rollback systems help manage uncertainty by giving you ways to detect, minimise and mitigate the impact if something slips through.
Safe change comes from the system of feedback and controls around the change, not from test suites in isolation.
So starting with what needs testing is probably too early. What we need to understand first is: What has changed, and what uncertainty does that change introduce?
This gives teams five areas to explore when thinking about whether a change is safe enough:
Understanding - Do we understand what is changing, why, and what it could affect?
Consequence - How bad would being wrong actually be?
Detectability - How quickly and reliably would we know something had gone wrong?
Containment - How much of the system/users could be affected if we’re wrong?
Reversibility - How quickly and safely could we undo, disable or mitigate the change?
These are not independent questions, and we’ll come back to how they interact later. But it helps to look at each one in turn.
Understanding
Understanding what has changed and why is key, as without it you can’t understand any of the other factors.
For example, are you adding, changing or removing behaviour? How might that affect existing behaviours, dependencies and assumptions?
It’s also about understanding the environment in which the change will occur, such as affected users or operational assumptions.
Understanding isn’t about eliminating uncertainty. It’s about knowing enough to make good decisions about the other four factors.
Consequence
Consequence is about how bad it would be if the change went wrong. If the consequence is minimal, for example, correcting a typo and nothing else, you might conclude that making the change is fine. But let’s say it’s a change to a payment flow. That could affect someone financially.
More testing might reduce our uncertainty, but it doesn’t necessarily reduce the consequence if we are wrong.
So you may also want a smaller rollout, stronger monitoring, and better reversibility to minimise and mitigate any potential impact.
Detectability
For the consequences you’ve identified, how would you know if they were becoming a reality?
This is where you look at your team’s ability to monitor and observe your system. Customer support, business metrics, app telemetry, app-store feedback, synthetic checks, and other operational data can all act as signals. You need signals that help you notice and understand unexpected behaviour. But speed is also crucial here, as detecting issues within minutes rather than days can dramatically reduce how long users are exposed to them.
Containment
If consequence is about the impact of something going wrong, then containment is how much exposure you allow.
For instance, imagine a failure in a new sign-up flow that affects your product’s ability to bring on new users. If it affects only 0.5% of new subscribers for a week, that might be more tolerable than if it affected 100% of new users for a week.
This is where you’d be thinking about how you roll out changes to users.
Reversibility
Reversibility is as it sounds: if, for whatever reason, a change shouldn’t remain in production, how easily can we reverse that change?
But rollbacks are not the only way to reverse a change. Disabling a feature, switching traffic, restoring data, pushing a patch and falling back to known behaviour are all ways to reverse a change.
A good approach to reversibility is to prefer the option that introduces the least amount of change, since a complex recovery process can introduce additional uncertainty to an issue.
Interestingly, for some teams, their reversibility strategy can also be their default containment approach. So instead of trying to contain an issue, they reverse it as soon as they detect it. Because of this, approaches to containment and reversibility can be conflated.
So, what makes a change safe enough?
These five factors are not gates to pass or a step-by-step decision framework to follow. They help you understand what evidence teams need before deciding a change is safe enough to release.
A change with high consequence, but poor detectability and weak reversibility, should make the team cautious. But a change with high uncertainty, fast detectability, strong containment and near-instant reversibility may be fine to gradually roll out.
The five factors influence one another, but they do not cancel each other out. Loss of user data is still loss of user data even if it affects less than 1% of users. Strength in one factor can change how much uncertainty we’re willing to accept elsewhere, but it doesn’t make the other factors irrelevant.
And even then, confidence in an individual change is not the same as confidence in the whole product. Changes interact, so we still need feedback about how the wider system behaves.
Taken together, this is why test coverage does not tell you whether a change is safe enough. It is one source of information within a wider set of system feedback and controls around the change.
Whether a change is safe enough is a judgement we make from the evidence and controls around it.
If this helped you think differently about what makes a change safe enough, subscribe to receive future Quality Engineering posts on testing, uncertainty, risk and building quality in.