Linky #37: You can’t know everything up front

This week’s links on release safety, systems thinking, feedback loops, and how we learn from what actually happens.

This week’s links circle around something I think is foundational to a lot of quality engineering: we can’t know everything up front.

We can test a change, adopt a new practice, build a prototype or run a transformation, but at some point we need feedback from the system to tell us what actually happened.

That means being able to observe what happens, learn from past events, look beyond the immediate team, and hold on to the knowledge we’ve already built up.

For me, a lot of quality engineering is about helping create those feedback loops so we can make better decisions even when we don’t have complete information.

Latest post from the QE Newsletter

Test coverage does not tell you whether a change is safe enough

I’ve been thinking about what makes a change safe to release, and usually that conversation comes down to whether something has been tested. That’s important because we still need to know something works as intended.

But testing doesn’t make it completely safe to release. We can’t test everything, and sometimes smaller issues can line up in production in just the right way, leading to a much bigger incident.

So alongside testing, we also need to understand what changed and the consequences of being wrong, then detect, contain, and reverse problems if they happen. Together, those factors give us more confidence that we can release safely.

This is a follow-on post to Start with what changed, which dives deeper into understanding what changed and what types of feedback help us reduce the uncertainty around that change.

Problem solving in software is more biology than engineering

Software is being asked to problem-solve more like biology, cognitive science, and all the fields that have always had to develop approaches to study systems we only partially observe.

Interesting post from Cat Hicks. This reminded me of what socio-technical systems and complexity theory show us: we can’t know or see everything about the systems we build and maintain.

Sometimes we need to observe what the system actually does and respond accordingly. So building in ways to better observe our systems and get feedback about how they’re behaving helps us diagnose and respond to issues. Via “Software developers will never read the code again!!” | Cat Hicks

Systems thinkers are not always the most welcome

With quality engineering, you’re often indirectly advocating for systems thinking, but as John Cutler points out, that doesn’t always make you the most welcome person in the organisation if people aren’t ready to hear what you’ve got to say.

As quality engineers, we often look beyond what caused the bug and ask what conditions around the work allowed it to happen. Or when people ask why the team isn’t going faster, we might ask: faster towards what outcome?

Those questions take you beyond what the team directly controls and into the wider system around them. For example, incentives, dependencies, structures and priorities. That can be uncomfortable, but it is often where the more useful changes are. Via A popular podcast guest recently proclaimed “We need systems thinkers!” But there’s a problem | John Cutler

The visible changes are the easy bit

This is from 2024 and one of the big three management consultancies, but it identifies some behaviours companies get wrong when trying to become more agile:

...they do not make the more difficult and substantive changes, such as establishing customer-facing business units fully aligned to value streams, setting up enabling supporting structures that own internal products and platforms, and resetting incentives and rewards.

They have a tendency to focus on the visible aspects of agile, things like tribes, squads, and ceremonies. The deeper changes are much harder because they need buy-in from across the organisation. Many agile transformations focus on tech teams because that’s what they can change.

But the point of the transformation should be to help the organisation achieve better outcomes, not just change how the team works. So judging it by the practices that changed only tells you people are doing things differently. What really counts is whether those changes are shifting the outcomes in a better direction.

Which brings me back to quality engineering. Are we just changing our practices, or are those practices actually changing the conditions that create quality? Via Why Companies Get Agile Right—and Wrong | BCG

Structured debriefs can improve performance by 20–25%

Just in case you needed something to remind people why retros are worth doing. From a 2013 meta-analysis:

- Debriefs are a quick, effective tool for improving team and individual performance.
- Meta-analytic results from 46 independent samples show that debriefs improve performance an average of 20% to 25%. Debriefs work equally well for teams and individuals.
- Debriefs work best when properly aligned: If the goal is to improve team performance, debriefs should be conducted with, measure, and focus on teams rather than individuals (and vice versa).
- Findings suggest that debriefs are even more effective when structured and facilitated.

While the paper doesn’t give you a step-by-step format for running a debrief, it does highlight some of the things that appear to make them work.

Focus on a specific event rather than general strengths and weaknesses. Make the intent learning and improvement, rather than judgement, appraisal or blame. Let the participants do the reflecting and sense-making rather than telling them what went right or wrong. Bring together different perspectives and data to build a clearer picture of what happened.

Retros are a great way to help teams learn from past events, so being able to step in and facilitate one as a QE feels like a really useful skill. It’s also a practical way to model the kinds of learning behaviours that help teams improve. Via Do Team and Individual Debriefs Enhance Performance? A Meta-Analysis

Build to learn and building to earn

strong product teams use AI very differently when they are building to learn (product discovery) versus building to earn (product delivery).

I really like Marty Cagan's idea of building to learn and building to earn. Building to learn is all about prototyping and discovering what the best solution to the problem you’ve identified is.

Whereas building to earn is about creating production-grade software thats reliable, observable, performant and can handle hundreds of use cases and complex business logic.

I like this framing because it makes explicit what each phase of building is trying to accomplish. Building to learn means the solution doesn’t need to be production-grade. It needs to be good enough to learn whether something works and safe enough to throw away without harming users or the business. Building to earn is about taking what you’ve learned and turning it into something that can deliver value reliably at scale.

And when building to learn involves putting something in front of real users or experimenting in production, the ability to detect, contain and reverse changes becomes even more important.

It lets you learn quickly without assuming you’ve already thought of everything that could go wrong. Via The AI Productivity Paradox | Silicon Valley Product Group

Is the code reuse argument dead?

With AI capable of producing code more cheaply and quickly than before, does code reuse matter less? Well, according to Russell Rutledge from the InnerSource Commons Foundation, no:

You don’t get one duplicate. You get a sprawl of near-identical, subtly inconsistent implementations multiplying across the codebase, and inconsistency is exactly where outages and security holes live.

He’s got a point: you’re better off reusing the one auth module that has all the edge cases that your company has discovered and built in rather than each team having to rediscover each one. So what do we do? Well, it’s about using the practices that InnerSource has been advocating all along, except now they’re not just useful for people, but AI too:

Every readiness practice we ever preached - a clear README, real tests, a good-first-issue, a welcoming CONTRIBUTING guide - was designed to help a capable stranger land a change without a personal tour.

Those readiness practices also reduce the uncertainty when someone else needs to change the code. Good tests, documentation and contribution guidance make more of the context around a component visible.

As QEs, advocating for that kind of readiness feels increasingly useful. It helps teams share the knowledge they’ve already built up, not just with new starters but with the AI tools they’re using too. Via Why AI Needs InnerSource

---

Enjoyed this Linky? Subscribe to get future Quality Engineering posts and Linkys in your inbox.

Thanks for reading and subscribing. If you’d like to support the newsletter and get access to my deeper posts and case studies, you can upgrade to paid.

---

Past Linkys