Three Ideas for Quality Engineers from the Testing Peers Conference
Three ideas from the Testing Peers conference 2024 that can help Quality Engineers develop their knowledge of quality engineering in building, maintaining, and losing quality.
By sharing my posts with your friends and colleagues, you are helping me grow and inspiring others to learn and grow with us. Let's come together and create a community where everyone can benefit from each other's knowledge and experiences.
I was lucky to speak at the first Testing Peers Conference in Nottingham this March and saw a few of the talks. I'm a big note-taker and usually create a summary of the talks I attended and my main takeaways, but this time, I'd like to try something a little different.
Instead of the usual summary, I will focus on three ideas from the conference that can help Quality Engineers (QEs) develop their knowledge of quality engineering in building, maintaining, and losing quality.
1] Building quality in by focusing on shippable risk
Heather Reid kicked off the conference with her keynote talk Wait! That's Not Tested, which was about helping testers balance value and feedback. Heather made some excellent points about failure, what testers do and how they need to be included in the full software life cycle. So if you can attend this talk, it'll be well worth your time. But what caught my eye was how her company used MSR over MVPs.
Minimum Viable Products (MVP) Vs Minimum Shippable Risk (MSR)
MVPs are a way for companies to share versions of their products with just enough features to get feedback on whether the product is what the customer wants. Essentially, they are a way to build something, measure what it did, learn, and go back around the loop until you've created the outcome you want.
However, as Heather explained, some key stakeholders' views on minimum can vary greatly, and companies end up shipping close to fully featured products. Therefore, their teams spend too long in the build phase and not enough on the learning stage. Spending too long on building and not shipping leads to large batches of delivering value, which goes against the Lean manufacturing principle of creating flow and, in this case, getting towards single-piece flow1, resulting in higher costs and less value delivered. So, to combat this, they reframed MVPs to Minimum Shippable Risk.
What I like about MSR reframing is that it gets people thinking about the smallest risk the stakeholders are willing to accept when shipping a working product. MSR pulls the conversations about what is and isn't acceptable to key stakeholders up front and centre and gets them thinking about what matters to them. This way, the engineering team finds out as early as possible, even before going into the build, measure, and learn cycle, what they'll need to build to ship value. This approach differs from what tends to happen with MVPs, where teams get stuck in the build cycle as each stakeholder increases the scope of the viable product.
Reframing to MSR helps QEs understand what stakeholders think the risks of a product are and, therefore, a proxy measure of what quality means to these people. By finding ways to lower the uncertainty around these risk areas, QEs can help teams build quality in and understand what they need to do to maintain it going forward. Another valuable aspect of this is that the information from the learning cycle can now be tailored towards the risks the stakeholders perceive, helping them understand how the market views their product.
2] Building quality in through layers within complex systems
Andrew Brown's talk Rethinking testing and risk within complex systems – the Swiss Cheese used a great story of the Gimli Glider, a Boeing 767 that ran out of fuel midway through a flight at 41,000 feet. The crew glided the plane back down to a converted airfield that at the time had a live race with 100s of spectators. Not only were they successful in gliding such a large plane, but they suffered only minor injuries to passengers and spectators, and the damage to the plane was repaired and remained in service for another 25 years.
Andrew used the Gimli Glider story to explain how different layers within the complex system of aviation all had to line up just so as to cause a catastrophic failure (running out of fuel mid-flight). This failure resulted from multiple human errors, but the irony of the situation is that the worst case (loss of human life) was averted due to humans in the process (and a lot of luck). Therefore, humans are the causes of failure but also the mitigators of them.
Swiss cheese model
Where the Swiss cheese model comes in is that it likens each layer in the system to a slice of cheese with holes in it. Each of the layers acts as a defence against failures passing through. Occasionally, some issues will line up with the holes in the layers and result in that failure getting through. This brilliant interactive article from BBC News on why vacancies alone will not stop the spread of COVID explains the Swiss cheese model really well and is worth a look.
QEs can use the Swiss cheese model to help engineering teams understand how building in quality at multiple system levels can lower the uncertainty that issues will occur in the live environment. But it's impossible to reduce that uncertainty to zero, and software systems can still fail even when teams have built-in and inspected for quality. By showing them:
Each layer can be a line of defence against different types of issues.
For example, techniques like pair programming, unit/integration/end-to-end tests, exploratory testing, etc., can all help capture issues before systems are made live to end users.
The more layers you have, the more likely issues will be caught at one of the defences. However:
Each layer can come at the cost of slowing down and releasing value, so teams must balance speed and quality.
No layer is perfect at catching all the issues; some will pass through to the end users.
Build quality into the system by enabling engineering teams to detect, capture, and correct issues at the layers where they are easiest and cheapest to correct:
Limiting the impact of issues in the live environment, using techniques such as feature flags, staged rollouts, monitoring, etc, to detect and limit the harm they can cause.
Carry out post-incident reviews to understand better what parts of our defence the issues were able to sidestep.
Update our defences to detect and alert us of the issue in the future.
Share this knowledge with other teams and QEs to help others learn from your system failures.
Andrew also referenced Richard Cook's paper How complex systems fail. This paper is a treasure trove of ideas that I plan to explore in future posts.
3] Building quality in through collaborative cultures
Sanne Visser's talk That which isn't good for the hive, isn't good for the bee was an excellent talk on why we need to limit competition and focus on cooperation and collaboration in our teams. Collaborative environments are where learning organisations thrive and new ideas can grow.
Sanne used a great analogy of honey bees and how their colonies work for the greater good of the hive and its survival, not for any individual, including the queen. Once the existing queen begins to slow in the production of eggs, the drones will begin feeding larvae royal jelly, allowing them to develop into a queen bee. Once the new queen bee emerges, the hive will swarm to find a new colony site. This new location is found by scout bees, who will do a special dance once they find a suitable site. Other bees will follow the scot and dance, too, if they feel the new site for the colony is suitable. If enough bees join, the whole hive will move to the new site and set up a new colony. The only way a hive can function and continue to survive is if every bee does the role they are assigned. If there were competition between bees, the hive would be unlikely to function at the scale they do (anywhere between 20,000 and 80,000 bees, depending on the season).
The honey bee analogy got me thinking about quality being everyone's responsibility in engineering teams. Whole team responsibility for quality will only work in team cultures focused on cooperation and collaboration rather than teams that are more adversarial and focused on individuals and groups being the best.
Quality is everyone's responsibility
Team members who focus on being the best are more likely to be competitive, which can lead to transactional-based relationships where people only do things if they get things in return. It will also likely lead to more information hoarding and stealing, where information can be weaponised to push you forward and hold others back. Individuals wanting to be "the best" at what they do further fuels competition, creating winners and losers, and no one wants to be a loser. All this competition creates stress which has all sorts of health implications. These are the environments where people say quality is everyone's responsibility but ends up being no ones as there is no incentives to help others in creating quality that goes beyond the individual.
As QEs, we need to be helping to incentivise environments that are leaning towards cooperation and collaboration. It's not about QEs owning quality but working to make it everyone's responsibility by assisting others to see how their skills help build quality at their work levels. These discussions are a great place to introduce ideas such as the Swiss cheese model to help disciplines see how they can create layers of defences against issues. As QEs, we should work with our leadership teams and help reward cooperation and collaboration by calling it out publicly, showing the benefits it's bringing to teams and how others can do the same.
Honorary mentions
Claire Reckless and Al Goodall gave two talks on leadership. While building quality can often seem more about practitioners than leadership, developing an understanding of the problems leaders face can be incredibly valuable, especially when you need more traction with your efforts. Your leadership may be facing other significant issues, so quality may take a back seat.
The amazing Leigh Rathbone delivered a fantastic second keynote of the day. This talk was the only one I took no notes on, as I just wanted to enjoy Leigh doing what he does so well: telling stories in his high-energy style. He said it was his last talk ever. I'm keeping my fingers crossed this isn't the case, as everyone in the quality engineering community will miss out.
Beth Clarke delivered the day's final talk on DevOps and how testers can be a part of the process, not pushed out of it. This slide sums it up brilliantly.
Should you attend Testing Peers 2025?
Overall, the Testing Peers conference was a huge success, and even more so considering it was their first conference. They've already announced that they'll be back again next year, so if this year was anything to go by, I highly recommend attending. I personally got a lot out of the talks.
This is a free post 🔓 so if you find it helpful, feel free to forward it on, as others may do, too. If you’re not already a subscriber, sign up so you can stay up-to-date with the latest Quality Engineering posts.
To learn more about single-piece flow, check out Lean Lesson: Batching Is Bad—Right?, which explains the idea using a simple example of making salads.