Linky #14 - Feedback, Bottlenecks, and Building Quality
Why quality emerges from iteration, insight, and working within constraints
This edition is all about feedback and limits. A conference talk by Callum Akehurst-Ryan inspired me to sketch out “software feedback rings” as a way of mapping which types of feedback matter most for reducing uncertainty. I’ve also been revisiting Dr Richard Cook’s work on how systems fail, looking at why AI’s impact often stalls at bottlenecks, and reading some fascinating reflections from inside OpenAI. The thread running through all of it? Quality is rarely a product of grand design. It grows through iteration, insight, and working within the constraints of real systems.
Latest post from the Quality Engineering Newsletter
One of the talks from BCS Software Testing Conference really got me thinking about uncertainty, feedback, and how that all connects to quality in our software systems. Inspired by the Quality Radar, I came up with the idea of software feedback rings. This first post sets up the concept, and I’ll be breaking down the rings soon to show what types of feedback help us lower uncertainty in different quadrants.
From the archive
One of my favourite series to write about explored Dr Richard Cook’s 'How Complex Systems Fail'. In this first part, I connected his research to the product layer of software systems, showing how quality can be lost at the product layer of systems. Writing this series has let me dive deep into Cook’s work and link it back to modern software. It’s really helped crystallise how I think about building quality in today.
How LLM work in 18 tweets
LLMs work because they:
learn from massive text via self-supervision
use Transformers to model token sequences
can be prompted/fine-tuned for any task
are aligned with human preferences
are optimized for fast inference
they're general-purpose text reasoning machines.
A good, simple thread on how LLMs work. Worth a read. Understanding this foundation helps us see both the power and the limits of these systems in real-world quality engineering via [what are large language models actually doing?](Title Unavailable | Site Unreachable)
Working smart is based on insight
Smarter approaches aren’t universal shortcuts, they’re contextual insights uncovered through deep, sustained effort.
Sometimes you just have to do the graft to figure out what the insights are. Via Exceptional Looks Boring
I’m speaking at a few conferences this year and the first is Lean Agile Scotland in Edinburgh. I started attending more non-testing focused conferences a few years back to open myself up to new ideas. LAScot is a great conference for doing just that!
See the programme for all the other great talks and use 10Jitesh to get 10% off.
AI doesn’t make organisations faster
Some of the findings:
∙ Developers using AI complete 21% more tasks and merge 98% more PRs
∙ But review times are up 91%, PRs are 154% bigger, and bugs per dev up 9%
∙ Delivery metrics (lead time, change failure rate) are flat
∙ At the company level: no measurable impactWhy? Because coding isn’t the bottleneck.
The report itself says it: "Downstream bottlenecks cancel out gains". Review, testing, and deployment can’t keep pace, so the system jams.
This is textbook theory of constraints (ToC): the bottleneck in this case is not the amount of work a developer can do, but rather whether their output meets our needs. To achieve gains with AI (or any process or tool) it must help alleviate the constraints within the delivery process. If improvements are made elsewhere, the work will still get caught at that bottleneck via Yet another study showing that AI-assisted coding isn’t resulting in any meaningful productivity benefits | Rob Bowley
I also dived deeper into what ToC is and how to apply it in past posts.
Reflections on OpenAI
Fascinating take from someone who worked at ChatGPT. Lots of little nuggets but a few that stood out:
“Good ideas can come from anywhere, and it's often not really clear which ideas will prove most fruitful ahead of time. Rather than a grand 'master plan', progress is iterative and uncovered as new research bears fruit.”
Quality can be built through iteration, not grand design: In software teams, quality often comes from experimentation and refinement, not from following a perfect upfront plan. Building quality in means embracing discovery and fast feedback loops.
Teams vary significantly in culture: some are sprinting flat-out all the time, others are babysitting big runs, some are moving along at a much more consistent pace. There's no single OpenAI experience, and research, applied, and GTM operate on very different time horizons.”
Culture can drives quality outcomes: Quality engineering practices need to flex to different team cultures and time horizons (research vs applied vs GTM). There is no one-size-fits-all. Quality grows when practices are adapted to context.
“There's a strong bias to action (you can just do things). It wasn't unusual for similar teams but unrelated teams to converge on various ideas.”
Bias to action can accelerate or compromise quality: This autonomy sparks innovation and speed, but without guardrails (CI, tooling, testability), it risks fragile systems. Building quality in requires balancing freedom to act with strong safety nets.
“There were a few areas where having a rapidly scaled eng team and not a lot of tooling created issues.”
Tooling and infrastructure matter for sustainable quality: A strong quality engineering foundation (fast, reliable CI/CD, solid test infra, observability) is critical when working at scale. Otherwise, growth outpaces the ability to maintain quality.
"[Codex was built in 7 weeks by] ~8 engineers, ~4 researchers, 2 designers, 2 GTM and a PM. Had we not had that group, I think we would've failed. Nobody needed much direction, but we did need a decent amount of coordination.”
Senior, cross-functional teams can create quality under pressure: Small, experienced, cross-functional groups can deliver high-quality outcomes quickly. Quality is a team sport, not just a software engineering responsibility.
“Nearly everything is a rounding error compared to GPU cost… a niche feature that was built as part of the Codex product had the same GPU cost footprint as our entire Segment infrastructure.”
Quality is shaped by constraints and costs: Quality engineering includes making systems not only correct and reliable but also cost-aware and efficient. Quality isn’t just about fewer bugs, it’s about optimising within constraints.
“Safety is actually more of a thing than you might guess… There's a large number of people working to develop safety systems.”
“Given the nature of OpenAI, I saw more focus on practical risks (hate speech, abuse, manipulating political biases, crafting bio-weapons, self-harm, prompt injection) than theoretical ones.”
Safety and risk management are part of quality: Building quality in isn’t just about technical excellence. It’s also about anticipating risks, protecting users, and embedding ethical safeguards.
Close
Each of these pieces reinforces a lesson I keep coming back to: building quality in isn’t about speed alone. It’s about feedback, context, and working within constraints.
I’d love to hear what stood out most for you. Reply to this email or leave a comment.
Past Linky Editions
Linky #13 - From AI Shortcuts to Expert Generalists
This week’s Linky has a bit of everything. From how AI “explains” itself (and why that’s not the full story) to what it means to be an expert generalist. There’s also something on improving what’s in front of you, chasing your way of working rather than titles, and why you should learn like an athlete.
Linky #12 - Confidence, Curiosity, and Cleaning Up Code
This week’s link covers: The role of confidence in QA conversations, what happens when software systems start testing themselves, and how AI agents might reshape the work of engineers. There’s also a look at why curiosity is a key quality engineering mindset, how we frame quality conversations, and a reminder that writing code has never really been the bottleneck.
Linky #11 - Tools Don’t Think. Engineers Do.
This week’s link covers: Reflections from Agile Manchester, how good habits shape team behaviour, why LLMs need context, not just prompts, what makes testing meaningful, and how metrics shape (or warp) team incentives