Rethink Your Understanding

Transforming Software Delivery

AI

AI in Software Delivery: Targeting the System, Not Just the Code

by

7 min read

This article is a follow-up to my earlier post, AI Is Improving Software Engineering. But It’s Only One Piece of the System. In that post, I explored how AI is already helping engineering teams work faster and better, but also why those gains can be diminished if the rest of the delivery system lags.

Here, I take a deeper look at that system-wide perspective. Adopting AI is about strengthening the entire system. We need to think about AI not only within specific teams but across the organizational level, ensuring its impact is felt throughout the value stream.

AI has the potential to improve how work flows through every part of our delivery system: product, QA, architecture, platform, and even business functions like sales, marketing, legal, and finance.

If you already have robust delivery metrics, you can pinpoint exactly where AI will have the most impact, focusing its efforts on the actual constraints rather than “speeding up” work at random. But for leaders who don’t yet have a clear set of system metrics and are still under pressure to show AI’s return on investment, I strongly recommend starting with a platform or framework that captures system delivery performance.

In my previous articles, I’ve outlined the benefits of SEI (Software Engineering Intelligence) tools, DORA metrics (debatable), and, ideally, Value Stream Management (VSM) platforms. These solutions measure and visualize delivery performance across the system, tracking indicators like cycle time, throughput, quality, and stability. They help you understand your current performance and also enable you to attribute improvements, whether from AI adoption or other changes, to specific areas of your workflow. Selecting the right solution depends on your organizational context, team maturity, and goals, but the key is having a measurement foundation before you try to quantify AI’s impact.

The Current Backlash and Why We Shouldn’t Overreact

Recent research and commentary have sparked a wave of caution around AI in software engineering.

A controlled trial by METR (2025) found that experienced developers using AI tools on their repositories took 19% longer to complete tasks than without AI, despite believing they were 20% faster. The 2024 DORA report found similar patterns: a 25% increase in AI adoption correlated with a 1.5% drop in delivery throughput and a 7.2% decrease in delivery stability. Developers felt more productive, but the system-level metrics told another story.

Articles like AI Promised Efficiency. Instead, It’s Making Us Work Harder (Afterburnout, n.d.) point to increased cognitive load, context switching, and the need for constant oversight of AI-generated work. These findings have fed a narrative that AI “isn’t working” or is causing burnout.

But from my perspective, this moment is less about AI failing and more about a familiar pattern: new technology initially disrupts before it levels up those who learn to use it well. The early data reflects an adoption phase, not the end state.

Our Teams’ Approach

Our organization is embracing an AI-first culture, driven by senior technology leadership and, additionally, senior engineers who are leading the charge, innovating, experimenting, and mastering the latest tools and LLMs. However, many teams are earlier in their adoption journey and can feel intimidated by these pioneers. In our division, my focus is on encouraging, training, and supporting engineers to adopt AI tools, gain hands-on experience, explore use cases, and identify gaps. The goal isn’t immediate mastery but building the skills and confidence to use these tools effectively over time.

Only after sustained, intentional use, months down the line, will we have an informed, experienced team that can provide meaningful feedback on the actual outcomes of adoption. That’s when we’ll honestly know where AI is moving the needle, and where it isn’t.

How I Respond When Asked “Is AI Working?”

This approach is inspired by Laura Tacho, CTO at DX, and her recent presentation at LeadDev London, How to Cut Through the Hype and Measure AI’s Real Impact (Tacho, 2025). As a leader, when I face the “how effective is AI?” debate, I ground my answer in three points:

1. How are we performing

We measure our system performance with the same Flow Metrics we used before AI: quality, stability, time-to-value, and other delivery health indicators. We document any AI-related changes to the system, tools, or workflows so we can tie changes in metrics back to their potential causes.

2. How AI is helping (or not helping)

We track where AI is making measurable improvements, where it’s neutral, and where it may be introducing new friction. This is about gaining an honest understanding of where AI is adding value and where it needs refinement.

3. What will we do next

Based on that data and team feedback, we adjust. We expand AI use where it’s working, redesign where it’s struggling, and stay disciplined about aligning AI experiments to actual system constraints.

This framework keeps the conversation grounded in facts, not hype, and shows that our AI adoption strategy is deliberate, measurable, and responsive.

What System Are We Optimizing?

When I refer to “the system,” I mean the structure and process by which ideas flow through our organization, become working software, and deliver measurable value to customers and the business.

Using a Value Stream Management and Product Operating Model approach together gives us that view:

  • Value stream: the whole journey of work from ideation to delivery to customer realization, including requirements, design, build, test, deploy, operate, and measure.
  • Product operating model: persistent, cross-functional teams aligned to products that own outcomes across the lifecycle.

Together, these models reveal not just who is doing the work, but how it flows and where the friction is. That’s where AI belongs, improving flow, clarity, quality, alignment, and feedback across the system.

The Mistake Many Are Making

Too many organizations inject AI into the wrong parts of the system, often where the constraint isn’t. Steve Pereira’s It’s time for AI to meet Flow (Pereira, 2025) captures it well: more AI output can mean more AI-supported rework if you’re upstream or downstream of the actual bottleneck.

This is why I believe AI must be tied to flow improvement:

  1. Make the work visible – Map how work moves, using both our existing metrics and AI to visualize queues, wait states, and handoffs.
  2. Identify what’s slowing it down – Use flow metrics like cycle time, WIP, and throughput to find constraints before applying AI.
  3. Align stakeholders – AI can synthesize input from OKRs, roadmaps, and feedback, so we’re solving the right problems.
  4. Prototype solutions quickly – Targeted, small-scale AI experiments validate whether a constraint can be relieved before scaling.

Role-by-Role AI Adoption Across the Value Stream

AI isn’t just for software engineers, it benefits every role on your cross-functional team. Here are just a few examples of how it can make an impact. There are many more ways for each role than listed below.

Product Managers / Owners

  • Generate Product Requirements Documentation
  • Analyze customer, market, and outcome metrics
  • Groom backlogs, draft user stories, and acceptance criteria.
  • Summarize customer feedback and support tickets.
  • Use AI to prepare for refinement and planning.

QA Engineers

  • Generate test cases from acceptance criteria or code diffs.
  • Detect coverage gaps and patterns in flaky tests.
  • Summarize PR changes to focus testing.

Domain Architects

  • Visualize system interactions and generate diagrams.
  • Validate design patterns and translate business rules into architecture.

Platform Teams

  • Generate CI/CD configurations.
  • Enforce architecture and security standards with automation.
  • Identify automation opportunities from delivery metrics.

InfoSec Liaisons

  • Scan commits and pull requests (PRs) for risky changes.
  • Draft compliance evidence from logs and release data.

Don’t Forget the Extended Team

Sales, marketing, legal, and finance all influence the delivery flow. AI can help here, too:

  • Sales: Analyze and generate leads, summarize customer engagements, and highlight trends for PMs.
  • Marketing: Draft launch content from release notes.
  • Legal: Flag risky language, summarize new regulations.
  • Finance: Model ROI of roadmap options, forecast budget impact.

Risk and Resilience

What happens when AI hits limits or becomes unavailable? Inference isn’t free; costs will rise, subsidies will fade, and usage may be capped. Do you have fallback workflows, maintain manual expertise, and measure AI’s ROI beyond activity? Another reason for us to gain experience with these tools is to improve our efficiency and understand usage patterns.

The Opportunity

We already have the data to see how our system performs. The real opportunity is to aim AI at the constraints those metrics reveal, removing friction, aligning teams, and improving decision-making. If we take the time to learn the tools now, we’ll be ready to use them where they matter most.

What Now?

We already have the metrics to see how our system performs. The real opportunity is to apply AI purposefully across the full lifecycle, from ideation and design, through development, testing, deployment, and into operations and business alignment. By directing AI toward the right constraints, we eliminate friction, unify our teams around clear metrics, and elevate decision-making at every step.

Yes, AI adoption is a learning journey. We’ll stumble, experiment, and iterate, but with intention, measurement, and collaboration, we can turn scattered experiments into a sustained competitive advantage. AI adoption is about transforming or improving the system itself.

AI isn’t failing, it’s maturing. We’re on the rise of the adoption curve. Our challenge and opportunity is to build the muscle and culture to deploy AI across the lifecycle, turning today’s experiments into tomorrow’s engineered advantage.

For anyone still hesitant, know this: AI isn’t going away. Whether it slows us down or speeds us up, we must learn to use it well, or we risk being left behind. Let’s learn. Let’s measure. Let’s apply AI where it’s most relevant and learn to understand its current benefits and limitations. There’s no going back, only forward.

Poking Holes

I invite your perspective on my posts. What are your thoughts?

Let’s talk: phil.clark@rethinkyourunderstanding.com

References

Afterburnout. (n.d.). AI promised efficiency. Instead, it’s making us work harder. Afterburnout. https://afterburnout.co/p/ai-promised-to-make-us-more-efficient

Clark, P. (2025, July). AI is improving software engineering. But it’s only one piece of the system. Rethink Your Understanding. https://rethinkyourunderstanding.com/2025/07/ai-is-improving-software-engineering-but-its-only-one-piece-of-the-system/

METR. (2025, July 10). Measuring the impact of early-2025 AI on experienced open-source developer productivity. METR. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Pereira, S. (2025, August 8). It’s time for AI to meet flow: Flow engineering for AI. Steve Pereira. https://stevep.ca/its-time-for-ai-to-meet-flow/

State of DevOps Research Program. (2024). 2024 DORA report. Google Cloud / DORA. (Direct URL to the report as applicable)

Tacho, L. (2025, June). How to cut through the hype and measure AI’s real impact. Presentation at LeadDev London.  https://youtu.be/qZv0YOoRLmg?si=aMes-VWyct_DEWz0

AI Is Improving Software Engineering. But It’s Only One Piece of the System

by

5 min read

A follow-up to my last post Leading Through the AI Hype in R&D, this piece explores how strong AI adoption still needs system thinking, responsibility, and better leadership focus.

Leaders are moving fast to adopt AI in engineering. The urgency is real, and the pressure is growing. But many are chasing the wrong kind of improvement, or rather, focusing too narrowly.

AI is transforming software engineering, but it addresses only one part of a much larger system. Speeding up code creation doesn’t solve deeper issues like unclear requirements, poor architecture, or slow feedback loops, and in some cases, it can amplify dysfunction when the system itself is flawed.

Engineers remain fully responsible for what they ship, regardless of how the code is written. The real opportunity is to increase team capacity and deliver value faster, not to reduce cost or inflate output metrics.

The bigger risk lies in how senior leaders respond to the hype. When buzzwords instead of measurable outcomes drive expectations, focus shifts to the wrong problems. AI is a powerful tool, but progress requires leadership that stays grounded, focuses on system-wide improvement, and prioritizes accountability over appearances.

A team member recently shared Writing Code Was Never the Bottleneck by Ordep. It cut through the noise. Speeding up code writing doesn’t solve the deeper issues in software delivery. That article echoed what I’ve written and experienced myself. AI helps, but not where many think it does, “currently”.

This post builds on my earlier post, Leading Through the AI Hype in R&D That post challenged hype-driven expectations. This one continues the conversation by focusing on responsibility, measurement, and real system outcomes.

Code Implementation Is Rarely the Bottleneck

Tools like Copilot, Claude Code, Cursor, Devon, … can help developers write code faster. But that’s not where most time is lost.

Delays come from vague requirements, missing context, architecture problems, slow reviews, and late feedback. Speeding up code generation in that environment doesn’t accelerate delivery. It accelerates dysfunction.

I Use AI in My Work

I’ve used agentic AI and tools to implement code, write services, and improve documentation. It’s productive. But it takes consistent reviews. I’ve paused, edited, and rewritten plenty of AI-generated output.

That’s why I support adoption. I created a tutorial to help engineers in my division learn to use AI effectively. It saves time. It adds value. But it’s not automatic. You still need structure, process, and alignment.

Engineers Must Own Impact, Not Just Output

Using AI doesn’t remove responsibility. Engineers are still accountable for what their code does once it runs.

They must monitor quality, performance, cost, and user impact. AI can generate a function. But if that function causes a spike in memory usage or breaks under scale, someone has to own that.

I covered this in Responsible Engineering: Beyond the Code – Owning the Impact. AI makes output faster. That makes responsibility more critical, not less. Code volume isn’t the goal. Ownership is.

Code Is One Step in a Larger System

Software delivery spans more than development. It includes discovery, planning, testing, release, and support. AI helps one step. But problems often live elsewhere.

If your system is broken before and after the code is written, AI won’t help. You need to fix flow, clarify ownership, and reduce friction across the whole value stream.

Small Teams Increase Risk Without System Support

Some leaders believe AI allows smaller teams to do more. That’s only true if the system around them improves too.

Smaller teams carry more scope. Cognitive load increases. Knowledge becomes harder to spread. Burnout rises.

Support pressure also grows. The same few experts get pulled into production issues. AI doesn’t take the call. It doesn’t debug or triage. That load falls on people already stretched thin.

When someone leaves, the risk is bigger. The team becomes fragile. Response times are slow. Delivery slips.

The Hard Part Is Not Writing the Code

One of my engineers said it well. Writing code is the easy part. The hard part is designing systems, maintaining quality, onboarding new people, and supporting the product in production.

AI helps with speed. It doesn’t build understanding.

AI Is a Tool. Not a Strategy

I support using AI. I’ve adopted it in my work and encourage others to do the same. But AI is a tool. It’s not a replacement for thinking.

Use it to reduce toil. Use it to improve iteration speed. But don’t treat it as a strategy. Don’t expect it to replace engineering judgment or improve systems on its own.

Some leaders see AI as a path to reduce headcount. That’s short-sighted. AI can increase team capacity. It can help deliver more features, faster. That can drive growth, expand market share, and increase revenue. The opportunity is to create more value, not simply lower cost.

The Metrics You Show Matter

Senior leaders face pressure to show results. Investors want proof that AI investments deliver value. That’s fair.

The mistake is reaching for the wrong metrics. Commit volume, pull requests, and code completions are easy to inflate with AI. They don’t reflect real outcomes.

This is where hype causes harm. Leaders start chasing numbers that match the story instead of measuring what matters. That weakens trust and obscures the impact.

If AI is helping, you’ll see a better flow. Fewer delays. Faster recovery. More predictable outcomes. If you’re not measuring those things, you’re missing the point.

AI Is No Longer Optional

AI adoption in software development is no longer a differentiator. It’s the new baseline.

Teams that resist it will fall behind. No investor would approve a team using hammers when nail guns are available. The expectation is clear. Adopt modern tools. Deliver better outcomes. Own the results.

What to Focus On

If you lead AI adoption, focus on the system, not the noise.

  • Improve how work moves across teams
  • Reduce delays between steps
  • Align teams on purpose and context
  • Use AI to support engineers, not replace them
  • Measure success with delivery metrics, not volume metrics
  • Expect engineers to own what they ship, with or without AI

You don’t need more code. You need better outcomes. AI can help, but only if the system is healthy and the people are accountable.

The hype will keep evolving. So will the tools. But your responsibility is clear. Focus on what’s real, what’s working, and what delivers value today.

Poking Holes

I invite your perspective on my posts. What are your thoughts?

Let’s talk: phil.clark@rethinkyourunderstanding.com

References

  1. Clark, Phil. Leading Through the AI Hype in R&D. Rethink Your Understanding. July 2025. Available at: https://rethinkyourunderstanding.com/2025/07/leading-through-the-ai-hype-in-rd
  2. Ordep. Writing Code Was Never the Bottleneck. Available at: https://ordep.dev/posts/writing-code-was-never-the-bottleneck
  3. Clark, Phil. Responsible Engineering: Beyond the Code – Owning the Impact. Rethink Your Understanding. March 2025. Available at: https://rethinkyourunderstanding.com/2025/03/responsible-engineering-beyond-the-code-owning-the-impact

Leading Through the AI Hype in R&D

by

7 min read

Note: AI is evolving rapidly, transforming workflows faster than expected. Most of us can’t predict how quickly or to what level AI will change our teams or workflow. My focus for this post is on the current state, pace of change, and the reality vs hype at the enterprise level. I promote the adoption of AI and encourage every team member to embrace it.

I’ve spent the past few weeks deeply immersed in “vibe coding” and experimenting with agentic AI tools during my nights and weekends, learning how specialized agents can orchestrate like real product teams when given proper context and structure. But in my day job as a senior technology leader, the tone shifts. I’ve found myself in increasingly chaotic meetings with senior leaders, chief technology officers, chief product officers, and engineering VPs, all trying to out-expert each other on the transformative power of AI on product and development (R&D) teams.

The energy often feels like a pitch room, not a boardroom. Someone declares Agile obsolete. Another suggests we can replace six engineers with AI agents. A few toss around claims of “30× productivity.” I listen, sometimes fascinated, often frustrated, at how quickly the conversation jumps to conclusions without asking the right questions. More troubling, many of these executives are under real pressure from investors and ownership to show ROI. If $1M is spent on AI adoption, how do we justify the return? What metrics will we use to report back?

Hearing the Hype (and Feeling the Exhaustion)

One executive confidently declared, “Agile and Lean are dead,” citing the rise of autonomous AI agents that can plan, code, test, and deploy without human guidance. His opinion echoed a recent blog post, Agile Is Dead: Long Live Agentic Development, which criticized Agile rituals like daily stand-ups and sprints as outdated and encouraged teams to let agents take over the workflow¹. Meanwhile, agile coaches argue that bad Agile, not Agile itself, is the real problem, and that AI can strengthen Agile if applied thoughtfully.

The hype escalates when someone shares stories of high-output engineering from one of the senior developers, keeping up with AI capabilities: 70 AI-assisted commits in a single night, barely touching the keyboard. Another proposes shrinking an 8-person team to just two engineers, one writing prompts and one overseeing quality, as the AI agents do the rest. These stories are becoming increasingly common, especially as research suggests that AI can dramatically reduce the number of engineers needed for many projects². Elad Gil even claimed most engineering teams could shrink by 5×–10×.

But these same reports caution against drawing premature conclusions. They warn that while AI enables productivity gains, smaller teams risk creating knowledge silos, reduced quality, and overloading the remaining developers². Other sources echo this risk: Software Engineering Intelligence (SEI) tools have flagged increased fragility and reduced clarity in AI-generated code when review practices and documentation are lacking³.

What If We’re Already Measuring the Right Things?

While executives debate whether Agile is dead, I find myself thinking: we already have the tools to measure AI’s impact, we just need to use them.

In my organization’s division, we’ve spent years developing a software delivery metrics strategy centered on Value Stream Management, Flow Metrics, and team sentiment. These metrics already show how work flows through the system, from idea to implementation to value. They include:

  • Flow metrics like distribution, throughput, time, efficiency, and load
  • Quality indicators like change failure rate and security defect rate
  • Sentiment and engagement data from team surveys
  • Outcome-oriented metrics like anticipated outcomes and goal (OKR) alignment

Recently, I aligned our Flow Metrics with the DX Core 4 Framework⁴ matrix, organizing them into four key categories: speed, effectiveness, quality, and impact. We made these visual and accessible, using this clear chart to show how each metric relates to delivery health. These metrics don’t assume Agile is obsolete or that AI is the solution. They track how effectively our teams are delivering value.

So when senior leaders asked, “How will we measure AI’s impact?” I reminded them, we already are. If AI helps us move faster, we’ll see it in flow time. If it increases capacity, we’ll see it in throughput (flow velocity). If it maintains or improves quality, our defect rates and sentiment scores will reflect that. The same value stream lens that shows us where work gets stuck will also reveal whether AI helps us unstick it.

Building on Existing Metrics: The AI Measurement Framework

Instead of creating an entirely new system, I layered an existing AI Measurement Framework on top of our existing performance metrics⁵. This format includes three categories:

  1. Utilization:
    • % of AI-generated code
    • % of developers using AI tools
    • Frequency of AI-agent use per task
  2. Impact:
    • Changes in flow metrics (faster cycle time)
    • Developer satisfaction or frustration
    • Delivered value per team or engineer
  3. Cost:
    • Time saved vs. licensing and premium token cost
    • Net benefit of AI subscriptions or infrastructure

This approach answers the following questions: Are developers using AI tools? Does that usage make a measurable difference? And does the difference justify the investment?

In a recent leadership meeting, someone asked, “What percentage of our engineers are using AI to check in code?” That’s an adoption metric, not a performance one. Others have asked whether we can measure AI-generated commits per engineer to report to the board. While technically feasible with specific developer tools, this approach risks reinforcing vanity metrics that prioritize motion over value. Without impact and ROI metrics, adoption alone can lead to gaming behavior, and teams might flood the system with low-value tasks to appear “AI productive.” What matters is whether AI is helping us delivery better, faster, and smarter.

I also recommend avoiding vanity metrics, such as lines of code or commits. These often mislead leaders into equating motion with value. Many vendors boast “AI wrote 50% of our code,” but as developer-experience researcher Laura Tacho explains, this usually counts accepted suggestions, not whether the code was modified, deleted, or even deployed.⁵ We must stay focused on outcomes, not outputs.

The Risk of Turning AI into a Headcount Strategy

One of the more concerning trends I’m seeing is the concept of “headcount conversion,” which involves reducing team size and utilizing the savings to fund enterprise AI licenses. If seven people can be replaced by two and an AI license, along with a premium token budget, some executives argue, then AI “pays for itself.” However, this assumes that AI can truly replace human capability and that the work will maintain its quality, context, and business value.

That might be true for narrow, repeatable tasks, or small organizations or startups struggling with costs and revenue. But it’s dangerous to generalize. AI doesn’t hold tribal knowledge, coach junior teammates, or understand long-term trade-offs. It’s not responsible for cultural dynamics, systemic thinking, or ethical decisions.

Instead of shrinking teams, we should consider expanding capacity. AI can help us do more with the same people. Developer productivity research indicates that engineers typically reinvest AI-enabled time savings into refactoring, enhancing test coverage, and implementing cross-team improvements², which compounds over time into stronger, more resilient software.

Slowing Down to Go Fast

Leaving those leadership meetings, I felt a mix of energy and exhaustion. Many people wanted to appear intelligent, but few were asking thoughtful questions. We were racing toward solutions without clarifying what problem we were solving or how we’d measure success.

So here’s my suggestion: Let’s slow down. Let’s agree on how we’ll track the impact of AI investments. Let’s integrate those measurements into systems we already trust. And let’s stop treating AI as a replacement for frameworks that still work; instead, let’s use it as a powerful tool that helps us deliver better, faster, and with more intention.

AI isn’t a framework. It’s an accelerator. And like any accelerator, it’s only valuable if we’re steering in the right direction.

Poking Holes

I invite your perspective on my posts. What are your thoughts?

Let’s talk: phil.clark@rethinkyourunderstanding.com

References

  1. Leschorn, J. (2025, May 29). Agile Is Dead: Long Live Agentic Development. Superwise. https://superwise.ai/blog/agile-is-dead-long-live-agentic-development/
  2. Ameenza, A. (2025, April 15). The New Minimum Viable Team: How AI Is Shrinking Software Development Teams. https://anshadameenza.com/blog/technology/ai-small-teams-software-development-revolution/
  3. Circei, A. (2025, March 13). Measuring AI in Engineering: What Leaders Need to Know About Productivity, Risk and ROI. Waydev. https://waydev.co/ai-in-engineering-productivity-risk-roi/
  4. Saunders, M. (2025, January 6). DX Unveils New Framework for Measuring Developer Productivity. InfoQ. https://www.infoq.com/news/2025/01/dx-core-4-framework/
  5. GetDX. (2025). Measuring AI Code Assistants and Agents. DX Research. https://getdx.com/research/measuring-ai-code-assistants-and-agents/