16 min read
![](https://rethinkyourunderstanding.com/wp-content/uploads/2024/08/WizardMetrics-1-1024x1024.jpg)
This article is designed to help senior leaders navigate the landscape of digital product delivery metrics platforms and tools. It offers insights into the various options available and the rationale for selecting one solution over another when determining the most suitable metrics platform.
In the rapidly evolving landscape of software engineering and digital product delivery, leaders are increasingly reevaluating their approaches to team performance, productivity, and metrics. These leaders are turning to quantitative and qualitative data to make delivery teams’ performance and efficiency improvement efforts more transparent. Teams seek data-driven insights to identify bottlenecks, uncover root causes, and evaluate potential enhancements. They aim to eliminate or refine these obstacles by experimenting with various changes to achieve greater efficiency.
Embracing new metrics for today’s work environment is not immune to Goodhart’s law and the potential pitfalls of gamification. Therefore, leaders and delivery teams must embrace these metrics differently and separate business discussions and team conversations when analyzing and utilizing them.
However, having data insights and optimizing the performance of your delivery team is not just a necessity but also a gateway to capturing performance insights, maintaining competitiveness, and enhancing processes, practices, automation, and team design to deliver value effectively. The potential benefits of these platforms are vast, and with the right approach, you can significantly improve your digital product delivery.
As a senior leader with extensive experience guiding technology teams through digital transformation, I have witnessed firsthand the vital role of selecting the right metrics platform. During 2020 and 2021, I faced a similar challenge while assessing and implementing a metrics platform to gain insights into our digital transformation efforts, team structure, and automation investments. Leveraging my background, I quickly gained a deeper understanding of modern metrics.
At that time, a senior leader in our division struggled to grasp the various metrics options and their relevance to our current delivery capabilities, often fixating on vendor pricing—a legitimate concern given our tight budget. This misunderstanding led to friction as we aimed to identify the best solution. While I acknowledged the advantages of different platforms, the leader’s emphasis on cost over functionality complicated our decision-making process.
After more than eight years of investing in our architecture, team design, automation, and delivery processes, we arrived at a critical decision point: Should we invest in a Software Engineering Intelligence (SEI) platform tailored for delivery, or should we pursue a comprehensive Value Stream Management (VSM) solution that encompasses the entire lifecycle from ideation to production? It’s important to note that the concept of an SEI platform didn’t even exist three years ago when we began our evaluation. At that time, we only had DORA and various vendors offering metrics platforms that needed more coverage across all stages, focused solely on parts of the delivery process, or had notable gaps in delivery insights, even among holistic solutions. My goal was to compare all vendor options, assess their coverage and focus, identify gaps across each workflow phase, and select the best fit.
We initially assessed ten vendors and ultimately narrowed our options to two: our preferred SEI solution and our top choice for the VSM platform. If budget constraints had not been an issue, I would have acquired both solutions to bridge the gaps between them seamlessly. I even lightheartedly suggested a potential merger for both vendors! We chose a VSMP solution based on where we assessed our bottlenecks and stage. This experience also inspired one of my early articles on the subject – Finally, Metrics That Help. Contact me if you’d like to explore my experiences, methodologies, discussions, and presentations related to this evolution in metrics.
Fast forward to 2024, and we discover the inspiration behind this article. I learned about a senior technology leader tasked with transforming and enhancing the large technology team he inherited upon joining an enterprise company. His primary objective is to turn the department around and to improve digital product delivery. He aims to reintegrate QA and product delivery responsibilities into teams—responsibilities removed during previous cost-cutting measures implemented by earlier stakeholders before he arrived in the organization. Based on my understanding, part of his strategy includes adopting delivery performance metrics to establish team metrics that will create a baseline for current performance, offer valuable insights, and initiate a pathway to enhancing overall performance and efficiency. He has focused explicitly on or selected DORA metrics to better understand product delivery roles and processes. Although I didn’t have the opportunity to discuss this with the leader before his decision, I am intrigued by how he arrived at the choice of DORA metrics. Given that this enterprise technology team has been delivering work effectively in an agile environment for years, I wonder if his decision truly addresses the more significant challenges faced by his division in improving the delivery process. Are these the appropriate metrics and feedback mechanisms to resolve workflow bottlenecks?
Stages of the Product Delivery Workflow
Once you know the workflow, you can analyze and pinpoint its bottlenecks. This awareness enables you to implement metrics that offer valuable feedback and insights on these bottlenecks, helping you identify areas for improvement.
The stages of the digital product workflow, often discussed in frameworks such as Value Stream Management (VSM) and Flow Engineering advocated by Steve Pereria and Andrew Davis and outlined in Mik works, like Project to Product, typically follow the journey from an initial idea to the moment the product delivers value to the customer. Understanding these phases is essential for grasping how work flows within an organization and identifying where value is created.
![](https://rethinkyourunderstanding.com/wp-content/uploads/2024/08/DigitalProductWorkflowStages.jpg)
The key stages or phases of the digital product workflow are:
- Ideation (or Discovery)
- Delivery
- Operations
- Support
Ideation (or Discovery): This phase focuses on generating, prioritizing, and refining ideas. It involves understanding what should be developed based on customer needs, market demands, and strategic objectives. This stage lays the groundwork for all future efforts and encompasses market research, gathering user feedback, and creating early design prototypes.
Delivery: Once an idea is validated, it transitions into the delivery phase, where actual development occurs. This includes coding, testing, and deploying the product. This phase emphasizes transforming ideas into tangible products ready for customer delivery. It also encompasses the CI/CD pipeline, where DORA metrics are often utilized.
Operations: Once the product is delivered, it transitions into the operations phase. This stage focuses on maintaining and managing the product within a live environment. Essential activities during this phase include performance monitoring, infrastructure management, and ensuring the product operates smoothly and efficiently.
Support: The final stage is support, encompassing customer assistance, incident management, and ongoing enhancements based on user feedback. This phase ensures that customers derive continued value from the product while effectively addressing any issues arising after deployment.
Value Stream Management resources, Product Flow, and Flow Engineering often highlight these stages. They offer a structured framework for understanding the workflow, guiding it from the initial idea to continuously delivering value to customers.
These stages enable organizations to grasp how value is generated and delivered, facilitating more effective management of the entire product lifecycle.
Understanding the Types of Metrics Platforms
Before exploring specific tools, it’s essential to grasp the three main categories of qualitative metrics platforms focused on delivery performance insights as I understand them: DORA metrics, Software Engineering Intelligence (SEI) platforms, and Value Stream Management (VSM), along with flow metrics.
DORA Metrics
DORA (DevOps Research and Assessment) metrics are designed to measure the effectiveness and efficiency of your software delivery process from code commit to deployment. These metrics include:
- Change lead time
- Deployment frequency
- Change fail percentage (previously change failure rate)
- Failed deployment recovery time (previously mean time to recovery or MTTR)
The information regarding throughput, stability, and metric definitions is sourced from the dora.dev website.1
Throughput
Throughput measures the velocity of changes that are being made. DORA assesses throughput using the following metrics:
- Change lead time – This metric measures the time it takes for a code commit or change to be successfully deployed to production. It reflects the efficiency of your delivery pipeline.
- Deployment frequency – This metric measures how often application changes are deployed to production. Higher deployment frequency indicates a more agile and responsive delivery process.
Stability
Stability measures the quality of the changes delivered and the team’s ability to repair failures. DORA assesses throughput using the following metrics:
- Change fail percentage – This metric measures the percentage of deployments that cause failures in production, requiring hotfixes or rollbacks. A lower change failure rate indicates a more reliable delivery process.
- Failed deployment recovery time – This metric measures the time it takes to recover from a failed deployment. A lower recovery time indicates a more resilient and responsive system.
Why Choose DORA?
The bottleneck is deployment.
- Focused on Delivery Efficiency: DORA metrics are ideal for organizations that need to optimize their CI/CD pipeline and improve deployment frequency and stability.
- Narrow Scope, High Impact: DORA metrics can provide precise insights to drive improvements if your primary bottlenecks relate to deployment.
Software Engineering Intelligence (SEI) Platforms
In recent years, Software Engineering Intelligence (SEI) platforms have evolved into a distinct category, broadening the spectrum of metrics they address beyond those defined by DORA. While DORA metrics primarily assess the efficiency of the CI/CD pipeline from code commit to deployment, SEI platforms take a more comprehensive approach by focusing on the entire delivery process. This includes everything from when a work item enters the product backlog to its eventual deployment in production.
SEI platforms offer a more comprehensive array of metrics encompassing the traditional DORA metrics and additional measures such as a broader definition of cycle time, throughput, work-in-progress (WIP), and quality indicators like pull request size and rework rates. These platforms prioritize integrating diverse engineering tools—including Git, CI/CD, and project management—to provide a cohesive view of the delivery process.
One of the primary distinctions between DORA and SEI metrics lies in SEI’s emphasis on actionable insights and benchmarking. SEI platforms frequently utilize data science to create benchmarks that enable teams to assess their performance against industry standards, a crucial factor for establishing informed improvement goals. Furthermore, SEI platforms are designed to enhance visibility into business value metrics, aligning engineering initiatives with overarching business objectives—an aspect that traditional DORA metrics often overlook.
A key feature of SEI platforms (and VSM platforms) is their focus on enhancing developer experience (DX) and promoting organizational efficiency. Many of these platforms assess developer productivity, which often sparks debate and varies by context—particularly when distinguishing between team productivity and individual performance. They also track health and sentiment through metrics such as work-in-progress (WIP) and burnout alerts. These metrics are vital for sustaining a healthy and efficient engineering team.
When comparing DORA and SEI platforms, it becomes evident that DORA offers valuable insights into the deployment phase. In contrast, SEI platforms provide a more comprehensive perspective on the broader delivery process, making them better suited for organizations aiming to optimize their operations beyond just CI/CD.
Considering these distinctions, your organization’s decision between DORA and SEI should be guided by its current bottlenecks and the specific challenges it aims to tackle. DORA metrics may be adequate if your primary obstacles lie within the deployment phase. However, an SEI platform would be the most suitable if your goal is to optimize the entire delivery pipeline and better align engineering efforts with business objectives.
Gartner has recently published a market guide on SEI platforms for those interested in exploring this further.
Here are some key types of metrics commonly found in some of the more known SEI platforms :
- Cycle Time: Measures the time it takes for a work item to move from the backlog to deployment. This metric helps identify inefficiencies in the development process and highlights areas where delays occur.
- Throughput: This metric tracks the number of work items completed over a specific period. It provides insight into team productivity and helps assess whether teams meet their delivery goals.
- Work in Progress (WIP): Monitors the number of items being worked on or in progress. High WIP can indicate bottlenecks or overloading of the team, which can slow overall delivery.
- Pull Request (PR) Size and Rework Rate: Evaluate the size of pull requests and the frequency of rework required. Smaller PRs with lower rework rates tend to be merged faster and with fewer issues, contributing to smoother deployments.
- Planning Accuracy: This compares planned versus actual delivery timelines. It helps teams understand how well they estimate their work and whether they consistently meet their commitments.
- Developer Experience (DX): SEI platforms often include metrics that assess developer satisfaction and efficiency, such as time spent in meetings versus coding and the overall impact of tools and processes on developer productivity.
Why Choose SEI?
Your bottleneck remains in the workflow’s delivery phase, but you have the budget for an SEI solution and need insights beyond code commit to delivery.y.
- Optimizing Development and Delivery: SEI platforms are perfect for teams looking to streamline the development and delivery process from backlog to production, balancing speed with quality and introducing team health insights.
- Broader Than DORA, Narrower Than VSM: SEI platforms offer a middle ground, providing insights into the development process without tracking the entire lifecycle.
Value Stream Management (VSM) Platforms and Flow Metrics
VSM offers an end-to-end perspective, focusing on the entire digital product lifecycle—from ideation through delivery to operations and support. The core idea behind VSM is to visualize, measure, and optimize the flow of value across all stages of this lifecycle. VSM platforms are designed to provide deep insights into how value is delivered across complex and interconnected systems, identifying inefficiencies and bottlenecks not just in the development process but across the entire organization. These platforms are ideal for companies looking to align their technical and business objectives, ensuring that every part of the process contributes to delivering value to the customer.
Organizations ready to adopt VSM typically have a mature or highly performant delivery practice. They have likely already optimized their CI/CD pipelines, backlog refinement (grooming), planning and technical analysis and design, and task breakdown practices, and seek to improve the overall flow of work across different teams and departments. These organizations often deal with complex value streams that require integration across multiple tools and departments. VSM helps them visualize and manage these streams to optimize the delivery of value to customers.
Organizations that are prepared to adopt Value Stream Management (VSM) typically possess a mature and efficient delivery practice (delivery is not or is no longer the bottleneck). They have likely optimized their CI/CD pipelines, backlog refinement, planning, technical analysis and design, and task breakdown, code development, and pre-code commit processes and seek to improve the overall flow of work across different stages. Usually, discovery is next, but it could be operation or support. These organizations often navigate complex value streams requiring integration across multiple tools and departments. VSM enables them to visualize and manage these streams effectively, optimizing the entire customer value delivery workflow.
Value Stream Management (VSM) platforms are designed to provide comprehensive insights into the technical aspects of software delivery and their alignment with broader business objectives. Leading VSM platforms offer a range of business metrics that empower organizations to assess the return on investment (ROI) from their software development efforts, ensuring that their initiatives effectively contribute to achieving desired business outcomes.
Here are some key business metrics commonly found in top VSM platforms today:
- Flow Velocity: This metric tracks the number of flow items—such as features, defects, risks, and debts—completed within a defined time frame. It provides valuable insight into the speed at which value is being delivered.
- Flow Efficiency: The ratio of active time to the total time a flow item spends in the value stream serves as a key metric for assessing the efficiency of work processing.
- Flow Time: The total duration for a flow item to progress from ideation to completion is a vital metric for assessing time-to-market.
- Flow Load: Tracks the number of flow items currently in progress. A high volume may signal potential bottlenecks or team overloads.
- Flow Distribution: Tracks the balance among various types of work—features, defects, risks, and technical debt. This metric ensures that all efforts align with strategic priorities.
- Cost of Delay: This approach assesses the financial consequences of postponing the delivery of features or projects, enabling organizations to prioritize tasks based on the potential business value lost due to these delays. By quantifying the cost of delay, companies can make informed decisions about which features to prioritize, ultimately maximizing their return on investment (ROI).
- Value Stream ROI: This metric evaluates the return on investment for various value streams by comparing development costs with the business value generated from delivered features. It enables businesses to assess the financial effectiveness of their development processes, allowing them to adjust resources or priorities to optimize returns.
Like various SEI platform solutions, many top VSM platforms provide valuable insights into team health and metrics associated with developer and team experiences.
Why Choose VSM?
- Holistic View of the Value Stream: VSM tools offer comprehensive visibility, insights, and feedback across the entire digital product workflow. They assist in optimizing each stage, from idea generation to production.
- Alignment with Strategic Goals: These platforms help ensure that all phases of your software lifecycle are aligned with your business objectives.
Choosing a Platform
I think selecting the appropriate metrics platform requires careful consideration of several factors, such as your budget, the locations of bottlenecks in your product workflow, the specific challenges you seek to resolve, and the maturity or efficiencies of your current processes.
Identify Your Bottlenecks
- Deployment Challenges: DORA metrics might be the ideal solution if you are on a tight budget and experience delays mainly after commits, characterized by lengthy lead times or frequent deployment failures. DORA-based tools and surveys can provide the insights necessary to enhance your CI/CD pipeline.
- Development and Delivery Challenges: If inefficiencies arise during the development phase—such as extended cycle times or excessive work-in-progress—SEI platforms can help you identify and effectively address these challenges. Additionally, if you encounter deployment bottlenecks, including issues from code commit to deployment, while also having a favorable budget, leveraging an SEI platform can offer significant advantages over DORA.
- Systemic Issues Across the Lifecycle: For organizations facing extensive inefficiencies from ideation to production, VSM platforms offer a comprehensive solution. These platforms adeptly identify and eliminate bottlenecks throughout the value stream, although they may also represent a significant budget investment.
Why Not Start with a Broad Scope VSM Solution?
While starting with a comprehensive Value Stream Mapping (VSM) solution for complete visibility across your value stream may seem reasonable, there is a strategic advantage in first targeting specific issues: bottlenecks. The reasoning is simple: optimizing downstream of a bottleneck often results in insufficient work while improving upstream processes without addressing the bottleneck can lead to a backlog at that critical choke point. Furthermore, it is crucial to consider several factors, including the organization’s overall understanding of digital product delivery practices — not just within product and technology teams — along with its size and scale, as well as the costs linked to different solutions.
You can start by identifying and optimizing the bottleneck in your process. DORA metrics and SEI platforms can be considered for diagnosing CI/CD pipeline issues and improvements. After resolving these challenges, you can broaden your focus using SEI or VSM platforms to optimize the broader scope of your value stream or product workflow. This comprehensive approach guarantees that your improvements yield significant and sustainable enhancements in overall efficiency.
When to Consider a Broader Scope Platform
Organizations that have invested in and optimized their delivery processes—by establishing highly automated CI/CD pipelines, minimizing waste, and reducing wait times between stages—are ideally positioned to embrace broader-scope platforms, provided they have the budget to support these tools. Here’s why:
- Emphasize the Early and Late Stages: After optimizing delivery, the subsequent focus should be on refining the ideation, operational, and support phases. Comprehensive platforms ensure that only the most valuable ideas progress, operations run efficiently, and support remains proactive and effective.
- Comprehensive Optimization: Platforms with a broader scope, such as VSM tools, offer the visibility required to optimize the entire lifecycle. This ensures that every stage—from ideation to support—remains aligned and efficient.
- Maximizing ROI: These platforms enhance your return on investment by tackling inefficiencies upstream and downstream of delivery, optimizing your existing investments in CI/CD and delivery processes.
- Summary and The Importance of Team Health Metrics
- Selecting the right metrics platform for your software delivery process is a crucial decision that hinges on your organization’s size, specific needs, and level of maturity. Whether your focus is refining your CI/CD pipeline with DORA metrics, enhancing the development process using SEI platforms, or achieving comprehensive visibility through VSM tools, aligning your choice with your strategic objectives is vital.
- Based on my experience in guiding technology teams through digital transformation and my recent deep dive into evaluating the market and adopting a platform, I’ve learned that while the allure of a more expansive platform can be tempting, it’s crucial to ensure that your leadership and organizational mindset are ready for the journey. Without this alignment, even the most sophisticated metrics platform may fail to achieve its full potential.
![](https://rethinkyourunderstanding.com/wp-content/uploads/2024/08/DigitalProductWorkflowStages-FullMetrics.jpg)
Team Sentiment: It is essential to consider team health and quality metrics, such as Developer Experience (DX), which emphasizes team sentiment and well-being. The team performance insights and optimization qualitative measures are critical to improvement. Still, team health metrics complement the quantitative data from delivery metrics, creating a comprehensive view of software efficiency alongside team health. This combination ensures that you optimize processes and nurture a motivated and effective team. While this topic could warrant a follow-up article, or you can read some of my related articles on adopting metrics, it’s essential to integrate qualitative and quantitative feedback into your overall strategy.
By carefully choosing an appropriate metrics platform and complementing it with team health indicators, you can foster continuous improvement, boost efficiency, and provide greater value to your customers and business.
Side Note: In August 2024, I participated in a digital software product delivery workshop. The session delved into the DORA, SPACE, and DevX frameworks Led by industry experts on these topics. The workshop brought together practitioners from delivery teams and C-level executives, enhancing our understanding of these areas related to digital software delivery, optimization, and team health.
References
Related Posts
- Finally, Metrics That Help: Boosting Productivity Through Improved Team Experience, Flow, and Bottlenecks. December 29, 2022.
- Outcome Metrics and the Difficulty of Reporting on Value. February 18, 2023.
- Mitigating Metric Misuse: Preventing the Misuse of Metrics and Prioritizing Outcomes Over Outputs. June 21, 2023
- Developer Experience: The Power of Sentiment Metrics in Building a TeamX Culture. June 18, 2023
- A Balanced Approach to Agile Metrics: Empowering Teams and Mitigating Risks. March 2, 2024
Poking Holes
I invite your perspective on my posts. What are your thoughts?.
Let’s talk: [email protected]