Rethink Your Understanding

Transforming Software Delivery

Navigating the Digital Product Workflow Metrics Landscape: From DORA to Comprehensive Value Stream Management Platform Solutions

by

19 min read

Using engineering metrics can greatly improve your work. Which metrics can help you and your teams improve practices and increase your ability to deliver value to customers more quickly, efficiently, and accurately?

This article is designed to empower senior leaders like you to navigate the landscape of digital product delivery metrics platforms and tools. It offers insights into the various options available and the rationale for selecting one solution over another when determining the most suitable metrics platform.

In the rapidly evolving landscape of software engineering and digital product delivery, leaders are increasingly reevaluating their approaches to team performance, productivity, and metrics. These leaders are turning to quantitative and qualitative data to make delivery teams’ performance and efficiency improvement efforts more transparent. Teams seek data-driven insights to identify bottlenecks, uncover root causes, and evaluate potential enhancements. They aim to eliminate or refine these obstacles by experimenting with various changes to achieve greater efficiency.

Assessing software engineering performance remains one of the most significant challenges for organizations and corporate enterprises alike.

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” – Charles Goodhart

“Goodhart’s Law tells us that ‘when a measure becomes a target, it ceases to be a good measure.” – Marilyn Strathern

Embracing new metrics for today’s work environment is not immune to Goodhart’s law and the potential pitfalls of gamification. Therefore, both leaders and delivery teams must embrace these metrics and distinguish between business discussions and team conversations when analyzing and utilizing them.

However, having data insights and optimizing the performance of your delivery team is not just a necessity but a gateway to capturing performance insights, maintaining competitiveness, and enhancing processes, practices, automation, and team design to deliver value effectively.

As a senior leader with extensive experience guiding technology teams through digital transformation, I have witnessed firsthand the vital role of selecting the right metrics platform. During 2020 and 2021, I faced a similar challenge while assessing and implementing a metrics platform to gain insights into our digital transformation efforts, team structure, and automation investments. Leveraging my background, I quickly gained a deeper understanding of modern metrics.

At that time, a senior leader in our division, much like some of you, struggled to grasp the various metrics options and their relevance to our current delivery capabilities, often fixating on vendor pricing—a legitimate concern given our tight budget. This misunderstanding led to friction as we aimed to identify the best solution. While I acknowledged the advantages of different platforms, the leader’s emphasis on cost over functionality complicated our decision-making process.

After more than eight years of investing in our architecture, team design, automation, and delivery processes, we arrived at a critical decision point: Should we invest in a Software Engineering Intelligence (SEI) platform tailored for delivery, or should we pursue a comprehensive Value Stream Management (VSM) solution that encompasses the entire lifecycle from ideation to production? It’s important to note that the concept of an SEI platform didn’t even exist three years ago when we began our evaluation. At that time, we only had DORA and various vendors offering metrics platforms that needed more coverage across all stages, focused solely on parts of the delivery process, or had notable gaps in delivery insights, even among holistic solutions. I aimed to compare all vendor options, assess their coverage and focus, identify gaps across each workflow phase, and select the best fit.

We initially assessed ten vendors and ultimately narrowed our options to two: our preferred SEI solution and our top choice for the VSM platform. If budget constraints had not been an issue, I would have acquired both solutions to seamlessly bridge the gaps between them. I even lightheartedly suggested a potential merger for both vendors! We chose a VSMP solution based on where we assessed our bottlenecks and stage. This experience also inspired one of my early articles on the subject – Finally, Metrics That Help. Contact me if you’d like to explore my experiences, methodologies, discussions, and presentations related to this evolution in metrics.

Fast forward to 2024, and we discover the inspiration behind this article. I learned about a senior technology leader tasked with transforming and enhancing the large technology team he inherited upon joining an enterprise company. His primary objective is to turn the department around and to improve digital product delivery. He aims to reintegrate QA and product delivery responsibilities into teams—responsibilities removed during previous cost-cutting measures implemented by earlier stakeholders before he arrived in the organization. Based on my understanding, part of his strategy includes adopting delivery performance metrics to establish team metrics that will create a baseline for current performance, offer valuable insights, and initiate a pathway to enhancing overall performance and efficiency. He has focused explicitly on or selected DORA metrics to better understand product delivery roles and processes. Although I didn’t have the opportunity to discuss this with the leader before his decision, I am intrigued by how he arrived at the choice of DORA metrics. This presents a unique opportunity for us to learn and grow in our understanding of metrics. Given that this enterprise technology team has been delivering work effectively in an agile environment for years, I wonder if his decision truly addresses the more significant challenges faced by his division in improving the delivery process. Are these the appropriate metrics and feedback mechanisms to resolve workflow bottlenecks?

Stages of the Product Delivery Workflow

Once you understand the workflow, you can analyze and pinpoint its bottlenecks. This understanding allows you to implement metrics that offer valuable feedback and insights on these bottlenecks. This feedback is crucial as it helps you identify areas for improvement in an iterative manner.

The stages of the digital product workflow, often discussed in frameworks such as Value Stream Management (VSM) and Flow Engineering, typically follow the journey from an initial idea to the moment the product delivers value to the customer. Understanding these phases is essential for grasping how work flows within an organization and identifying where value is created.

The key stages or phases of the digital product workflow are:

  • Ideation (or Discovery)
  • Delivery
  • Operations
  • Support

Ideation (or Discovery): This phase focuses on generating, prioritizing, and refining ideas. It involves understanding what should be developed based on customer needs, market demands, and strategic objectives. This stage lays the groundwork for all future efforts and encompasses market research, gathering user feedback, and creating early design prototypes.

Delivery: Once an idea is validated, it transitions into the delivery phase, where actual development occurs. Delivery includes coding, testing, and deploying the product. This phase emphasizes transforming ideas into tangible products ready for customer delivery. It also encompasses the CI/CD pipeline, where DORA metrics are often utilized.

Operations: Once the product is delivered, it transitions into the operations phase. This stage is crucial in maintaining and managing the product within a live environment. Your work in performance monitoring, infrastructure management, and ensuring the product operates smoothly and efficiently is key to the product’s quality.

Support: The final stage is support, encompassing customer service, incident management, and ongoing enhancements based on user feedback. Your continuous efforts in this phase ensure that customers derive continued value from the product while effectively addressing any issues arising after deployment.

Value Stream Management resources, Product Flow, and Flow Engineering often highlight these stages. They offer a structured framework for understanding the workflow, guiding it from the initial idea to continuously delivering value to customers. These stages enable organizations to grasp how value is generated and delivered, facilitating more effective management of the entire product lifecycle.

Understanding the Types of Metrics Platforms

Before exploring specific tools, it’s essential to grasp the three main categories of qualitative metrics platforms focused on delivery performance insights as I understand them: DORA metrics, DX Core 4, Software Engineering Intelligence (SEI) platforms, and Value Stream Management (VSM), along with flow metrics.

DORA Metrics

“The bottleneck is Deployment.” Focus is on a very narrow slice of the overall work flow.

DORA (DevOps Research and Assessment) metrics are designed to measure the effectiveness and efficiency of your software delivery process from code commit to deployment. These metrics include:

  • Change lead time
  • Deployment frequency
  • Change fail percentage (previously change failure rate)
  • Failed deployment recovery time (previously mean time to recovery or MTTR)

The information regarding throughput, stability, and metric definitions is sourced from the dora.dev website.1

Throughput

Throughput measures the velocity of changes that are being made. DORA assesses throughput using the following metrics:

  • Change lead time – This metric measures the time it takes for a code commit or change to be successfully deployed to production. It reflects the efficiency of your delivery pipeline.
  • Deployment frequency – This metric measures how often application changes are deployed to production. Higher deployment frequency indicates a more agile and responsive delivery process.

Stability

Stability measures the quality of the changes delivered and the team’s ability to repair failures. DORA assesses throughput using the following metrics:

  • Change fail percentage – This metric measures the percentage of deployments that cause failures in production, requiring hotfixes or rollbacks. A lower change failure rate indicates a more reliable delivery process.
  • Failed deployment recovery time – This metric measures the time it takes to recover from a failed deployment. A lower recovery time indicates a more resilient and responsive system.

Why Choose DORA?

Budget concerns. The bottleneck is deployment (release) and stability.

  • Focused on Deployment Efficiency: DORA metrics are ideal for organizations that need to optimize their CI/CD pipeline and improve deployment frequency and stability.
  • Narrow Scope, High Impact: DORA metrics can provide precise insights to drive improvements if your primary bottlenecks relate to deployment.

DX Core 4 Metrics

“The bottleneck is primarily deployment” and insights into team engagement.

The DX Core 4 is a new entrant in the metrics landscape for 2023 and 2024. This comprehensive framework aims to measure and enhance productivity among developers and delivery teams by focusing on four essential dimensions: speed, effectiveness, quality, and business impact. Designed to integrate insights from established frameworks such as DORA, SPACE, and DevEx, it provides organizations with actionable insights that bridge the gap from frontline teams to executive leadership.2 Similar to DORA, this framework emphasizes the Delivery phase of Flow.

Here are the four dimensions of the DX Core 4 framework:

Speed: Measures how quickly software is developed and delivered, from inception to production.

Key Metrics:

  • Cycle Time: How long a task or feature takes to move through the development pipeline.
  • Lead Time for Changes: Similar to DORA, this measures the time from code commit to production.

Effectiveness: Assess how well development processes achieve their intended outcomes.

Key Metrics:

  • Workflow Efficiency: How efficiently teams move work through different stages.
  • Goal Completion: The ability of the team to meet sprint or project objectives.

Quality: Evaluate the quality of the software being produced.

Key Metrics:

  • Defect Rate: The number of defects or issues found during or after deployment.
  • Code Quality: Measured by the percentage of code needing rework or post-deployment fixes.
  • User Satisfaction: User feedback, often through surveys or Net Promoter Scores (NPS).

Business Impact: Measures how well software development efforts align with and support overall business objectives.

Key Metrics:

  • Feature Adoption: Tracks how customers adopt new features.
  • Revenue Impact: How specific development activities contribute to business revenue or growth.
  • Alignment with Business Goals: Measures the percentage of development work directly contributing to strategic business objectives.

These four dimensions provide a balanced view of development performance, ensuring that teams deliver quickly and produce high-quality work that drives business outcomes. The DX Core 4 framework integrates well with other methodologies like DORA, SPACE, and DevEx, offering a holistic picture (quantitative and qualitative) of productivity.

Why Choose DX Core 4?

The bottleneck or focus is on Delivery performance and Team Sentiment.

Consider DX Core 4 if you aim to accelerate delivery while ensuring that your software meets high-quality standards and aligns with business objectives. This framework enables you to focus on both the technical and business dimensions of software development, providing a balanced approach to enhancing developer productivity in the organization.

Software Engineering Intelligence (SEI) Platforms

“Our bottleneck is Delivery.” Focus is on the Delivery stage.

In recent years, Software Engineering Intelligence (SEI) platforms have emerged as a distinct category, extending beyond traditional DORA metrics primarily focusing on the CI/CD pipeline. Additionally, some analysts, such as McKinsey and solution providers, refer to these as Engineering Management Platforms (EMPs). For the purpose of this post, I will refer to them as SEI platforms.

SEI platforms cover the entire delivery process, from when a work item enters the backlog to its deployment, incorporating metrics like cycle time, throughput, work-in-progress (WIP), and quality indicators such as pull request size and rework rates. These platforms excel by integrating various tools (e.g., Git, CI/CD, project management) to offer a unified view of the entire delivery pipeline.

Some SEI platforms focus on actionable insights and benchmarking, often utilizing data to compare team performance against industry standards. More importantly, these SEI solutions align engineering efforts with broader business objectives, a crucial aspect that traditional DORA metrics often overlook but is of strategic value to any organization.

Some SEI platforms also provide a comprehensive view of the development process, including visibility in developer experience (DX) by tracking productivity and team health, using metrics like WIP and burnout alerts. This holistic view makes them ideal for organizations looking to optimize workflows beyond just CI/CD, instilling confidence in the thoroughness of the approach.

While DORA metrics provide critical insights into deployment efficiency, SEI platforms offer a more comprehensive approach, ideal for organizations seeking to optimize the full delivery pipeline and align with business goals.

Gartner has recently published a market guide on SEI platforms for those interested in exploring this further3.

Here are some key types of metrics commonly found in some of the more known SEI platforms :

  • Cycle Time: Measures the time it takes for a work item to move from the backlog to deployment. This metric helps identify inefficiencies in the development process and highlights areas where delays occur.
  • Throughput: This metric tracks the number of work items completed over a specific period. It provides insight into team productivity and helps assess whether teams meet their delivery goals.
  • Work in Progress (WIP): Monitors the number of items being worked on or in progress. High WIP can indicate bottlenecks or overloading of the team, which can slow overall delivery.
  • Pull Request (PR) Size and Rework Rate: Evaluate the size of pull requests and the frequency of rework required. Smaller PRs with lower rework rates tend to be merged faster and with fewer issues, contributing to smoother deployments.
  • Planning Accuracy: This compares planned versus actual delivery timelines. It helps teams understand how well they estimate their work and whether they consistently meet their commitments.
  • Developer Experience (DX): SEI platforms often include metrics that assess developer satisfaction and efficiency, such as time spent in meetings versus coding and the overall impact of tools and processes on developer productivity.

Why Choose SEI?

Your bottleneck remains in the workflow’s delivery phase, but you have the budget for an SEI solution and need insights beyond code commit to delivery.

  • Optimizing Planning, Development, Deployment, and Stability: SEI platforms are perfect for teams looking to streamline the development and delivery process from backlog to production, balancing speed with quality and introducing team health insights.
  • Broader Than DORA, Narrower Than VSM: SEI platforms offer a middle ground, providing insights into the development process without tracking the entire lifecycle.

Value Stream Management (VSM) Platforms and Flow Metrics

“The bottleneck is upstream.” Insights cover the entire Value Stream (work flow).

Value Stream Management Platforms (VSMP) optimizes the entire digital product lifecycle, from ideation to operation and support. It enables organizations to visualize, measure, and enhance the flow of value across complex systems, identifying inefficiencies and aligning technical and business objectives. Companies ready to adopt VSM typically have mature delivery practices and aim to improve collaboration across teams. VSM platforms provide insights into bottlenecks, work-in-progress limits, and metrics that help organizations assess ROI and align their efforts with business goals, ultimately enhancing predictability and efficiency in software delivery.

Enterprises and companies that have embraced agile and DevOps practices and seek to bridge the divide between business and technology will find these platforms especially beneficial for enhancing predictability, speeding up discovery and delivery, and achieving alignment with business goals across the organization. Like various SEI platform solutions, many top VSM platforms provide valuable insights into team health and metrics associated with developer and team experiences.

Here are some key business metrics commonly found in top VSM platforms today:

  • Flow Velocity: This metric tracks the number of flow items—such as features, defects, risks, and debts—completed within a defined time frame. It provides valuable insight into the amount of value being delivered.
  • Flow Efficiency: The ratio of active time to the total time a flow item spends in the value stream serves as a key metric for assessing the efficiency of work processing.
  • Flow Time: The total duration for a flow item to progress from ideation to completion is a vital metric for assessing time-to-market.
  • Flow Load: Tracks the number of flow items currently in progress. A high volume may signal potential bottlenecks or team overloads.
  • Flow Distribution: Tracks the balance among various types of work—features, defects, risks, and technical debt. This metric ensures that all efforts align with strategic priorities.
  • Cost of Delay: This approach assesses the financial consequences of postponing the delivery of features or projects, enabling organizations to prioritize tasks based on the potential business value lost due to these delays. By quantifying the cost of delay, companies can make informed decisions about which features to prioritize, ultimately maximizing their return on investment (ROI).
  • Value Stream ROI: This metric evaluates the return on investment for various value streams by comparing development costs with the business value generated from delivered features. It enables businesses to assess the financial effectiveness of their development processes, allowing them to adjust resources or priorities to optimize returns.

Why Choose VSM?

  • Holistic View of the Value Stream: VSM tools offer comprehensive visibility, insights, and feedback across the entire digital product workflow. They assist in optimizing each stage, from idea generation to production.
  • Alignment with Strategic Goals: These platforms help ensure that all phases of your software lifecycle are efficient and aligned with your business objectives.

Choosing a Platform

Selecting the appropriate metrics platform requires careful consideration of several factors, such as your budget, the locations of bottlenecks in your product workflow, the specific challenges you seek to resolve, the maturity or efficiencies of your current processes, and the readiness and awareness of your leadership team across the stages.

Identify Your Bottlenecks

  • Deployment Challenges: DORA metrics might be the ideal solution if you are on a tight budget and experience delays mainly after commits, characterized by lengthy lead times or frequent deployment failures. DORA-based tools and surveys can provide the insights necessary to enhance your CI/CD pipeline.
  • Development and Delivery Challenges: If inefficiencies arise during the development phase—such as extended cycle times or excessive work-in-progress—SEI platforms can help you identify and effectively address these challenges. Additionally, if you encounter deployment bottlenecks, including issues from code commit to deployment, while also having a favorable budget, leveraging an SEI platform can offer significant advantages over DORA.
  • Systemic Issues Across the Lifecycle: For organizations facing extensive inefficiencies from ideation to production, VSM platforms offer a comprehensive solution. These platforms adeptly identify and eliminate bottlenecks throughout the value stream, although they may also represent a significant budget investment.

Why Not Start with a Broad Scope VSM Solution?

While starting with a comprehensive Value Stream Mapping (VSM) solution for complete visibility across your value stream may appear logical, strategically targeting specific issues — namely, bottlenecks — can offer significant advantages. The rationale is straightforward: optimizing processes downstream of a bottleneck often leads to underutilization, while enhancing upstream operations without addressing the bottleneck can result in a backlog at that critical choke point. 

You can start by identifying and optimizing the bottleneck in your process. DORA metrics and SEI platforms can be considered for diagnosing CI/CD pipeline issues and delivery phase improvements. After resolving these challenges, you can broaden your focus using SEI or VSM platforms to optimize the broader scope of your value stream or product workflow. This comprehensive approach guarantees that your improvements yield significant and sustainable enhancements in overall efficiency.

Additionally, it is essential to consider various factors, such as the organization’s size, budget constraints associated with different solutions, the readiness of divisional leadership, and the overall comprehension of digital product delivery practices — not just within product and technology teams but across the entire organization.

When to Consider a Broader Scope Platform

Organizations that have invested in and optimized their delivery processes—by establishing highly automated CI/CD pipelines, minimizing waste, and reducing wait times between stages—are ideally positioned to embrace broader-scope platforms, provided they have the budget to support these tools. Here’s why:

  • Emphasize the Early and Late Stages: After optimizing delivery, the subsequent focus should be on refining the ideation, operational, and support phases. Comprehensive platforms ensure that only the most valuable ideas progress, operations run efficiently, and support remains proactive and effective.
  • Comprehensive Optimization: Platforms with a broader scope, such as VSM tools, offer the visibility required to optimize the entire lifecycle. This ensures that every stage—from ideation to support—remains aligned and efficient.
  • Maximizing ROI: These platforms enhance your return on investment by tackling inefficiencies upstream and downstream of delivery, optimizing your existing investments in CI/CD and delivery processes.

Team Sentiment

Team Sentiment: Incorporating team health and sentiment qualitative metrics into your metrics solutions or selections is crucial. Metrics like Developer Experience (DX) and Team Experience (TX) should prioritize team sentiment and well-being, highlighting their importance in fostering a positive work environment. Team health metrics complement the quantitative data from delivery metrics, creating a comprehensive view of software efficiency alongside team health. This combination ensures you optimize processes and nurture a motivated and effective team. While this topic could warrant a follow-up article, or you can read some of my related articles on adopting metrics, it’s essential to integrate qualitative and quantitative feedback into your overall strategy.


Summary

Whether your goal is to refine your CI/CD pipeline using DORA metrics, improve your development process with SEI platforms, or achieve comprehensive visibility with VSM tools, aligning your choice with your strategic objectives is essential. This post highlights the numerous options and solutions available when considering implementing a metrics platform.

By thoughtfully selecting an appropriate platform and pairing it with team health indicators, you can drive continuous improvement, enhance efficiency, and deliver exceptional value to your customers and business. Selecting the appropriate metrics platform for your software delivery process is a vital decision influenced by your organization’s size, specific requirements, and maturity level. Ultimately, it can come down to your organization’s context, budget, and leadership mindset.  

A combination of solutions or tools serves you best, as they can complement one another by filling the gaps left by individual platforms. This holds when finding a solution that offers quantitative performance and qualitative team health and engagement.

From my experience in guiding technology teams through digital transformation and my recent in-depth market evaluation and platform adoption, I’ve learned that readiness is paramount. While the appeal of a more extensive platform can be strong, it’s vital to ensure that your leadership and organizational mindset are prepared for the journey ahead. Without this alignment, even the most advanced metrics platform may fall short of its potential. Therefore, before making the leap, confirm that your team is on board and your organization is ready for the change. Know your constraints, list your anticipated results, and what you want to achieve. Then, pick a solution or combination to help you achieve your expected outcomes.

Options in the Market

I’ve conducted thorough research to choose the best solution for our specific context and have my preferred partners. Additionally, we spent a year exploring the build-versus-buy option, beginning our journey with an in-house solution. If you want to learn more about my experiences, please reach out.

Disclaimer: In this article, I will not endorse any specific vendor. I will provide a foundational understanding based on insights from various vendors and their websites. I do not claim that these mentions represent the only options available in the market or that they accurately capture the purpose of each vendor within their respective categories. Additionally, I need more insight to assess which platform might be superior, as I need to familiarize myself with your organization’s specific context. My aim is simply to give you a head start in your research.

DORA

Several platforms on the market leverage DORA (DevOps Research and Assessment) metrics to evaluate software delivery performance. These platforms emphasize key DORA metrics, including deployment frequency, lead time for changes, mean time to restore (MTTR), and change failure rate. Below are some noteworthy platforms and vendor solutions:

DX Core 4

As the market’s newest framework, few platforms integrate the DX Core 4 metrics into their offerings to help organizations track developer productivity and experience across speed, effectiveness, quality, and business impact. The DX Platform is a primary example, designed by leading researchers behind DORA and SPACE, specifically created to measure developer experience and productivity through qualitative and quantitative insights.

Here are some notable platforms that utilize the DX Core 4 metrics:

  • DX Platform: The DX Platform was specifically designed to measure and improve developer productivity through the DX Core 4 framework.

Alternatives to consider that may not directly incorporate DX Core 4 metrics yet provide comparable insights include:

  • LinearB: Although LinearB does not explicitly use the DX Core 4 framework, it tracks several metrics aligned with speed, effectiveness, and quality, similar to the DX Core 4 dimensions.
  • Pluralsight Flow (formerly GitPrime): aligns with some DX Core 4 dimensions, especially in speed and quality.
  • GitLab: aligns with some of the core tenets of the DX Core 4 framework.

Software Engineering Intelligence (SEI) Platforms

Several platforms explicitly position themselves as Software Engineering Intelligence (SEI) platforms, offering metrics to help organizations optimize software delivery, developer productivity, and business alignment. These platforms go beyond traditional CI/CD metrics and provide deeper insights into development workflows, team efficiency, and alignment with business goals. Here are some leading SEI platforms available today:

Value Stream Management Platforms

Numerous platforms actively position themselves as Value Stream Management (VSM) solutions, offering metrics that track the flow of value throughout an organization’s discovery and development lifecycles. These platforms enable organizations to align software delivery with business objectives, streamline workflows, and reduce inefficiencies within the value stream. VSM platforms are particularly beneficial for enterprises requiring comprehensive governance and oversight in complex release management scenarios. Below, we highlight some of the leading VSM platforms currently available on the market:

Update: September 18, 2024 – Following the publication of this article, Planview has announced its acquisition of Plutora. Press Release.

This article aimed to help senior leaders or other product and technology managers navigate the landscape of performance metrics platforms and tools to measure the performance and engagement of software delivery teams. I hope this post offers valuable insights if you are contemplating the adoption of a metrics platform.

References

  1. DORA, https://dora.dev/guides/dora-metrics-four-keys/
  2. DX Core 4, https://getdx.com/research/measuring-developer-productivity-with-the-dx-core-4/
  3. Software Engineering Intelligence Platforms Reviews and Ratings, https://www.gartner.com/reviews/market/software-engineering-intelligence-platforms, Garnter.com

Related Posts

Poking Holes

I invite your perspective on my posts. What are your thoughts?.

Let’s talk: [email protected]