Measuring Output vs. Measuring Value, using Lean Six Sigma metrics to measure agile projects. An Ethical Reflection
From the The Agile Enterprise Blog
by Stelian ROMAN
"The Agile Enterprise" explores Agility at the Enterprise level, examining how Agile principles can be implemented throughout the organization beyond IT. The blog is inspired by the concept of an Agile Enterprise, introduced by the Agile Manufacturing Forum (1991) and the Manifesto for Agile Software Development (2001). Agility is examined from a Project Management perspective with a focus on areas not covered by frameworks that emerged from the work of small software development teams, such as Risk Management, Ethics, Organisational Change Management and Financial Management.
Recent Posts
Measuring Output vs. Measuring Value, using Lean Six Sigma metrics to measure agile projects. An Ethical Reflection
Accountability Avoidance: The Ethical Challenge of Performance Theatre in Agile Teams
Psychological Safety and Team Ethics: The Heart of Agile Success
Challenges of AI and Ethical Product Delivery
Delivering Customer Value vs Public Good: Navigating the Ethical Challenge in Agile Teams
Categories
Agile,
Artificial Intelligence,
Benefits Realization,
Change Management,
Communications Management,
Complexity,
Consulting,
Decision Making,
Disciplined Agile,
Diversity,
Earned Value Management,
Estimating,
Ethics,
General,
Governance,
History,
Innovation,
Knowledge Management,
Leadership,
Lessons Learned,
Metrics,
Organizational Culture,
Product Management,
Risk Management,
Scope Management,
Scrum,
Social Impact,
Stakeholder Management,
Teams,
Testing/Test Management,
Using PMI Standards
Date
Introduction
Agile delivery is based on trust; “Individuals and interactions over processes and tools” and “Working software over comprehensive documentation” are two of the core values, often misinterpreted as a replacement of traditional ‘rigid’ governance. Initially, a challenge to the success of Lean Six Sigma, Agile, intentionally or not, didn’t provide specific metrics, seen as a sign of risk-averse organisations. However, in the quest for higher performance, and without clear and specific Agile metrics, organizations often turn to metrics from Lean Six Sigma—such as cycle time, throughput, defect rates, and process efficiency—to measure the success of Agile projects. These metrics seem objective and actionable, promising to optimize the flow of work. However, a deep ethical debate simmers beneath the surface: Are we measuring what truly matters? Are we rewarding teams for activity, or for genuine customer value? This blog post explores the ethical dilemma of measuring output versus measuring value in Agile environments, from an ethical perspective.
Challenges
The core challenge lies in the tension between what is easy to measure and what is truly meaningful. Lean Six Sigma metrics—cycle time, throughput, defect rates, and process efficiency—offer quantitative data that can be tracked and improved. In Agile projects, these metrics frequently become KPIs. In the true Agile mindset, the intent is to foster transparency and continuous improvement. However, many Agile practitioners argue that such metrics are against the Agile principles and risk shifting team focus from customer outcomes to productivity statistics.
For example, a team may prioritize closing more tickets or delivering more story points each sprint, believing that these numbers reflect success. But what if those tickets represent features that customers don’t use, or story points accrue for technical tasks disconnected from business value? As Ron Jeffries, one of the authors of the Agile Manifesto, has cautioned, “Delivering more stories faster does not ensure delivering more value.”. Moreover, there are teams assigning story points to defects and adding them to ‘velocity’. Sizing defects is useful only for planning and for calculating the percentage of rework, another Lean Six Sigma metric, but not to improve the team’s ‘performance’
Teams can, intentionally or not, game the system—maximizing throughput and activity without a corresponding increase in value delivered. The Agile Practice Guide and PMBOK both highlight the danger of “vanity metrics” that reward teams for busywork rather than outcomes.
This brings us to an ethical crossroads. According to the PMI Code of Ethics and Professional Conduct, project practitioners are obligated to act with responsibility, respect, fairness, and honesty. Is it fair or honest to reward a team for high throughput if their work has little impact on customer satisfaction? ISO 31000, the international standard for risk management, reminds us that risk is not just a matter of likelihood and impact but also of values and perceptions. If measurement systems promote the wrong behaviours, they introduce ethical risk into the organization.
Recommendations
How can teams and organizations resolve this ethical tension? The answer lies in re-balancing what we measure. Activity metrics have their place—they help identify bottlenecks and waste. However, they must be complemented by value-based metrics that reflect customer outcomes and business impact.
Here are five recommendations for ethically sound measurement:
- Prioritize Outcome Metrics: Include metrics such as customer satisfaction, feature adoption, Net Promoter Score (NPS), and business impact. These offer a direct line to customer value.
- Use Activity Metrics as Diagnostic Tools, Not Rewards: Cycle time and throughput help teams identify inefficiencies, but they should not be the sole basis for recognition or incentives.
- Align Metrics with Agile Principles: As the Agile Manifesto states, the highest priority is “to satisfy the customer through early and continuous delivery of valuable software.” Measurement systems should reinforce this, not undermine it.
- Encourage Transparency and Integrity: Make metric goals and their rationale clear to all stakeholders. Use data to foster honest conversations about improvement, not as a weapon.
- Review Metrics Regularly: Collaborate with teams and customers to ensure that what you’re measuring remains relevant and ethical as business needs evolve.
The PMBOK and Agile Practice Guide both advocate for a balanced approach to measurement, highlighting the importance of systems thinking, optimizing for the whole, not just the parts.
The Bottom Line
The ethical debate over measuring output versus value is not a theoretical exercise. It has real consequences for team behaviour, customer satisfaction, and organizational integrity. Rewarding teams solely for higher output can unintentionally incentivize waste, undermine Agile principles, and erode trust. Lean Six Sigma metrics are powerful when applied judiciously, but organizations must ensure their measurement systems are ethically aligned to deliver meaningful value to customers.
Questions for reflection:
- How does your organization balance output metrics with value-based outcomes in Agile projects?
- Have you ever seen (or experienced) negative behaviours arising from an overemphasis on productivity metrics?
- What ethical guidelines should leaders use when designing measurement systems for Agile teams?
Posted on: September 24, 2026 06:44 PM |
Permalink
Comments (13)
Please login or join to subscribe to this item
Luis Branco
CEO| Business Insight, Consultores de Gestão, Ldª
Carcavelos, Lisboa, Portugal
Stelian, I agree with the concern about allowing measures of activity or operational performance to become substitutes for value.
I would, however, question whether the fundamental distinction is really output versus value.
Cycle time, throughput and defect rates do not measure the same thing. They tell us something about flow, volume and quality respectively.
Outputs, outcomes, benefits and value are different again.
Each can provide useful evidence, but none should be allowed to answer a question it was not designed to answer.
I would apply the same caution to what we call “value-based metrics.”
Customer satisfaction, feature adoption or NPS may provide evidence relevant to some dimensions of value, but they are not value itself.
Once any of these becomes a consequential target, we may reproduce exactly the problem your article warns about.
Goodhart and Campbell are particularly relevant here.
Once a proxy becomes consequential, we need to consider not only whether it remains a valid representation of what we care about, but also how the pressure attached to it may change behaviour and distort the system being measured.
That is why I think the ethical issue goes beyond choosing better metrics.
It also concerns the architecture of the measurement system: what a metric represents, what it leaves out, what decisions and incentives are attached to it, what behaviours it encourages, and who bears consequences that the metric may not capture.
There is also an attribution problem.
Moving from operational measures towards outcomes may bring us closer to what ultimately matters, but it often makes causal attribution harder.
If customer satisfaction, adoption or business results improve, what evidence allows us to attribute that improvement to the team being evaluated rather than to other decisions, actors or contextual changes?
So I would be cautious about replacing an output bias with an outcome or “value metric” bias.
Different measures can answer different questions, but we need to preserve the distinction between the measure and the phenomenon it represents, and continually examine what happens when that measure becomes consequential.
For me, the ethical test is therefore not simply, “Are we measuring output or value?”
It is also: “What does this metric actually measure, and whose value does it represent? What behaviour does it encourage? What does it make visible or invisible? And what conclusions are we legitimately entitled to draw from it?”
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
Luis, the metrics 'borrowed' from Lean Six Sigma, like takt time and cycle time, are process metrics because the focus of DMAIC is on process improvement. 'Value' for the customer is mostly expressed in quality and price; hence 6 Sigma.
I agree that NPS, etc., are not really measuring value. However, they are metrics collected from customers; therefore, from a value perspective more relevant than the process metrics.
My ethical question is about "using Lean Six Sigma metrics to measure Agile projects." Is it ethical to use Western values as a benchmark for other geographical areas' values?
Agile, even the MASD version, is about responding to change, and I assume that the change should come from the end user. As an end user, I don't care if the product is developed in an Agile way.
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
Luis, re"What does this metric actually measure, and whose value does it represent? What behaviour does it encourage? What does it make visible or invisible? And what conclusions are we legitimately entitled to draw from it?” I totally agree. However, my dilemma is how ethical it is to use process-measurement metrics as Agile metrics. I searched the web for 'Agile metrics'. The result inspired this post. No other comments.
"Jira Simplifies Work for All Teams, Even on the Most Complex Projects. Visualize Work and Improve Collaboration With Flexible Project Management Software....Agile metrics are quantitative indicators—such as velocity, cycle time, lead time, throughput, and burndown—that measure delivery speed, workflow efficiency, predictability, quality, and team health in Agile teams. These metrics help teams make data‑driven decisions, detect bottlenecks, and improve continuous delivery performance.
"What Agile Metrics Are?"
Agile metrics are measurements that show how work flows, how predictably teams deliver, and whether the delivered work creates value. They support learning and improvement rather than reporting for control. (scrumalliance.org)
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
Luis, another example from agilemania.com:
13 Agile Metrics and KPIs to Track:
1. Velocity
2. Sprint Burndown
3. Release Burndown
4. Lead Time
5. Cycle Time
6. Throughput
7. Work in Progress (WIP)
8. Escaped Defects
9. Defect Removal Efficiency (DRE)
10. Customer Satisfaction Score (CSAT)
CSAT measures how satisfied customers are with your product or service. It is typically gauged through surveys where customers rate their experience on a scale (usually 1-5 or 1-10).
Formula:
CSAT= Total responses/ Positive responses×100
11. Employee Satisfaction
12. Cumulative Flow Diagram (CFD)
13. Team Happiness Metric
“On a scale from 1 to 10, how happy are you at work?”
Out of 13, how many were never used before 2001? How many can be found in LSS books?
My question is not about how good, useful, etc. those metrics are, but how ethical it is to present traditional (waterfall) metrics as Agile? My read of "we are uncovering better ways" is that "we" are creating NEW ways. I believe that using LSS metrics is a good way to demonstrate the superiority of Agile, but they remain LSS metrics.
Luis Branco
CEO| Business Insight, Consultores de Gestão, Ldª
Carcavelos, Lisboa, Portugal
Stelian, I think your examples raise an important question about provenance, but I would separate that from whether a metric can legitimately be used in an Agile context.
Several of the measures now grouped under “Agile metrics” clearly predate the Agile Manifesto, some by decades. But that does not necessarily make them Lean Six Sigma metrics, and even less necessarily “waterfall metrics.” Their genealogies are different. Lead time, cycle time, WIP, throughput, defect measures, and customer or employee measures come from different traditions, while others, such as velocity and burndown, are much more directly associated with the pre-Manifesto XP/Scrum lineage.
There is also an interesting complication with using 2001 as the dividing line. Scrum and XP already existed before the Manifesto, so “used before 2001” does not by itself mean “non-Agile.”
I think we therefore need to distinguish at least four questions: Where did a metric originate? Where was it subsequently adopted? Was it adapted or given a particular role within a later approach? And what are we claiming when we call it an “Agile metric”?
For me, the last question is where the ethical issue becomes particularly interesting.
If “Agile metric” means “a metric invented by Agile,” then historical provenance matters, and attributing an earlier metric to Agile would be misleading. But if it means “a metric used to understand a system operating in an Agile context,” the label does not necessarily imply invention.
CFD is an interesting example. Its underlying logic has roots in flow and queuing concepts. David Anderson played an important role in applying and popularizing cumulative flow diagrams in Agile/Lean software development, and CFDs subsequently became strongly associated with Kanban. Calling CFD a Kanban or Agile metric today may therefore tell us something about its use without necessarily telling us its complete intellectual provenance.
I would apply the same discipline to your point about demonstrating the superiority of Agile. A metric may provide evidence about cycle time, throughput, quality, or another particular property. But improvement in any of those measures cannot, by itself, establish the much broader claim that Agile is “superior.”
That brings me back to the principle behind my original comment: a metric should not be credited to a tradition merely because that tradition uses it, but neither should it be allowed to support a conclusion broader than the phenomenon it actually measures.
So perhaps the ethical question is not whether Agile uses metrics that existed before Agile. It is whether we represent their provenance accurately, use them for questions they can legitimately answer, and distinguish between using a metric in an Agile context and claiming that Agile originated it.
The examples you cite seem to establish the first. I am not yet sure they establish the second.
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
@Luis. Thank you for your response. This post is about the ethics of using existing metrics that were not introduced by Agile as Agile metrics. I agree that any metric can be used to measure any project, Agile included, but IMHO, when cycle time or takt time are used to measure agility without mentioning that they are manufacturing metrics, I see a potential ethical issue, similar to presenting Agile as the only iterative and incremental delivery approach, discovered in 2001.
Your 4 questions may solve the ethical dilemma. Have you seen them in practice?
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
@luis "A metric should not be credited to a tradition merely because that tradition uses it, but neither should it be allowed to support a conclusion broader than the phenomenon it actually measures." I totally agree. Six Sigma originated as a statistical tool. It was adopted in manufacturing to measure the quality of a process. Maybe I should replace LSS metrics with manufacturing metrics. In 1980s, some groups tried, and failed, to use manufacturing processes for software development. But about that some other time,
Luis Branco
CEO| Business Insight, Consultores de Gestão, Ldª
Carcavelos, Lisboa, Portugal
Stelian, I have seen the underlying distinctions used in practice, although not yet these four questions used together in exactly that form. Goal-oriented measurement starts from the question or information need rather than from the available metric. Situational Method Engineering explicitly works with practices or method fragments taken from existing methods, then selected, adapted and recombined for new contexts. Agile itself provides empirical examples of this kind of tailoring. There is also research that separately examines whether the label “agile” is appropriate to a method.
What I have not yet found is the complete sequence applied specifically to metric provenance: origin, subsequent adoption, adaptation or changed role, and then the claim implied by associating the metric with a later method.
That distinction seems important here because provenance, applicability and attribution are different questions. A metric can originate in one domain, be legitimately adapted and used in another, without the later domain thereby claiming to have invented it.
So I think your ethical concern becomes strongest not simply when an older metric is used in Agile, but when the way it is presented materially creates a misleading impression about its origin, novelty or attribution.
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
Luis, "So I think your ethical concern becomes strongest not simply when an older metric is used in Agile, but when the way it is presented materially creates a misleading impression about its origin, novelty or attribution."
Thank you, that's exactly my point.
In the Agile spirit, teams should, I would say must, experiment using any practice or tool old or new. Anything can be Agile or ada adapted to be Agile, but when a concept, practice, tool, or metric that has worked for decades is presented as a new thing, Agile, to demonstrate the need for 'new ways of working,' then IMHO, is unethical.
I've seen value and experience going out the door because of consultants and senior managers forcing "Agile". That's not only sad for the people impacted, but a loss for the organisation and society at large.
Luis Branco
CEO| Business Insight, Consultores de Gestão, Ldª
Carcavelos, Lisboa, Portugal
Thank you, Stelian. I think we have reached common ground on the provenance question.
Your final point raises a different issue that I find particularly important. When experienced people, established practices or accumulated knowledge disappear during a transformation, it is tempting to conclude either that valuable capability has been destroyed or, from the opposite perspective, that what disappeared was simply no longer needed.
I would be cautious about both conclusions.
The more difficult question is what capabilities the organization was actually relying on before the change, whether those capabilities are still needed afterwards and, if so, where and how they are now being provided and regenerated.
People, roles and practices may legitimately change or disappear. But if they were carrying knowledge, judgment, relationships or integrative capacity that the new way of working still requires, their disappearance raises a further question: do those capabilities remain available elsewhere in the system, or has something the organization continues to depend on actually been lost?
Perhaps that is one of the harder tests of any transformation: not simply whether it replaces old ways of working with new ones, but whether it understands which capabilities the organization will still need and how they will continue to exist.
Stelian ROMAN
Project Manager| MicroSafety
Carlingford, New South Wales, Australia
@Luis, "Perhaps that is one of the harder tests of any transformation: not simply whether it replaces old ways of working with new ones, but whether it understands which capabilities the organization will still need and how they will continue to exist.'
I totally agree. I've seen "Agile Coaches" recommending the removal of project managers and replacing them with Scrum Masters.
Sadly, the very people following that advice should've known how important a good manager is. The Leader/Manager debate is, IMHO, just snake oil sold by (some) consultants. A good Manager is a leader, and vice versa.
Please Login/Register to leave a comment.
|
"Put your hand on a hot stove for a minute, and it seems like an hour. Sit with a pretty girl for an hour, and it seems like a minute. THAT's relativity."
- Albert Einstein
|
