Statistical Misuse of Ordinal Scales: The Mathematical and Ethical Flaws of Averaging Planning Poker Metrics
| Statistical Misuse of Ordinal Scales: The Mathematical and Ethical Flaws of Averaging Planning Poker Metrics Introduction In Agile software development, metrics like Planning Poker story points are widely used to estimate the size and complexity of work items. These metrics are based on ordinal scales—a type of ranking where the relative order of items matters, but the exact differences between them do not. Despite this, it’s common practice to calculate averages, run regressions, and otherwise apply standard mathematical operations to such data. This statistical misuse isn’t just a technical mistake; it has real-world consequences for decision-making and can cross into the realm of ethical misrepresentation. In this blog post, we examine the nature of ordinal data, why treating it as interval data is problematic, and the ethical implications for teams and organizations. We also provide guidance to help avoid these pitfalls, concluding with a question for readers to reflect on their own experiences. Understanding Ordinal Scales in Agile Contexts What Is an Ordinal Scale? An ordinal scale is a way of ranking items or outcomes according to some criterion, but without specifying the degree of difference between them. For example, a restaurant rating system (poor, fair, good, excellent) or a pain scale (mild, moderate, severe) are ordinal. In Agile, Planning Poker uses a sequence of numbers (often Fibonacci: 1, 2, 3, 5, 8, 13, etc.) to estimate effort, but the gaps between these numbers are not consistent or meaningful in a mathematical sense. Why Do Teams Use Ordinal Scales? Ordinal scales like Planning Poker sequences are practical for group estimation, helping to drive consensus and discussion. They acknowledge the uncertainty and subjectivity inherent in software estimation, allowing teams to quickly rank work items from smallest to largest without worrying about precise measurement. Statistical Misuse: Averages and Regressions on Ordinal Data The Mathematics of Ordinal Data Ordinal data only tells us the order of items, not the magnitude of differences. For example, the difference in effort between a 2-point and a 3-point story is not necessarily the same as between a 5-point and an 8-point story. Treating these numbers as if they are evenly spaced (like real numbers on a ruler) violates the fundamental properties of ordinal data. The Flaws of Mathematical Averages Despite this, many teams and organizations calculate the average story point value for a sprint, or the average velocity across sprints. They may even run regressions to forecast future delivery. However, calculating averages or running arithmetic operations on ordinal data is mathematically unsound because:
Some organizations take it further, applying regression analysis or more complex statistical models to ordinal data. These methods assume interval or ratio-level data, where arithmetic operations are valid. Using them on ordinal metrics produces results that are, at best, spurious and, at worst, drive misguided decisions. Real-World Consequences of Statistical Misuse Poor Decision-Making Relying on mathematically flawed averages or projections leads to poor planning, unrealistic commitments, and ultimately, failed projects. Teams may be pushed to deliver "average" story sizes that are not grounded in reality or pressured to meet forecasted velocities that have no statistical validity. Erosion of Trust When stakeholders realize that the numbers don’t add up—or worse, when projects fail due to flawed metrics—trust in the estimation process and in leadership breaks down. Ethical Implications Misrepresenting ordinal metrics as if they were interval or ratio data is more than just a technical error; it’s an ethical lapse. It can:
Best Practices: Using Ordinal Metrics Responsibly
Ordinal metrics like Planning Poker story points have value when used as intended—to facilitate team discussion and consensus. But applying standard mathematical operations to these numbers is both mathematically invalid and ethically questionable. By respecting the true nature of ordinal data and reporting it with integrity, teams and organizations can avoid misleading themselves and their stakeholders, making better decisions and building greater trust. Question for Readers: Have you encountered situations where averages or advanced analytics were applied to ordinal metrics like story points or Planning Poker estimates? How did it affect planning, transparency, or trust in your teams? Share your experiences and insights below. |
Metric Integrity, Semiquantitative Traps & Ethics: The Fallacy of Velocity as a Performance Metric
| Introduction In the fast-paced world of Agile software development, metrics like story points and velocity are commonly used to estimate, plan, and track progress. However, when organizations and leaders start treating these metrics as absolute measures of productivity, they fall into a dangerous trap—one that not only undermines the integrity of the data but can also violate fundamental ethical principles, particularly the pillar of Honesty in data reporting. This blog post delves into the nuances of metric integrity, the pitfalls of semiquantitative metrics, and the ethical responsibilities that come with reporting and interpreting team performance.
When organizations start treating velocity as an absolute, quantitative performance indicator, several problems arise:
|
Goodhart's Law in Agile Delivery: When Metrics Become Targets
| Introduction In the pursuit of productivity and predictability, organizations often turn to metrics to track progress and drive improvement. In Agile software delivery, measures like story points and velocity have become ubiquitous tools for estimation and forecasting. Yet, as the British economist Charles Goodhart famously observed, “When a measure becomes a target, it ceases to be a good measure.” This principle—known as Goodhart’s Law—captures a dangerous dynamic: when management fixates on metrics as ends in themselves, teams adapt their behaviour to meet the numbers, often at the expense of genuine progress and transparency. This blog post explores how Goodhart’s Law manifests in Agile delivery, why it leads teams to inflate point sizing, and what organizations can do to foster healthier measurement cultures.
Rather than working faster or delivering more value, teams may unconsciously or deliberately inflate story point estimates to make their velocity appear higher. The logic is simple: if a 3-point story is now estimated as a 5, the same work results in a higher velocity. Over time, the relative calibration that made story points useful is lost. The Consequences
|
Fabricating Estimates Under Executive Pressure: Navigating the Ethics of Adjusting to Fit the Budget
|
Why Is This Unethical?
|
Velocity Misuse and Performance Pressure: Rethinking Agile Metrics
Agile introduced velocity as a simple tool: a way for teams to estimate how much work they can deliver in a sprint, supporting better planning and realistic forecasting. Yet, over time, velocity has been repurposed—and sometimes misused—as a performance metric, leading to unintended consequences for teams and organizations.The Problem: Planning Tool or Performance Benchmark?Velocity was never meant to be a Key Performance Indicator (KPI) or a tool for comparing teams. However, it’s common to see organizations:
The Ethical DilemmaWhen velocity becomes the yardstick for performance, teams face a fundamental question:
A New Direction: Value and Outcomes Over OutputThe hottest trend in Agile metrics is a move away from output-based measurements like velocity toward value-driven and outcomes-based approaches. This shift means:
The Bottom Line: Velocity is a useful planning tool—but it’s not a measure of team worth. The future of Agile metrics lies in focusing on value, outcomes, and ethical practices that support both team wellbeing and organizational goals. How is your team measuring success? Are your metrics driving value—or just numbers? |



