A common question that we get from customers who are new to Disciplined Agile (DA) is how do you roll up metrics from solution delivery teams into a portfolio dashboard? A more interesting question is how do you do this when the teams are working in different ways? Remember, DA teams choose their way of working (WoW) because Context Counts and Choice is Good. Even more interesting is the question “How do you roll up team metrics when you still have some traditional teams as well as some agile/lean teams?” In this blog we answer these questions one at a time, in order.
Note: We’re going to talk in terms of a single portfolio in this article, but the strategies we describe can apply at the program (a large team of teams) too, and then the program-level metrics are further rolled up to higher levels.
How Do You Roll Up Agile Team Metrics Into a Portfolio Dashboard?
Pretty much the same way you roll up metrics from traditional teams. There tends to be several potential challenges to doing this, challenges which non-agile teams also face:
How Do You Roll Up Agile Team Metrics Into a Portfolio Dashboard When the Teams Choose Their WoW, and it’s Different For Each Team?
When a team is allowed to choose it’s way of working (WoW), or “own their own process,” the team will often choose to measure itself in a manner that is appropriate to it’s WoW. This makes a lot of sense because to improve your WoW you will want to experiment with techniques, measure their effectiveness for your team within your current context, and then adopt the techniques that work best for you. So teams will need to have metrics in place that provide them with insight into how well they are working, and because each team is unique the set of metrics they collect will vary by team. For example, in Figure 1 below we see that the Data Warehouse (DW) team has decided to collect a different sent of metrics to measure stakeholder satisfaction than the Mobile Development team. The DW team needs to determine which reports are being run by their end users, and more importantly they need to identify new reports that provide valuable information to end users – this is why they have measures for Reports run (to measure usage) and NPS (to measure satisfaction). The Mobile team on the other hand needs to attract and retain users, so they measure things like session length and time in app to determine usage, and user retention and NPS to measure satisfaction.
Figure 1. Applying consistent metrics categories across disparate teams (click on it for a larger version).
Furthermore, the nature of the problem that a team faces will also motivate them to choose metrics that are appropriate for them. In Figure 1 we see that each team has a different set of quality metrics: the DW team measures data quality, the mobile team measures code quality, and the package implementation team measures user acceptance test (UAT) results. Although production incidents and automated test coverage are measured by all three teams, the remaining metrics are unique.
The point is that instead of following the consistent metrics practice across teams by insisting that each team collects the same collection of metrics, it is better to ask for consistent metric categories across teams. So instead of saying “thou shalt collect metrics X, Y, and Z” we instead say “Thou shalt collect metrics that explore Category A, Category B, and Category C.” So, as you can see in Figure 1, each team is asked to collect quality metrics, time to market metrics, and stakeholder satisfaction metrics but it is left up to them what metrics they will choose to collect. The important point is that they need to collect sufficient metrics in each category to provide insight into how well the team addresses it. This enables the teams to be flexible in their approach and collect metrics that are meaningful for them, while providing the governance people within our organization the information that they need to guide the teams effectively.
So how do you roll up the metrics when they’re not consistent across teams? Each team is responsible for taking the metrics that they collect in each category and calculating a score for that category. It is likely that a team will need to work with the governance body to develop this calculation. For example, in Figure 2 we see that the each team has a unique dashboard for their team metrics, yet at the portfolio level the metrics are rolled up into a stoplight status scorecard strategy for each category (Green = Good, Yellow = Questionable, Red = Problem). Calculating a stoplight value is one approach, you could get more sophisticated and calculate a numerical score if you like. This is something the governance body would need to decide upon and then work with teams to implement.
Figure 2. Rolling up metrics categories (click on it for a larger version).
From the looks of the Portfolio dashboard in Figure 2 there is a heat map indicating the overall status of the team (using green, yellow, and red again) and the size of the effort (indicated by the size of the circle). Anyone looking at the portfolio dashboard should be able to click on one of the circles or team stoplights and be taken to the dashboard for that specific team. The status value for the heatmap would be calculated consistently for each team based on the category statuses for that team – this is a calculation that the governance body would need to develop and then implement. The size of the effort would likely come from a financial reporting system or perhaps your people management systems.
How Do You Roll Up Team Metrics When Some Teams Are Still Traditional?
With a consistent categories approach it doesn’t really matter what paradigm the team is following. You simply allow them to collect whatever metrics are appropriate for their situation within each category and require them to develop the calculation to roll the metrics up accordingly. If they can’t come up with a reasonable calculation then the worst case would be for the Team Lead (or Project Manager in the case of a traditional team) to manually indicate/enter the status value for each category.
For the consistent categories strategy to work the governance people need to be able to look at the dashboard for a team, which will have a unique collection of widgets on it, and be able to understand what the dashboard indicates. This will require some knowledge and sophistication from our governance people, which isn’t unreasonable to ask for in our opinion. Effective leaders know that metrics only provide insight but that they shouldn’t manage by the numbers. Instead they should follow the lean concept of “gemba” and go see what is happening in the team, collaborating with them to help the team understand and overcome any challenges they may face.
In Time Tracking and Agile Software Development we overviewed why teams should consider tracking their time. Primary reasons include:
A secondary reason to track time is because the team wants to measure where they are spending time so as to target potential areas to improve. This is more of a side benefit than anything else – if this was your only reason to track time you’d be better off simply discussing these sorts of issues in your retrospectives. But if you were already tracking time then running a quick report to provide the team with intel likely makes sense for you.
So what are your options for recording time? Potential strategies, which are compared in the following table, include:
Table: Comparing options for tracking time.
This blog posting was motivated by a conversation that I had with Stacey Vetzal on Twitter.
On March 21 2017 we ran a webinar entitled Measuring Agile: A Discipline Agile Approach to Metrics. We unfortunately didn’t have enough time to answer all of the great questions that we received so we said that we’d write a blog with the answers. This is it.
We’ve organized the blog into the following sections:
First and foremost, please read our blog entitled 7 “Easy” Steps to Convince Management to Support a Hard Change. This blog has a lot of great advice for getting support from senior management for the changes asked about below.
Q: What is the message to deliver to the board of directors of a company and/or sponsors of projects when it comes to metrics?
The principles section of the webcast made this pretty clear I think. The principles we discussed are:
If I had to pick a key message, it’s that you need to be flexible and enable teams to take a context sensitive approach.
Q: How do you address the resistance to ranged estimates? We see a lot of resistence to this with leaders!
Sadly this is all too common. There is a desire for an “exact number” for the cost/schedule for an IT project, the belief being that such precision leads to lower financial risk. This never seems to work out, and in practice it tends to increase overall financial risk. Worse yet it motivates some very ethically questionable management practices such as padding the budget (lying about the cost to get the fixed estimate as close to the upper end of the range as possible), dropping scope late in the lifecycle (lying about what will be delivered and wasting money working on stuff you had no hope of delivering in the first place), or asking for more money when it’s clear you’re not going to deliver it (this is arguably extortion). In the article Estimating on Agile Projects I go into further detail about the need for ranged estimates.
As we discuss in the 7 Easy Steps blog, you need to educate management on the trade-offs involved with their current approach and help the to see that it isn’t working out well for them in practice.
Q: My management definitely wants to know if they are on time and on budget… how should I handle this problem?
This is also another common problem in more traditional organizations, and is sadly a mindset that is often promoted by project management canon. The first thing to do is ask them why “on time and on budget” is important to them, and to continue to explore their goals via an evolutionary “5 why” strategy until you get to the heart of the matter. Usually the real issue is that they want to reduce schedule risk and financial risk but they only know to ask for on time and on budget respectively. Help them to recognize that they can do better than that. For example, a more effective question to ask instead of “Are we on budget?” would be “Are we spending the money wisely?”, the latter question requiring competent governance to truly answer. Similarly, a more effective question to ask that “Are we on schedule?” would be “Are we going to deliver when the functionality is actually needed?”, also a question requiring better governance than many organizations seem to have today.
Q: Executives and senior management wants common metrics period… can you talk to this – how should a change agent (me) handle this?
Sadly another very common problem. This proves to be selfishly motivated in most cases – they want a common set of metrics to make it easy for them to monitor teams (they have less thinking to do when all the dashboards look the same). As we discussed in the webinar, this isn’t realistic because every team is unique, they face a unique situation, and they have a unique set of priorities. Yes, there are commonalities between teams but also differences. For example, it makes sense for teams to measure quality. BUT, surely it’s obvious that a data warehousing team will measure quality different than a mobile app team, which in turn measures quality differently than a team sustaining a mainframe-based system. This is why I promoted the idea of asking teams to address common categories (such as Quality, Stakeholder Satisfaction, Finance, and so on) but to do so in a manner that makes sense for them.
Q: In a transformation is the ‘stakeholder vision’ milestone something that you see ‘could’ be used as a way to guage adoption of the toolkit across an Enterprise?
Stakeholder vision is a milestone that marks the end of the Inception phase. I’m not sure how it could be used to gauge adoption. It could be used to gauge readiness to move forward with the transformation though.
Q: Is there more thoughts you have on transformation metrics?
Context counts. In general, take a context-sensitive approach where you measure what is important to you. I worked through a lightweight approach to Goal-Quality-Metric (GQM) during the webinar and the OKR approach is also good. There is no one single answer.
Q: How do start-up agile projects best survive in a really non-agile organization?
Sadly, they generally don’t. If you go poking around on the web you’ll find that there’s a lot of advice around this sort of issue, and a large portion of it advising you to jump to another organization.
Q: Please throw some light on Accelaration metric. I’d like to implement that in my org.
We have a detailed blog about acceleration.
Q: Velocity, used to calculate acceleration, can only be calculated if velocy is based on the same point system… I don’t agree acceleration factors out the points, the points have different meaning potentially by project – can you talk to this some more?
You’re wrong, Here’s a quick answer:
Q: Is value is only monetary value is $?
No, it doesn’t have to be but often is. You should measure value in units that are important to your stakeholders. Perhaps value may be measured in market share by them, for example.
Q: Excecutive leaders want to measure “team health” using metrics and compare teams – maybe reward or punish based on these metrics – please talk to this.
This is generally recognized as bad practice because as soon as teams realized that they are being judged by management they’re motivated to game the metrics as best they can. In the case of automated metrics that are difficult to game, then perhaps its possible to compare against that (code quality metrics come to mind). But, management will get what they measure. For example, if they judge teams based on how many defects they fix, chances are pretty good that the team will start identifying relatively trivial defects and then “fix” them to make their metrics looks good. However, if they judge teams based on code quality trends (i.e. Is code quality improving?) they will likely get higher quality code in the long run.
As I said in the webinar, the primary usage of metrics should be to provide insights to teams to help them to improve. Monitoring teams, part of your overall governance strategy, should be an important but secondary concern.
Q: What if the teams don’t agree with the metrics established by management?
My advice is for teams to identify the metrics that make sense for the situation that they face via a lightweight GQM approach (or something similar such as OKR). Management may want to guide teams, perhaps by insisting on certain categories of metrics, see the earlier discussion, or even by suggesting metrics being collected by other teams.
If it’s a situation where management is trying to inflict a common metrics strategy across all teams, which is a relatively bad idea as I discussed earlier, then I think that the team should justify why they don’t agree with the metrics prescribed by management. I also hope that they would suggest a more appropriate strategy and that management would listen to them.
Q: What are some of your favorite tools for metrics?
I typically don’t like recommending tools because the answer to this question is context dependent. Tool choice will be driven by:
So, identify what you situation is then do a bit of research to identify the tools that are right for you.
Q: Is it realistic at scale to have multiple dashboards? +900 teams.
I’ll turn this one around. Is it reasonable to ask teams not to have dashboards simply because your organization is big? Is it reasonable to give up on monitoring teams because your organization is big? Is it reasonable to give up on automation because your organization is big? You see where I’m going with this.
A common question that customers ask us is how do you measure productivity on agile teams. If you can easily measure productivity you can easily identify what is working for you in given situations, or what is not working for you, and adjust accordingly. One way to do so is to look at acceleration, which is the change in velocity.
A common metric captured by agile teams is their velocity. Velocity is an agile measure of how much work a team can do during a given iteration. Velocity is typically measured using an arbitrary point system that is unique to a given team or program. For example, my team might estimate that a given work item is two points worth of effort whereas your team might think that it’s seven points of effort, the important thing is that it’s consistent. So if there is another work item requiring similar effort, my team should estimate that it’s two points and your team seven points. With a consistent point system in place, each team can accurately estimate the amount of work that they can do in the current iteration by assuming that they can achieve the same amount of work as last iteration (an XP concept called “yesterday’s weather”). So, if my team delivered 27 points of functionality last iteration we would reasonably assume that all things being equal we can do the same this iteration.
It generally isn’t possible to use velocity as a measure of productivity. You can’t compare the velocity of the two teams because they’re measuring in different units. For example, we have two teams, A and B, each of 5 people and each working on a web site and each having two-week long iterations. Team A reports a velocity of 17 points for their current iteration and team B a velocity of 51 points. They’re both comprised of 5 people, therefore team B must be three times (51/17) as productive as team A. No! Team A is reporting in their point system and B in their point system, so you can’t compare them directly. The traditional strategy, one that is also suggested in the Scaled Agile Framework (SAFe), would be to ask the teams to use the same unit of points. This might be a viable strategy with a small number of teams although with five or more teams it would likely require more effort than it was worth. Regardless of the number of teams that you have it would minimally require some coordination to normalize the units and perhaps even some training and development and support of velocity calculation guidelines. Sounds like unnecessary bureaucracy that I would prefer to avoid. Worse yet, so-called “consistent” measurements such as FPs are anything but consistent because there’s always some sort of fudge factor involved in the calculation process that will vary by individual estimator.
An easier solution exists. Instead of comparing velocities you instead calculate the acceleration of each team. Acceleration is the change in velocity over time. The exact formula for acceleration is:
(NewVelocity – InitialVelocity)/InitialVelocity
For example, consider the reported velocities of each team shown in the table below. Team A’s velocity is increasing over time whereas team B’s velocity is trending downwards. All things being equal, you can assume that team A’s productivity is increasing whereas B’s is decreasing. At the end of iteration 10, if we wanted to calculate the acceleration since the previous iteration (#9), it would be (23-22)/22= 4.5% for Team A and (40-41)/41 = -2.4% for Team B. So Team A improved their productivity 4.5% during iteration 10 and Team decreased their productivity 2.4% that iteration. A better way to calculate acceleration is to look at the difference in velocity between multiple iterations as this will help to smooth out the numbers over time because as you see in the table the velocity will fluctuate naturally over time (something scientists might refer to as noise). Let’s calculate acceleration over 5 iterations instead of just one, in this case comparing the differences in velocity from iteration 6 to iteration 10. For Team A the acceleration would be (23-20)/20 = 15% and for Team B (40-45)/45 = -11.1% during the 5 iteration period, or 3% and -3.7% respectively on average per iteration.
Normalizing Acceleration for Team Size
The calculations that we performed above assumed that everything on the two teams remained the same. That assumption is likely a bit naïve. It could very well be that people joined or left either of those teams, something that would clearly impact the team’s velocity and therefore it’s acceleration. Let’s work through an example. We’ve expanded the first table to include the size of the team each iteration. We’ve also added a column showing the average velocity per person per iteration for each team, calculated by dividing the velocity by the team size for that iteration. Taking the effect of team size into account, the average acceleration between the last five iterations for Team A is (1.9-1.8)/1.8/5 = 1.1% and for Team B is (5-5)/5/5 = 0.
Similarly, perhaps there was a holiday during one iteration. When there are ten working days per iteration and you lose one or more of them due to holidays it can have a substantial impact on velocity as well. As a result you may want to take into account the number of working days each iteration in your calculation. You would effectively calculate average acceleration per person per day in this case. Frankly I’m not too worried about that issue as it would affect everyone within your organization in pretty much the same way, and it’s easy to understand why there was a “blip” in the data for that iteration.
What Does Acceleration Tell You?
For how you use acceleration in practice, there are three scenarios to consider:
Of course it’s not wise to govern simply by the numbers, so instead of assuming what is going on we would rather go and talk with the people on the two teams. Doing so you might find out that team A has adopted quality-oriented practices such as continuous integration and static code analysis which team B has not, indicating that you might want to help team A share their learnings with other teams.
This is fairly straightforward to do. For example, assume your acceleration is 0.7%, that there are five people on the team, your annual burdened cost per person is $150,000 (your senior management staff should be able to tell you what this number is), and that you have two week iterations. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person).
Another approach is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (note that you want to avoid comparing iterations near any major holidays). So, if the team velocity the first week of February 2015 was 20 points, the same team’s velocity the first week of February 2016 was 23 points, that’s an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
Advantages of the Acceleration Metric
There are several advantages to using acceleration as an indicator of productivity over traditional techniques such as FP counting:
Potential Disadvantages of Acceleration
Of course, nothing is perfect, and there are a few potential disadvantages:
We hope that you’ve found this blog post valuable.
A common question that we get all the time is how do you take a disciplined agile approach to metrics. This is a fairly straightforward question, but it has a potentially complex answer. At a very high level the answer is to keep your metrics strategy as light weight and focused as possible. One way to do this is to adopt an agile version of the Goal Question Metric (GQM) strategy. The fundamental idea behind GQM is that you first identify a goal that you would like to achieve, a set of questions whose answers are pertinent to determining how well you’re achieving that goal, and then the metric(s) that could help you to answer each question.
Consider the goal of improving time to market (reducing overall cycle time in lean parlance). The following table summarizes potential questions, and their supporting metrics, that we could ask to help us to determine how well we’re addressing that goal.
Of course, this would only be one of several goals that you likely have for a given team. I would hope that you have goals around improving quality, stakeholder satisfaction, and staff morale to say the least.
It’s important to note that a single metric can help to answer multiple questions. For example, a ranged release burndown/up chart can potentially help to answer two of the questions in the table above.
Keeping GQM Agile
Unfortunately GQM has gotten a bit of a bad name for itself in the past, often because organizations took a far too heavy approach to applying it. The technique can in fact be applied in an agile manner if you so choose. There are several things that you need to do to keep GQM agile:
Adopting an agile approach to GQM, or something similar, is only one aspect of your overall agile measurement program, and measurement is an important part of your overall governance strategy. More on both of these important topics in future blog postings.