A common question that customers ask us is how do you measure productivity on agile teams. If you can easily measure productivity you can easily identify what is working for you in given situations, or what is not working for you, and adjust accordingly. One way to do so is to look at acceleration, which is the change in velocity.
A common metric captured by agile teams is their velocity. Velocity is an agile measure of how much work a team can do during a given iteration. Velocity is typically measured using an arbitrary point system that is unique to a given team or program. For example, my team might estimate that a given work item is two points worth of effort whereas your team might think that it’s seven points of effort, the important thing is that it’s consistent. So if there is another work item requiring similar effort, my team should estimate that it’s two points and your team seven points. With a consistent point system in place, each team can accurately estimate the amount of work that they can do in the current iteration by assuming that they can achieve the same amount of work as last iteration (an XP concept called “yesterday’s weather”). So, if my team delivered 27 points of functionality last iteration we would reasonably assume that all things being equal we can do the same this iteration.
It generally isn’t possible to use velocity as a measure of productivity. You can’t compare the velocity of the two teams because they’re measuring in different units. For example, we have two teams, A and B, each of 5 people and each working on a web site and each having two-week long iterations. Team A reports a velocity of 17 points for their current iteration and team B a velocity of 51 points. They’re both comprised of 5 people, therefore team B must be three times (51/17) as productive as team A. No! Team A is reporting in their point system and B in their point system, so you can’t compare them directly. The traditional strategy, one that is also suggested in the Scaled Agile Framework (SAFe), would be to ask the teams to use the same unit of points. This might be a viable strategy with a small number of teams although with five or more teams it would likely require more effort than it was worth. Regardless of the number of teams that you have it would minimally require some coordination to normalize the units and perhaps even some training and development and support of velocity calculation guidelines. Sounds like unnecessary bureaucracy that I would prefer to avoid. Worse yet, so-called “consistent” measurements such as FPs are anything but consistent because there’s always some sort of fudge factor involved in the calculation process that will vary by individual estimator.
An easier solution exists. Instead of comparing velocities you instead calculate the acceleration of each team. Acceleration is the change in velocity over time. The exact formula for acceleration is:
(NewVelocity – InitialVelocity)/InitialVelocity
For example, consider the reported velocities of each team shown in the table below. Team A’s velocity is increasing over time whereas team B’s velocity is trending downwards. All things being equal, you can assume that team A’s productivity is increasing whereas B’s is decreasing. At the end of iteration 10, if we wanted to calculate the acceleration since the previous iteration (#9), it would be (23-22)/22= 4.5% for Team A and (40-41)/41 = -2.4% for Team B. So Team A improved their productivity 4.5% during iteration 10 and Team decreased their productivity 2.4% that iteration. A better way to calculate acceleration is to look at the difference in velocity between multiple iterations as this will help to smooth out the numbers over time because as you see in the table the velocity will fluctuate naturally over time (something scientists might refer to as noise). Let’s calculate acceleration over 5 iterations instead of just one, in this case comparing the differences in velocity from iteration 6 to iteration 10. For Team A the acceleration would be (23-20)/20 = 15% and for Team B (40-45)/45 = -11.1% during the 5 iteration period, or 3% and -3.7% respectively on average per iteration.
Normalizing Acceleration for Team Size
The calculations that we performed above assumed that everything on the two teams remained the same. That assumption is likely a bit naïve. It could very well be that people joined or left either of those teams, something that would clearly impact the team’s velocity and therefore it’s acceleration. Let’s work through an example. We’ve expanded the first table to include the size of the team each iteration. We’ve also added a column showing the average velocity per person per iteration for each team, calculated by dividing the velocity by the team size for that iteration. Taking the effect of team size into account, the average acceleration between the last five iterations for Team A is (1.9-1.8)/1.8/5 = 1.1% and for Team B is (5-5)/5/5 = 0.
Similarly, perhaps there was a holiday during one iteration. When there are ten working days per iteration and you lose one or more of them due to holidays it can have a substantial impact on velocity as well. As a result you may want to take into account the number of working days each iteration in your calculation. You would effectively calculate average acceleration per person per day in this case. Frankly I’m not too worried about that issue as it would affect everyone within your organization in pretty much the same way, and it’s easy to understand why there was a “blip” in the data for that iteration.
What Does Acceleration Tell You?
For how you use acceleration in practice, there are three scenarios to consider:
Of course it’s not wise to govern simply by the numbers, so instead of assuming what is going on we would rather go and talk with the people on the two teams. Doing so you might find out that team A has adopted quality-oriented practices such as continuous integration and static code analysis which team B has not, indicating that you might want to help team A share their learnings with other teams.
This is fairly straightforward to do. For example, assume your acceleration is 0.7%, that there are five people on the team, your annual burdened cost per person is $150,000 (your senior management staff should be able to tell you what this number is), and that you have two week iterations. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person).
Another approach is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (note that you want to avoid comparing iterations near any major holidays). So, if the team velocity the first week of February 2015 was 20 points, the same team’s velocity the first week of February 2016 was 23 points, that’s an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
Advantages of the Acceleration Metric
There are several advantages to using acceleration as an indicator of productivity over traditional techniques such as FP counting:
Potential Disadvantages of Acceleration
Of course, nothing is perfect, and there are a few potential disadvantages:
We hope that you’ve found this blog post valuable.
One of the key philosophies of the Disciplined Agile (DA) toolkit is that it presents software development teams with choices and guides them through making the right choices given the situation they face. In other words, it helps them to truly own their process. Part of doing so is to choose the software delivery lifecycle (SDLC) that is the best fit for their context. In this blog posting we overview the DAD Exploratory lifecycle which is based in part on Lean Startup strategies.
This lifecycle can be applied in two ways:
The following diagram overviews the DAD Exploratory lifecycle. This lifecycle is followed by agile or lean teams that find themselves in startup or research situations where their stakeholders have an idea for a new product but they do yet understand what is actually needed by their user base. As a result they need to quickly explore what the market wants via a series of quick learning experiments.
Now let’s describe how the Exploratory lifecycle works. There are six activities to this lifecycle:
To summarize, the DAD process framework takes a flexible, non-prescriptive approach to software-based solution delivery. As a result of this philosophy DAD supports several development lifecycles, one of which is the Lean-Startup-based Exploratory lifecycle described in this posting. This lifecycle is typically followed in situations where you are unsure of what your user base wants, and sometimes even when you are unsure of who your user base (your customers) will even be.
An important philosophy within both the agile and lean communities is that a team should own its process. In fact, one of the principles behind the Agile Manifesto is “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” The idea is that teams should be empowered to tailor their approach, including both their team structure and the process that they follow, to meet the unique needs of the situation that they find themselves in. Teams that own their process will tailor it over time as they learn how to work together, adopting new techniques and tweaking existing ones to increase their effectiveness.
As with most philosophies this one is easy to proselytize but not so easy to actually adopt. When it comes to process improvement, teams will exhibit a range of behavior in practice. Some teams see process as a problem and actively seek to ignore process-related issues. Some teams are ambivalent towards process improvement and generally stick with what they’ve been told to do. And some teams see process improvement as an opportunity to become more effective both as a team and as individuals. This range of behaviors isn’t surprising from a psychology point of view although it can be a bit disappointing from an agile or lean point of view. It has led me to think that perhaps some teams choose to “own” their process but many more still seem to prefer to simple rent it.
The behaviors of people who rent something are generally different than those who own something. Take flats for example. When you rent a flat (called an apartment in North America) you might do a bit of cosmetic work, such as painting and hanging curtains, to make it suitable for your needs. But people rarely put much more effort than that into tailoring their rental flat because they don’t want to invest money in something that isn’t theirs, even though they may live in the flat for several years. It isn’t perfect but it’s good enough. When you own a flat (called a condo in North America) you are much more likely to tailor it to meet your needs. Painting and window dressings are a good start, but you may also choose to renovate the kitchen and bathroom, update the flooring, and even reconfigure the layout by knocking down or moving some walls. One of the reasons why you choose to own a flat is so that you can modify it to meet your specific needs and taste.
You can observe similar behaviors when it comes to software process. Teams that are merely “process renters” will invest a bit of time to adopt a process, perhaps taking a two-day course where they’re taught a few basic concepts. They may make a few initial tailorings of the process, adopt some new role names, and even rework their workspace to better fit the situation that they face. From then on they do little to change the way that they work together. They rarely hold process improvement sessions such as retrospectives, and if they do they typically adopt changes that require minimal effort. Harder improvements, particularly those requiring new skills that require time and effort to learn, are put off to some point in the distant future which never seems to come. Such behavior may be a sign that this “team” is not even be a team at all, but instead a group of individuals who are marginally working together on the same solution. They adopt the trappings of the method, perhaps they spout new terminology and hold the right meetings, but few meaningful changes are actually made.
Process owners behave much differently. Teams that own their process will regularly reflect on how well they’re working and actively seek to get better. They experiment with new techniques and some teams will even measure how successful they are implementing the change. Teams that are process owners will often get coaching to help them improve, both at the individual and at the team level. Process owners strive to understand their process options, even the ones that are not perfectly agile or lean, and choose the ones that are best for the situation they find themselves in.
The Disciplined Agile (DA) toolkit is geared for teams that want to own their process. The DA toolkit is process goal-driven, not prescriptive, making your process choices explicit and more importantly providing guidance for selecting the options that make the most sense for your team. This guidance helps your team to get going in the right direction and provides options when you realize that you need to improve. DAD also supports multiple lifecycles because we realize that teams find themselves in a range of situations – sometimes a Scrum-based lifecycle makes sense, sometimes a lean lifecycle is a better fit, sometimes a continuous delivery approach is best, and sometimes you find yourself in a situation where an exploratory (or “Lean Startup”) lifecycle is the way to go.
You have choices, and DAD helps guide you to making the choices that are right for you in your given context. By providing process guidance DAD enables your team to more easily own its own process and thereby increase the benefit of following agile or lean approaches.
A common question we get regarding Disciplined Agile Delivery (DAD) is “What makes DAD more disciplined than other approaches to agile?” It’s a fair question, particularly from someone who is new to DAD. This blog posting explores this question, explicitly summarizing the critical strategies that exhibit the greater levels of discipline in DAD as compared with what Mark and I see in many agile teams today.
It is clear that many common agile practices require discipline. For example, agile teams it takes discipline to hold concise, streamlined coordination/Scrum meetings; to consistently deliver business value every iteration; to test continuously throughout the lifecycle; to improve your process “in flight”; to work closely with stakeholders and many more things. Discipline is very important to the success of agile teams, see The Discipline of Agile for a detailed discussion, and DAD takes it to the next level in the following ways:
Adopting a disciplined approach to agile delivery requires the courage to rethink some of the agile rhetoric and make compromises where necessary for the benefit of the “whole enterprise” and not just the whole team. In our experience most agile projects make certain compromises that are not classically agile in order to get the job done. Rather than hiding this and fearing reprisals from those who would accuse you of regressing to a traditional approach, it is better to have the courage to take a pragmatic approach to using agile in your situation.
Effective application of DAD certainly requires discipline and skill, but in our experience the key determinant of success is the ability and willingness of the team to work well together and with stakeholders, both within and external to the team. For more writings about discipline within DAD, select “Discipline” from the blog category list.
Early in the lifecycle, during the Inception phase, disciplined agile teams will invest some time in initial requirements envisioning and initial architecture envisioning. One of the issues to be considered as part of requirements envisioning is to identify non-functional requirement (NFRs), also called quality of service (QoS) or simply quality requirements. The NFRs will drive many of your technical decisions that you make when envisioning your initial architectural strategy. These NFRs should be captured someone and implemented during Construction. It isn’t sufficient to simply implement the NFRs, you must also validate that you have done so appropriately. In this blog posting I overview a collection of agile strategies that you can apply to validate NFRs.
A mainstay of agile validation is the philosophy of whole team testing. The basic idea is that the team itself is responsible for validating its own work, they don’t simply write some code and then throw it over the wall to testers to validate. For organizations new to agile this means that testers sit side-by-side with developers, working together and learning from one another in a collaborative manner. Eventually people become generalizing specialists, T-skilled people, who have sufficient testing skills (and other skills).
Minimally your developers should be performing regression testing to the best of their ability, adopting a continuous integration (CI) strategy in which the regression test suite(s) are run automatically many times a day. Advanced agile teams will take a test-driven development (TDD) approach where a single test is written just before sufficient production code which fulfills that test. Regardless of when tests are written by the development team, either before or after the writing of the production code, some tests will validate functional requirements and some will validate non-functional requirements.
Whole team testing is great in theory, and it is strategy that I wholeheartedly recommend, but in some situations it proves insufficient. It is wonderful to strive to have teams with sufficient skills to get the job done, but sometimes the situation is too complex to allow that. There are some types of NFRs which require significant expertise to address properly: NFRs pertaining to security, usability, and reliability for example. To validate these types of requirements, worse yet even to identify them, requires skill and sometimes even specialized (read expensive) tooling. It would be a stretch to assume that all of your delivery teams will have this expertise and access to these tools.
Recognizing that whole team testing may not sufficiently address validating NFRs many organizations will supplement their whole team testing efforts with parallel independent testing . With this approach a delivery team makes their working builds available to a test team on a regular basis, minimally at the end of each iteration, and the testers perform the types of testing on it that the delivery team is either unable or unlikely to perform. Knowing that some classes of NFRs may be missed by the team, independent test teams will look for those types of defects. They will also perform pre-production system integration testing and exploratory testing to name a few. Parallel independent testing is also common in regulatory compliance environments.
From a verification point of view some agile teams will perform either formal or informal reviews. Experienced agilists prefer to avoid reviews due to their inherently long feedback cycle, which increases the average cost of addressing found defects, in favor of non-solo development strategies such as pair programming and modeling with others. The challenge with non-solo strategies is that managers unfamiliar with agile techniques, or perhaps the real problem is that they’re still overly influenced by disproved traditional theories of yesteryear, believe that non-solo strategies reduce team productivity. When done right non-solo strategies increase overall productivity, but the political battle required to convince management to allow your team to succeed often isn’t worth the trouble.
Another strategy for validating NFRs code analysis, both dynamic and static. There is a range of analysis tools available to you that can address NFR types such as security, performance, and more. These tools will not only identify potential problems with your code many of them will also provide summaries of what they found, metrics that you can leverage in your automated project dashboards. This strategy of leveraging tool-generated metrics such as this is a technique which IBM calls Development Intelligence and is highly suggested as an enabler of agile governance in DAD. Disciplined agile teams will include invocation of code analysis tools from you CI scripts to support continuous validation throughout the lifecycle.
Your least effective validation option is end-of-lifecycle testing, in the traditional development world this would be referred to as a testing phase. The problem with this strategy is that you in effect push significant risk, and significant costs, to the end of the lifecycle. It has been known for several decades know that the average cost of fixing defects rises the longer it takes you to identify them, motivating you to adopt the more agile forms of testing that I described earlier. Having said that I still run into organizations in the process of adopting agile techniques that haven’t really made embraced agile, as a result still leave most of their testing effort to the least effective time to do such work. If you find yourself in that situation you will need to validate NFRs in addition to functional requirements.
To summarize, you have many options for validating NFRs on agile delivery teams. The secret is to pick the right one(s) for the situation that you find yourself in. The DA toolkit helps to guide you through these important process decisions, describing your options and the trade-offs associated with each one.