Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
There are quite a few frameworks, claiming to be Agile, that promise do deliver twice as much in half the time. The state of Agile reports and the Chaos reports have also very optimistic views on the effectiveness of Agile transition.
I am using Agile frameworks since 2002 when I was in software development. We were in the "perfect storm" using XP and then Scrum in a mid size software company were we had full support from the owners and direct interaction with the users.
Since that time I worked with many Scrum teams and I tried to compare their productivity. It was hard even inside the same organisation.
What are some meaningful metrics that can be used to compare Agile teams? Saving Changes...
Team-based comparison is somewhat opposed to the concept that maturity is a context-specific thing. Benchmarking can certainly be done organizationally (e.g. ROI of portfolio) but doing it at the team level can create unhealthy competition and gaming the system.
No two products are the same so comparing teams is an apples to oranges exercise unless you can strictly control all the variables (e.g. project complexity, team member experience, project constraints).
Kiron
...
1 reply by Stelian ROMAN
Jan 18, 2019 9:39 AM
Stelian ROMAN
...
Team based is ok, many teams are benchmarking inside their organisation. I agree that it requires maturity but it's not bad. Comparing team members is what must be avoided. Benchamrking sprints for the same team is a very common practice, although I am not sure that there is any value if they use SP.
Unlike SP FP can't be gamed but counting FP requires skills and experience hard to find in an Agile team.
there are a lot of weightings that can be applied for complexity, technology etc
It may be impossible, and is quite difficult in any case. This is partly due to the differing requirements of each sprint, which can require different team members (or outside dependencies) as well as the need for team members to "wear different hats" across sprints. In addition, some companies have teams dedicated to selected subject matter, such as finance or marketing, which also makes comparisons difficult. I suggest that it may be easier to compare/contrast metrics for teams across sprints rather than across teams.
If some method of rating requirement difficulty, such as function points, is used, then comparisons can be made more easily. Whether it is worthwhile to count function points for every item in the backlog is also debatable.
...
1 reply by Stelian ROMAN
Jan 18, 2019 9:34 AM
Stelian ROMAN
...
It is possible but it is not easy. There is a methodology that works for any project but require experts in project sizing. FP should not be counted at story level, that will be a waste. In my experience 1 FP was almost a week work, a user story could go to 1 hr or less. The margin of error at a fine granularity defies the purpose of measuring.
Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Jan 18, 2019 8:43 AM
Replying to John Herman
...
It may be impossible, and is quite difficult in any case. This is partly due to the differing requirements of each sprint, which can require different team members (or outside dependencies) as well as the need for team members to "wear different hats" across sprints. In addition, some companies have teams dedicated to selected subject matter, such as finance or marketing, which also makes comparisons difficult. I suggest that it may be easier to compare/contrast metrics for teams across sprints rather than across teams.
If some method of rating requirement difficulty, such as function points, is used, then comparisons can be made more easily. Whether it is worthwhile to count function points for every item in the backlog is also debatable.
It is possible but it is not easy. There is a methodology that works for any project but require experts in project sizing. FP should not be counted at story level, that will be a waste. In my experience 1 FP was almost a week work, a user story could go to 1 hr or less. The margin of error at a fine granularity defies the purpose of measuring.
...
1 reply by John Herman
Jan 18, 2019 10:16 AM
John Herman
...
You are free to count Function Points in a manner of your choosing (as long as you are consistent). However, according to DDWAY (ddway.it), a function point is closer to 1.13 hrs +/- 10%. (www.webratio.com).
Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Jan 18, 2019 8:39 AM
Replying to Kiron Bondale
...
Stelian -
Team-based comparison is somewhat opposed to the concept that maturity is a context-specific thing. Benchmarking can certainly be done organizationally (e.g. ROI of portfolio) but doing it at the team level can create unhealthy competition and gaming the system.
No two products are the same so comparing teams is an apples to oranges exercise unless you can strictly control all the variables (e.g. project complexity, team member experience, project constraints).
Kiron
Team based is ok, many teams are benchmarking inside their organisation. I agree that it requires maturity but it's not bad. Comparing team members is what must be avoided. Benchamrking sprints for the same team is a very common practice, although I am not sure that there is any value if they use SP.
Unlike SP FP can't be gamed but counting FP requires skills and experience hard to find in an Agile team.
there are a lot of weightings that can be applied for complexity, technology etc Saving Changes...
It is possible but it is not easy. There is a methodology that works for any project but require experts in project sizing. FP should not be counted at story level, that will be a waste. In my experience 1 FP was almost a week work, a user story could go to 1 hr or less. The margin of error at a fine granularity defies the purpose of measuring.
You are free to count Function Points in a manner of your choosing (as long as you are consistent). However, according to DDWAY (ddway.it), a function point is closer to 1.13 hrs +/- 10%. (www.webratio.com).
...
1 reply by Stelian ROMAN
Jan 18, 2019 5:07 PM
Stelian ROMAN
...
There is not that much flexibility in counting FP. Professional FP counters won't have more than 2% difference between counts.
According to IFPUG the average cost is $1000/FP. I worked on projects where it was close to 4k.
can you please share the link to the page that has the 1.13h +/-10%?
I tried the web address and it is redirected to a financial company.
Pench BattaEnterprise Lean Agile DevOps Coach /SAFe Program Consultant (SPC6)| Capgemini, Inc.Bentonville, Ar, United States
Not a good idea to compare agile teams. It is like comparing oranges and apples! Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Jan 18, 2019 10:16 AM
Replying to John Herman
...
You are free to count Function Points in a manner of your choosing (as long as you are consistent). However, according to DDWAY (ddway.it), a function point is closer to 1.13 hrs +/- 10%. (www.webratio.com).
There is not that much flexibility in counting FP. Professional FP counters won't have more than 2% difference between counts.
According to IFPUG the average cost is $1000/FP. I worked on projects where it was close to 4k.
can you please share the link to the page that has the 1.13h +/-10%?
I tried the web address and it is redirected to a financial company.
it can also be reached via the ddway.it site, but that's a direct link
Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Pench, not only that comparing any teams is a good idea but it is must when you adopt an Agile framework. The main purpose of Agile is to uncover 'better ways' (please read the manifesto). Better must be quantified, at least in terms of cost/benefit. Agile doesn't mean that everything that was done prior to 2001 is not good. Process Improvement is one of the practices that won't change because of Agile, same as mathematics. Some Agile people want to prove that 5 4 (smaller than). Metrics in Agile is a very sensitive topic. In my opinion those who understand metrics know that Agile is more expensive than planned approach. A good indication that people did compare Agile Teams is the emergency of old Lean practices over the chaotic Scrum but implementations. Kanban and SAFe are Lean frameworks, not that much Agile.
BTW, even apples and oranges can be compared in terms of cost.
...
1 reply by John Herman
Jan 19, 2019 11:05 AM
John Herman
...
Indeed, the Upper Management and HR folks are very interested in comparing Agile teams (and individuals). It (presumably) helps them manage performance bonuses and raises, as well as in helping them identify people who are underperforming and need help improving. Even if you have a general qualitative grade for the performance of individuals and teams, those groups will push you for quantitative data, as it may be required to support inquiries of their decisions to award raises and bonuses and let employees go.
it can also be reached via the ddway.it site, but that's a direct link
...
1 reply by Stelian ROMAN
Jan 23, 2019 3:57 PM
Stelian ROMAN
...
"The aim of the paper is to evaluate the productivity level of projects developed and deployed with the application development platform IFML WebRatio by our Competence Center." Of course that they can do it for ONE project at 1.13 hrs/FP. There is a very ambitious framework called project half double that claims that they can deliver a project in half the time and at half the cost. I couldn't find any hard evidence beyond their white papers. One of the famous Agile books claims the same double productivity at half the cost. I have 30 years experience and I haven't seen that kind of improvement. A 20% improvement is a big achievement, 5-10% means that you recovered the investment but going from days to 1 hr per FP means that you are the Wizard of OZ.
IFPUG has a pretty good database that can be used for benchmarking. That is my preference for industry benchmarking, although it is pretty optimistic. 1 because nobody will report a bad project and 2 because a company that used professional FP counters will do as part of a process improvement initiative. I know is a very productive framework (https://www.lansa.com/) that can deliver faster and cheaper at less than 1 hr/FP. It is 30 years old and was initially developed for AS/400. It does what it promise BUT ... you have to use their language, have a very systematic design process and have brilliant developers. Having the last 2 helps regardless the framework but that's not the norm.
I am very cautions using promotional white papers for benchmarking.
Pench, not only that comparing any teams is a good idea but it is must when you adopt an Agile framework. The main purpose of Agile is to uncover 'better ways' (please read the manifesto). Better must be quantified, at least in terms of cost/benefit. Agile doesn't mean that everything that was done prior to 2001 is not good. Process Improvement is one of the practices that won't change because of Agile, same as mathematics. Some Agile people want to prove that 5 4 (smaller than). Metrics in Agile is a very sensitive topic. In my opinion those who understand metrics know that Agile is more expensive than planned approach. A good indication that people did compare Agile Teams is the emergency of old Lean practices over the chaotic Scrum but implementations. Kanban and SAFe are Lean frameworks, not that much Agile.
BTW, even apples and oranges can be compared in terms of cost.
Indeed, the Upper Management and HR folks are very interested in comparing Agile teams (and individuals). It (presumably) helps them manage performance bonuses and raises, as well as in helping them identify people who are underperforming and need help improving. Even if you have a general qualitative grade for the performance of individuals and teams, those groups will push you for quantitative data, as it may be required to support inquiries of their decisions to award raises and bonuses and let employees go.
...
1 reply by Stelian ROMAN
Jan 23, 2019 4:00 PM
Stelian ROMAN
...
My question is about comparing Agile teams not individuals. I believe that any team should define a baseline to measure their efficiency. It is also mandatory for the organisation to benchmark teams working on the same product as a way of identifying good practices.
As an individual comparing yourself with others is a sign of maturity and professionalism. Everyone has strengths and weaknesses.
I recommend to read how the pair programming started. The 'best' developer tried to show off and discovered that you can always learn from less skilled and experienced.
