I have been contemplating this question for quite some time now.
In the course of this contemplation, I have come across many opinions, theories and software tools that indicated that objective estimation is indeed possible.
The basic recipe seems simple: Define the scope of the software to be built using an objective, unambiguous method (Specification); Define test cases the will exercise the resulting software in order to determine correctness, in such a way that there is traceability to the specification (Test cases); Design a solution that satisfies the specification; Build the software according to design; Test the resulting software; Correct errors; Support, organise and manage the effort, risk and scope and deliver the expected outcome within time and budgetary constraints. Variations on the theme include ways to manage risk better (Iterative SDLCs etc.).
The crux of the recipe is the formulation of the scope of the work – Scope translates into specifications from which design and implementation must follow. The proposition is: If you can define requirements formally, using a grammar that is unambiguous and rich (enough to describe the problem domain), then you can determine the size of the required code (a measure of complexity like KLOC, Function Points, etc) objectively, with an associated confidence level. Statistical variations (relating to the confidence level of the estimate) are in linked to the process capability of the company/team/individuals that build the software (See Capability Maturity Models)
From here it is simple – size translates through effort into time and money (plus the cost of support services and capital equipment). Given an objective estimate all you have to do is to manage the scope and the effort and you are guaranteed success.
Nothing new so far. This is the view that I shared until recently with many people, until I read an article on algorithmic complexity (attached), which shattered this dream.
The implications of this article are profound, and should be noted by all Project Managers in the software development field.
I have seen this article before as well. It was interesting but I wondered how much is theory and how much is proven. Though just because it may not have been proven, does not mean the theory is wrong! I read this as the dreaded "art of estimation". This is another example that estimation is about 40% science and 60% art (my numbers).
As we all know there can be a few variables that add uncertainty even when using useful historical data.
Domain. I can use the same N-tier framework, screen architecture etc across domains. The business rules and data model can be drastically different, hence the complexity.
Team Skill set/experience. Can vary from project to project. There are of course others variables.
Maybe I am taking to simple a look but I read this as boiling down to project risk. The estimate is uncertain. Why we think it is uncertain helps determine the mitigation plans. Perhaps a Monte carol simulation but that is not a guarantee. Using proof of concept or pilot project to help better understand the technical challenges for the team. Bring in an expert or as you have suggested variations on SDLC's. Really anything to help remove the uncertainty of the estimate is game.
I think the question we need to ask ourselves is will Objective Estimation work in my case. Do I have a lot of estimation risk? New technology, no historical data, new domain, don't know the team well ( can you trust their estimates?), is the team motivated or burned out, etc. As you point out the safety valve may be risk management to support the project plan and/or development approach.
Your comment that this should be noted by all PM's is correct. I believe Objective Estimation can be done quite successfully in many cases. I have done it successfully as I am sure many have as well. I do agree that it does not work under many circumstances. Which is a problem for us PM's. You can manage the project properly from a-z as you have outlined. But if your estimates are way off... you are done! Hopefully we are able to quantify and qualify the estimation risks, which can be difficult. Saving Changes...
Appreciate that you have made accurate estimates in the past. What objective method did you make use of?.
Stuart Saving Changes...
Michael WoodProject Manager / Business Analyst / Business Process Improvement Guru| Independent ContractorGig Harbor, Wa, United States
Can't help but to put my 2 cents in. Estimating the time it will take to code a specification depends on four basic factors:
1 Complexity 2 Completeness of the Specification 3 Competency of the Programmer 4 Tools used Complexity in code is a function of number of I/Os performed, parameters to be managed and intricacy of algorithms.
Thus a program can be rated in terms of its complexity by assigning a weighting to it based on the above factors. Each weighting point can then be associated to an effort level for an average programmer given a complete specification (i.e. one the leaves little or nothing to the imagination). In order to assign effort level the development environment must be known. Adjustments factors can be assigned for competency levels. However it is impossible to create adjustment factors for erroneous or incomplete specifications. As soon as programmer becomes designer the numbers of variables that can impact the finished product become unruly.
The complication to all this is that virtually all specifications are weak. Too many programmers want to create in the code as they fancy themselves as architects not construction workers. Practically speaking the weighting factors must be calibrated to specific environments. This will require some benchmarking and assessment of the programming staff. I think it is also important to recognize that some programmers are competent in one area of application development and not in others. Some are better at writing reports than data entry. Some do better at writing interfaces than reports. While others are outstanding at coding periodic batch processes.
I have estimated projects at the Micro and Macro levels with equal success. However my success at the Macro level only comes from doing this for over 20 years. On final note, remember that sizing and projecting time to complete are two very different animals. And don't forget EGOS the one factor that cannot be calibrated or factored with metrics (at least not publicly) Saving Changes...
Are we talking order of magnitude or design level estimates. I am assuming design level from your detailed description of how to get there.
We have not used anything as fancy as function points or KLOC. Though I have wanted to try function points. What is your experience with them? We normally use bottom up estimating based on the WBS and experience of having done it before, hopefully with a hint of historical data. Phased and iterative SDLC’s help as well.
I think your approach is sound. We follow a very similar path. Even with large projects (as high as 10 person years) we have been rather successful at design level estimating using very much the same approach as you originally described. My belief is a large part of that success comes from having very experienced people do the estimates.
There’s the rub - experience.
You don’t always have experience with new technology or different team members. How do you benchmark even against the industry if the technology is quite new? I have seen terrible results when the experience factor goes away. It then becomes very difficult to determine the complexity and large projects just add to the problem.
I helped create our internal estimation course that we hope will help spread the knowledge. It really is just some best practices to practice. You can’t teach experience in a course.
Michael summarizes it well. Both micro and macro depend on experience, particularly macro estimates. Experience plays a part in the factors you can see ( the 4 Michael suggested) and the ones you can’t, that list can be long. Michael listed some of them. As Sizing and time to complete are similar, yet have different variables that affect the estimate. If sizing is off significantly, forget time to complete being accurate!
Micro is easier as you hopefully have a very defined requirement/specification and know the team. Macro has far less information so historical info and experience are about all you have. That’s why I like presenting macro estimates as a range, sometimes a very large range.
Stuart, why do you say the dream is shattered? I definitely agree it brings to light the complexity of estimating and the importance of putting proper time into the estimating process.
It seems we agree so far - no accurate estimation process without experience, and the process is subjective.
I have long been a proponent of the CMM for Software (CMU/SEI-93-TR-24). In this very important work the following is claimed:
“[In a mature organization] There is an objective quantitative basis for judging product quality and analysing problems with the product and process. Schedules and budgets are based on historical performance and are realistic”
What I have read leads me to believe that the claim of objectivity is false. Hence the dream shattering.
My belief that is can be done subjectively, and that organisational maturity helps has in no way been altered. One simply needs the right combination of people, experience mix, their commitment, planning models and tools to succeed in delivering against the estimate (plan)
Stuart.
P.S The message in this article is VERY clear. Claims laid to objective estimation are unethical, and in my view Ethics may well be the most important issue in Project Management. Saving Changes...
You list complexity as a factor. If complexity cannot be feasibly calculated, then how is it a usefull parameter in estimation.
From the article:
“KCS noncomputability theorem: there is no algorithm for computing the AC of an arbitrary string.” And “Rephrasing this for our purposes, there is no algorithm for finding the shortest program with a desired behavior.”
i.e. no matter how detailed a specification for a piece of code is, there is no mathematical means of telling what the lower bound on the number of lines of code is, and therefore the shortest time it will take to write. Therefore no objectivity.
QED Saving Changes...
Michael WoodProject Manager / Business Analyst / Business Process Improvement Guru| Independent ContractorGig Harbor, Wa, United States
Clinically speaking complexity factors are hard to nail down. However, from a practical view point complexity can be assessed. Simple file maintenance routines and listing reports are simple compared to data entry for example. Its not precision I am after but more an impact to the project. Think of each point of increasing complexity being translated into a percentage adjustment against a simple programmatic process (read a record, format it and print it. Or, enter a record on a screen and save it with only an update or two for indexes. If these are equal to a complexity of one and it takes a half day to build a program perform the function then its possible and practical to begin creating adjustment factors that can be applied to complexities in formatting, calculating, writing, parsing, etc. Lines of code, given the tools of today, is not the best metric for estimating. Functional complexity has far more merit in my opinion. However with all that said, in the real world the number of variables that are outside of our control that can impact an estimate make precision no more than an intellectual pursuit. Better to GAP an estimate and then calibrate towards it. Since most IT projects run 2 to 5 times over budget and only 1 in 8 ever get implemented it would appear that the best learning comes from studying those who have consistently delivered on time and on budget. I believe that successful developers focus far more on human factors than mathematics to achieve success. Saving Changes...
Size estimation is a holy grail of the CMM. The full power of the size estimation seems to manifest only itself while you track it. (Original estimation is always better or worse guess; exact estimation is an oxymoron, isn’t it?) If you have some reasonable expression/link that binds size estimation of the work product with its work duration you use to build your schedule, by tracking it you can (a) validate the quality of the estimating procedure itself, and more importantly, (b) discover early potential effects of the faulty duration related data, suspected by the revealed estimation data discrepancies (estimation process or the actual size…) Saving Changes...
