I noticed recently in Australia 2 big projects which delivered on time, under budget but when the chips were down, the systems failed. Both appear to be due to software being unable to cope with a larger than expected load. How can project managers "recover" from a situation where the perception is that something was missed?
Saving Changes...
Frank WintersPhotographer and ConservationistSandwich, Ma, United States
The answer to this question is "they can't." Capacity planning is an aspect of estimation that must be gotten right. It is far better to over engineer a system so that it can handle loads larger than those encountered at first than to deploy a system that fails as soon as it is under load.
Modeling and prototyping can be used to prevent this kind of failure, and these tools must be used early in the lifecycle, rather than waiting for load testing after the system is designed and built. Saving Changes...
I agree - in the planning stages assume that the system will be overloaded. However there are two other things which may help you. Firstly, ensure that you know exactly what the system is supposed to DO - the specs alone won't tell you this, you'll have to ask. You may find that the specs do not sufficiently address the customer's need. Secondly, you set clear parameters on your "warranty" of the sysetm. Just like a car is good for so many kilometres, so too your system will function as it should within certain boundaries. This is really customer expectation management, and should commence when the project is first scoped. Saving Changes...
System scalability and performance problem is not new in software industry. Project manager should have included performance criterion and expected number of users/processes in initial scope or requirement documents. This would help system architect to decide the design and testing lead would also include load or stress test cases, so these issues would have identified before final release to client.
In this case project manager had missed in requirement capture and quality planning. There are many automated software testing tools available to simulate large number of user groups.
Saving Changes...
Isabelle BadinandAgile Delivery Manager| AvivaLondon, United Kingdom
I am surprise no one mentions cost. A system that has to be available 24 * 7 and support a very large number of users and is scalable is very expensive. There is usually a push from budget holders for reducing over engineering. Especially as validating performance and load requirements can be very tricky. Maybe the project manager went through all the hoops but the requirements on those 2 projects were wrong and underestimated the popularity of the site or did not model properly usage patterns (to uncover peaks etc...). Saving Changes...
I believe the fundamental question remains unanswered. The perception management is an ongaoing activity where communication to stakeholders, sometimes seemingly trivial, is critical to the success of a project. Afterall success is always a relative term based on the expectations set at the begining of the project or its alignment during the execution of the project. Since cost is a critical factor in determining what tools to use and exclude, benchmarking against these criteria is also important that must be communicated to the stakehodlers and sponsors. In situations where project is apprently a success in terms of delivery timeline and cost, but a failure in terms of aniticipated performance, it is best to first acknowledge the deficiencies in the system and then go about educating the stakeholders and sponsors. Explaining them the architectural scalability capability of the system. It is important for the stakeholders to realize that any given system cannot be optimized to nth. If a car has a limitation on the performance vis-a-vis its horse power so has a software system. Saving Changes...
I'm afraid I'm with the customer on this. From what has been said, I’d consider these projects to have failed.
I’ll continue the use of the car metaphor. If my 2002 Honda Civic stops running when I’m on my way down the freeway I’m not going to call it a success.
Now if I take my car off-road and go hunting for Elk with it, I might be the one to blame. So the question to ask is, “Where they Elk hunting with a Honda Civic.” Several questions need to be answered to determine this:
1. Was the expected load of the system specified?
2. Was the expected environment documented in the requirements?
In other words, were the requirements complete? My guess is they weren’t. If this is the case a failure occurred during requirements gathering.
Another point that was made in the posts below is that there was a conflict between budget and quality assurance of the product. If you have two requirements that conflict they must be resolved or the project must be cancelled. If you can’t build the product for the desired price then you have such a conflict.
Saving Changes...
Think of it this way .... in 2001 a Canadian airline had a plane run out of fuel over the Atlantic (not related to 9/11). Although there were many factors in the plane making it to the Azores, I'm sure the original project team(s) at Airbus were a big contributing factor in it.
Although airplane specs are very complex, the most basic requirement is to move X number of people X number of miles. Had the project team stuck with that, they likely would have put together a great aircraft. They could have demonstrated the ability for the aircraft to meet or exceed those expectations. However, they take on the processes necessary to dig that little bit deeper, and find out what all the requirements are (even the unspoken ones).
As a result of exceeding the expectations a 400K lbs aircraft glided without power for 20 minutes and no-one died. It never ceases to amaze me how you can make that much weight fly with power, let alone without power.
Now I'm no airplane expert, but I like the analogies when talking about risk or project process. The engineers at Airbus have processes they apply when designing aircraft. It's my opinion when you apply process to your project, you should ensure you use enough tools to find the hidden requirements.
For example, the Airbus engineers would have identified the risk of running out of fuel at a critical time. They would have mitigated it by ensuring the airplane can glide for an acceptable time, and the controls have some form of power to allow the pilots to land it.
Although for most of us lives are not at stake, you should still go through the steps. For example, a WBS showing system sizing/testing may have shown a task required to prevent the failure. Alternatively a risk assessment may have uncovered a risk that large production files may come to the server if all the cards line up.
Regardless of your approach ... ensure you have all the right stakeholders present, including your customer(s).
Saving Changes...
Frank WintersPhotographer and ConservationistSandwich, Ma, United States
Good analogy, Michael. My friend Jeff Schwartz has written for GH using a similar analogy. By the way Gordon Bethune, CEO of Continental expands on the goals you mention for airlines -- "get the passengers from pioint X to Point Y (on time he says) and with their underwear.
Cheers,
Frank Saving Changes...
Frank WintersPhotographer and ConservationistSandwich, Ma, United States
Yes , I have seen this happening a lot as because the management donot put emphasis on QA - How many of you think that a dedicated QA is a prudent necessity of an Engineering effort and what part of the schedule would you allocate? Saving Changes...
