I am currently part of a Telecommunications company that is starting a program office to support all IT projects within our CIO. Our current scope includes $600m of projects over the next 4 years.
I've been asked to help develop the metrics to measure the success of the program office itself (there are separate metrics for each of the programs/projects).
Are there any standard metrics defined for PMO success measurements? Does anyone have any suggested metrics we could track?
We've developed things like: customer surveys; response time to risks/issues/scope changes; completeness and timeliness of scorecard reporting. Are there any others that would be recommended? Saving Changes...
Mark MullalyPresident| Interthink Consulting IncorporatedToronto, Ontario, Canada
Kathy:
The challenge of defining appropriate success measures for a PMO is a significant one. Part of this problem stems from the fact that there is no common definition of what a PMO does, and while there are a relatively well defined number of reasons why a PMO is defined, it's measures of success can be more elusive.
If you could give me a bit more background as to both the functions and the rationale for implementing the PMO in your organization, I would be happy to offer some more specific suggestions.
The following challenges are driving the need to develop an "FMO" Program Office:
- Lack of understanding across the business of the "FMO" program plan, impacts and benefits - delays experienced in getting projects approved and launched - Lack of consolidated program data to drive prioritization and decision making by senior executives - Lack of coordination for initiatives - Lack in the standardization in the project management disciplines across projects
Essentially, the "FMO" is made up of a portfolio of business capabilities that our business is trying to deliver. Plans have been developed that apply process, people and technology changes to deliver on the business capabilities. Our program office is responsible for coming up with the "migration" plan for implementing these changes. Part of that responsibility is during the execution phases where we track, monitor and manage cross program risks, issues and change controls, provide performance management reporting, resource forecasting and, where possible, resource management (althought that's very limited as there's another department that actually owns that role).
What we've been asked to do is provide some metrics for senior executives to understand how successful we are. The challenge is coming up with metrics we have any control over. We've come up with the following:
1) Response times to escalated risks/issues/change requests 2) Customer satisfaction surveys (where we would survey our key stakeholder groups 3) Scorecard presentation - timeliness; completeness; etc. Saving Changes...
Mark MullalyPresident| Interthink Consulting IncorporatedToronto, Ontario, Canada
Kathy:
By the sounds of it, part of the challenge you are encountering is a lack of control/responsibility for the full outcome of the program. What is not clear -- and it may be a dual responsibility, but I'm asking just to confirm -- is that your PMO has both implementation responsibility and central reporting/resource estimation/resource management (sometimes, as you say...) Is the reporting for the overall work (including the development) or just for your implementation?
In any event, as well as making sure that you define performance measures for what you are truly responsible for and have control over, you should also have measures in place to monitor the hand-overs from other teams/departments/organizations on whose output you depend. This will allow you to demonstrate your performance in the context of hand-offs, as well as the actual results of your PMO. Especially where control is limited, consistency is lacking or there is a risk of the PMO becoming a scapegoat, this becomes critical.
Some of the measures you might want to contemplate include:
hand-off dates
acceptance against quality/deliverable criteria
rework effort
timeliness of input to the reporting cycle
number/magnitude of changes and issues -- a means of evaluating initial plan effectiveness
variances of resource estimates (budget vs. actual)
resource availability (variance against initial allocation)
One technique that I have found to be useful in developing performance measures is the technique of "Goal-Question-Measure":
Goal: What outcome are you trying to demonstrate? What objective are you trying to deliver on? Or from a risk management perspective, what event are you seeking to avoid?
Question:What question(s) will demonstrate your attainment of the defined goal.
Measure:What measures can you define to answer the questions?
To be useful, it is important to define and understand the objectives your PMO is being looked to in order to provide, separate from the other activities within the organization.
I hope this helps. If you would like any additional input, please let me know.
Mark -- I'm in general agreement with your comments, but I do feel a need to comment on some of measures you offer for contemplation...
? hand-off dates
Focus on task due dates will drive a train schedule mentality in projects and a set of behaviors that will be geared to meeting, but not exceeding expectations. Task hand-off dates are a source of Parkinson's Law. If thing's go well, there's no sense of urgency. If things don't go well, there's no buffer of accumulated good luck to use to protect the real project promise hand off -- project completion.
? variances of resource estimates (budget vs. actual)
Similar to dates, measuring estimate variance will run the risk of driving behaviors to make actuals match estimates. In any event, this also depends on the oxymoronic and quixotic concept of the "accurate estimate."
Rather than track accuracy of estimates, I would suggest that if one is using the common sense approach of range estimates (a practice not limited to my beloved Critical Chain), a longer term trend of reduction of the variation of estimates (the difference between the safe and agressive estimates) might be worth watching as a measure of understanding and driving out sources of that uncertainty. (In my Critical Chain world, this would result in shorter project promises as buffer sizes are reduced.)
I had a co-worker send me a link to a site that explains project metrics quite well. With a little work they can easily adapt to programs. Thought I'd share the link here. Hopefully I'm doing it right and it's not a problem for anyone.
Just found this and its the answer to my prayers - special thanks to Bruce who answered the question I've been grappling with for the last 3 months! Saving Changes...
Similarly to Kathy Z. (on June 11th, 2004), I was set to develop the metrics to measure effectiveness on IT project/portfolio basis for a financial services company.
