How often have you looked at a set of data points and wondered what the… point was of capturing them at all? I’ve seen PMOs with measures for how many projects were closed in a month but not measures for whether those closed projects were done on time and on budget, for example. Sometimes I think our desire to catalogue and record everything goes a little bit too far. There isn’t any point in capturing data that no one cares about.
This graphic shows three core principles for creating and using metrics. They are universally applicable but good for project managers because we have to think up (and then find ways to track) KPIs and success criteria measures for our projects. And if you’re in a PMO role, you must have a bunch of data points you are collecting too – I wonder how many of them make it into the reports you produce for stakeholders? And then how many stakeholders actually read and care about them?
For more on the topic of what makes a good measure for your project, take a look at this article, which digs into some examples and how to make sure that you are creating measures that are useful, in more detail.
How do you track the performance of your project? Here are 5 ways that you can do it.
1. Fixed Formula
This is pretty easy. Assign a fixed percentage when the work starts. Then make it up to 100% by allocating the rest when the task finishes.
The Pareto principle is a good one to use if you don’t know how to split your task percentages. Assign 20% complete to the job when it starts, and the remaining 80% when your team member reports the work complete (because at that point it’s 100% complete).
It’s simple to work out but it’s not terribly accurate. It’s only good for small pieces of work and short tasks where it would be too difficult or not worth it to work out percent complete across a couple of days. It can also be used where the task is no longer than a week long. If you are updating your project plan once a week, and the task is no longer than a week long, the task is either started or finished so the 20/80 split (or 50/50 or whatever you think is appropriate) works out pretty well.
2. Weighted Milestones
When you’ve got plenty of milestones along the way, you can work out project performance by tracking how many you’ve hit.
In other words, you can pre-assign progress (percent complete) to certain milestones or parts of tasks. When you hit Milestone 1 you can say the project is 15% complete, at Milestone 2 it goes up to 20%, at Milestone 3 you’re 65% complete and so on. Until your final milestone at project completion where your project (or task) gets updated to be 100% complete.
This is also a way to split payments to vendors – many contracts have a schedule of payments linked to the achievement of key milestones. Your budget could be weighted in the same way as how you track performance.
3. Percentage Complete
We’ve talked about % complete already, but this version of it is just based on the project manager’s
This isn’t hugely accurate either – although it depends on the project manager. It takes experience to be able to pick a percentage out of the air and have it reflect reality. Generally, project management tools can help with this by playing back to you the % complete of your project schedule, so at least in that case you should have something underpinning the number you give.
4. Percentage Complete with Gates
This is similar to weighted milestones but instead of waiting to hit the ‘gate’ point, you can report any percentage complete up to the approved limit for that milestone.
For example, when you hit Milestone 1 you can say the project is 15% complete, as we saw above. With weighted milestones, until that point the project would be 0% complete. With gates, you can set a % complete every day if you like, working up to 15% at the point of hitting the gate (the target milestone date or achievement).
It’s like a blend of using your professional judgement but being constrained to not say you are too far ahead because you can only ever hit a certain percent complete through the nature of where you are on the project.
It sounds complicated to explain but this is my favourite approach for measuring project performance.
5. Level of Effort
Finally, you can track effort against the elapsed time. Alternatively, you can track against some other task or work package on the plan. For example, ‘Complete Testing Documentation’ might be linked to ‘Complete Testing’ and the two activities progress in parallel.
You’d track performance for ‘Complete Testing’ and then, as you know that testing documents are being updated as you go, apply the same % complete to ‘Complete Testing Documentation.’
Which one(s) of these do you use? And which do you avoid? Do you use these as standalone techniques or do they link to your Earned Value Management activities? Let us know in the comments below!
The Gartner PPM and IT Governance Summit was held in London last week. Lars Mieritz, a Gartner analyst, gave a good presentation about using metrics for communicating project success and effectiveness. He started out by saying that the top concerns of C-level executives and their direct reports, based on their research, were:
Gartner has many years of research in this area and he shared an interesting trend: in 2003 the top concern of CIOs with regards to IT strategy was the need to improve governance and provide leadership. In 2013 the top concern was delivering business solutions. There’s been a shift towards integrating IT more effectively with the rest of the business.
Why metrics count
Project management metrics based around portfolio performance help improve the perception of success. When researchers asked people what they thought of their PMO the views were pretty poor:
As Lars said, he hasn’t spend decades in PPM only to be considered a useful admin function, so it’s clear that PMO customers believe there is certainly room for improvement in the services the PMO provides.
KPIs, said the verbatim comments that formed part of the feedback of this study, don’t link to business strategies. They don’t talk about shareholder value. Metrics relating to output don’t explain how to improve this output. Metrics don’t identify or clarify issues relating to performance. And finally, business management teams can’t relate to technical measures.
So, PMOs have plenty of metrics, but PMO customers don’t understand them or think they are useful. What should we be doing instead?
Creating good metrics
The main takeaway for me from this section of the presentation was that you shouldn’t copy someone else’s dashboard. What works for one business isn’t going to work for another, so templates aren’t actually that useful here. Every PMO is unique and provides a variety of bespoke functions to the rest of the organisation, so you should tailor the reporting and metrics in use to reflect that.
Metrics, Lars said, should reflect a strong customer focus to ensure they are well-understood and take a business view of project success. How is success viewed? he asked. Then shape management communications (i.e. reporting and metrics) along those lines. He offered some questions to ask yourself when putting together the relevant metrics for your PMO:
Is there organisational acceptance for standardisation around a single approach or method of reporting? This is particularly relevant if different business units have built their own measures and techniques over time or due to mergers and acquisitions. Different techniques mean different results.
Are the definitions of KPIs and metrics simple and useful for external comparisons? Even if you don’t need to compare performance of your PMO and projects with other companies right now you can be that someone will ask you to benchmark performance at some point in the future. Make your life easy and prepare for that now.
Implementing project metrics
Once you’ve defined what metrics are relevant for your PMO, you then have to go ahead and implement them. Lars recommended engaging with users to ensure they understand the process and how these metrics will help improve things. For example:
“Good dashboards show where the problems are,” he said. And that gives you the chance to show what you are doing to eliminate the problems.
Dashboards should track what is important. He also had a great recommendation for PMO leaders who have to produce short reports for their colleagues or sponsors: if you don’t have much time to present statistics (or many pages to present on) then cycle round the metrics you show each time. For example, average number of training days per project manager won’t change much from month to month so show this one month and then report on something else next month. Over time it will give you the chance to report on more metrics, and of course if someone wants to see the training days figures they can always ask for them.
In summary, Lars concluded that metrics should enable us to take action, otherwise what’s the point of them? He said that we should seek metrics that are simple, intuitive and focus on goals and that finally metrics don’t replace judgement. You’ll always have to apply your contextual knowledge to a dashboard to interpret the full situation.
I attended the Summit as a guest of Genius Project.
On your project you will have a number of measures and metrics. Key performance indicators (KPIs), critical success factors (CSFs) or success criteria and benefits can all be measured, and you may have some other targets to achieve as well such as customer satisfaction measures.
So what makes a good metric?
According to John Seddon in Freedom from Command and Control: A better way to make the work work, there are 3 principles for making sure your metrics are robust and useful.
1. Does it help understanding?
Seddon believes that the test of a good measure is that it helps us understand and improve performance. If it doesn’t, there isn’t much point in gathering the data. Targets, for example, are measures that don’t help us understand performance. They are arbitrary figures, created perhaps on a whim. Worse, they can focus people on the wrong things, and not on how best to do the work.
Capability measures look at the system of work (in a systems thinking environment), so they encourage people to look at how things are done and therefore they help improve the work. They can be more motivating than arbitrary targets.
In a project setting, by this definition a target held by the Project Management Office of closing 10 projects each quarter is not a good metric. A measure of how long it takes to complete the testing phase is useful, as that gives you data you can compare across a number of projects. Then you can draw some conclusions as to why some projects have a longer testing phase than others and in turn this will help with your estimating.
2. Does it relate to purpose?
The second test of a good measure, according to Seddon, is that the measure must relate to “the purpose of an organisation from the customers’ point of view”. In other words, measures must relate to something that means something to someone.
In a project environment, the person who is likely to be deciding what is important is your main project stakeholder or sponsor. Whatever matters to them is the thing you should be focusing on. For a project manager, this could be the triple constraint in the form of the time-cost-quality triangle. Or you could be told that the response times on the new website you are building are the most important thing and the sponsor wants reports every month about how much you have been able to improve them by. A lot of project measurements don’t actually relate back to what internal and external customers care about, so it is worth looking at what your project is measuring and checking that you aren’t wasting your time.
For more about what customers care about and how you can identify what is important to them, my book, Customer-Centric Project Management, includes a couple of case studies and some templates.
3. Does it integrate with the work?
The third test of a good measure is that it is easy enough to record. If it doesn’t integrate with the way that the work is done, no one will collect the data. That’s true enough – I’m sure you have occasionally been asked to produce reports and have struggled to find a way to represent the data or even to find it, as it isn’t ‘natural’ to be capturing project information in that way.
Seddon also believes that metrics that look backwards aren’t much help. Making decisions based on data that was captured in the past isn’t a good way to plan work going forward. I don’t agree – in a project environment capturing lessons learned, time measures and data that will help future estimates is, in my opinion, a good use of time. But I do see what he means in a service environment. His book is aimed at people in a call centre-type service organisation or manufacturing, where knowing that last week you answered 30 calls an hour or built 60 widgets doesn’t give you an accurate prediction of what will happen this week.
How many of your project measurements meet Seddon’s 3 criteria? And if they don’t adhere to these principles, are you happy about why not?
“There are two kinds of measures of use in managing a system: permanent and temporary,” writes John Seddon in Freedom from Command & Control: A better way to make the work work. “Permanent measures, those that are related to purpose, are the guiding measures for all performance-improvement work. Temporary measures are those that are useful in ascertaining the nature and size of a problem before action is taken.”
Seddon is an occupational psychologist who has spent his career trying to improve systems by taking the waste out of processes. One of the ways he has done this is by focusing on what gets measured, a kind of ‘What gets measured gets done’ approach to improvement. His work has no doubt resulted in a lot of projects, particularly those that relate to continuous process improvement, but his book doesn’t cover project environments particularly. Still, I think that the way he splits measures into these two types is relevant to project managers – and we are focusing on governance this month at ProjectManagement.com. Let’s take a closer look at the types of measures Seddon explores in his book.
Permanent measures, are, as you have probably guessed by the name, permanent. As nothing much is permanent on a project, these are the kind of measures you are likely to find in your Project Management Office or Portfolio Office. They are the measures that help manage the work overall. Some examples of permanent measures are:
Capacity and demand. This can help with resource scheduling on an ongoing basis. It helps to know what work is due to complete and when project managers will be free to take on other projects.
End-to-end time. This is how long it takes to complete the work from the customer’s perspective. As not many projects are repetitive, this is a bit of a pointless measure on projects. However, if you are repeating similar initiatives over a period of time (such as rolling out new software to an office at a time), it can be very useful to know how long it takes to complete one implementation.
Accuracy. In a project environment, this would be covered by project audits. The ‘accuracy’ of a project could equate to scope items covered, fit with requirements or how customer-centric you are and how much value you create for customers.
Temporary measures are those that don’t last long. They aren’t permanently built into the fabric of how companies report projects. As such, you could have temporary measures on your project, especially if your project has a RAG status of Red and is in trouble. If you are trying to rescue a failing project, you will want to gather some data on what has been done to date and how best you can recover the situation.
It’s less easy to see how Seddon’s examples of temporary measures relate to projects. Here they are.
Type and frequency of demand. This could be a PMO measure, especially if you are looking to recruit an additional project manager or more staff in other capacities. You could measure the type and frequency of demand over a short period of time to check that you really do have cause to hire someone. Equally, on your project, you could use this type of metric for resource management. For example, as you move into the testing phase your resource needs will change. This is something that you won’t measure when the project is over as staffing up to support the product is a task for the operational team.
Type and frequency of ‘dirt’ in input. Seddon is mainly writing about service organisations like call centres, so he means incomplete application forms, letters that don’t have enough information in and so on. In the world of projects, this could be things like project initiation forms that don’t have enough information in for the stakeholders to sign off the scope, incomplete business cases, requests for projects that appear in the form on an email with a good idea rather than a thought-through assessment for a new project etc. He recommends that a few small changes could make a big difference, such as changing the forms or template letters used. For example, you could amend the form on your company intranet so that it is rejected unless someone enters all the detail required to submit a new project for consideration.
Type and frequency of waste. ‘Waste’ to Seddon means steps in the process that don’t add any value. These could be things like handovers between teams, lost time, rework and errors. There are probably numerous places on a project or in your project management processes where things get lost or handoffs between teams are not as slick as they could be.
I’m a big believer in taking what we can from other areas of management practice and applying them to projects, so I think it is worth considering what we can learn from the type of metrics used in service organisations and the like. Do your project or PMO metrics fit any of these categories?