The Money Files

Metrics are an important way to learn about how the project is going and to reflect on what has happened in the past so you can do things different in the future. Or repeat the things that work well.

I learned a few lessons about project metrics when I worked on an ERP implementation a while ago now. We measured internal customer satisfaction from the angle of the stakeholders’ experience of being on the project. We used standard questions and asked them to rank our performance on a scale of 1 to 10. (I write about all of this in Customer-Centric Project Management, which I co-wrote with a colleague.)

In my experience, workstream leads scored reasonably based on the context: no one ‘played politics’ to get what they wanted. But there was always room for improvement based on our scores. We had plenty of armchair debates in the lobbies of hotels while working on the road, talking over the scores and why they were ranking project performance the way they did. They weren’t my favourite conversations, but they were extremely useful in building great stakeholder relationships and goodwill over time.

The big lesson for me came when I was asking my own colleagues in the IT department to rank the project and they scored it badly. I took it personally as the project lead as you can imagine! But it was a huge wake up call for not taking my colleagues and friends for granted: I was pouring all my stakeholder engagement effort into people outside of my own team.

Luckily it was easy to fix. I set up conference calls for team Q&A and made time for regular communications. If you listen to what people want and give it to them, you can make a quick difference to perceptions and how easy it is for them to do their jobs.

The takeaways for me, specifically around metrics were these.

Identify stakeholders in the process

Put some time into identifying stakeholders and don’t miss the obvious ones like I did!

Ensure the measures are representative of all stakeholders

If your measures are not objective and are not representative of all stakeholder, consider having different versions of the measures for different things. That’s OK as long as there is some longevity baked in for comparison purposes.

Decide on how to record results

In my case, it was better to keep individual stakeholder results separate instead of creating an aggregate of stakeholder satisfaction scores. That gave us greater insights into how each workstream was feeling. An average would be unrepresentative of the community overall.

Sense check

Are the metrics telling you what your instincts are telling you? If not, why?

As project leaders, it’s important that we set up metrics to measure what matters (I’m sure you’ve heard that before). We need to know who matters and their experience influences the overall metrics on something like satisfaction or the interpretation of project value.

Metrics are only useful if they include or are representative of all stakeholders, and all interested parties, even if you then split those groups out further.

You probably measure a range of things on your project. I’ve seen PMOs track number of open risks, projects closed this month and other numbers that are pretty much meaningless out of context. Here are 5 ways to make your metrics work better and give you more useful information, inspired by the Measuring What Matters report from PMI.

1. Measure more often

A study by PMI and PwC found that only 41% of PMOs were consistently measuring and reviewing performance. If you don’t measure regularly, how can you monitor trends?

They also found that around half of PMOs spent time communicating to the C-suite about milestones and project impacts. I know that in some organisations the PMO doesn’t have a direct line to the execs (although I think they should). Improving the perception of the PMO relies on the right people having the right information.

Action: Make sure you have a regular schedule for measurement so you can capture and track trends.

2. Collaborate up the organisation

Another thing highlighted by the PMO study was that metrics are often set by people who are not the PMO and who are not C-suite execs. Whether you are a project sponsor, senior leader or project manager, make sure you involve the right people in the conversation about what should be measured.

More collaboration between the PMO and delivery teams and the executives responsible for setting strategy should mean that you are measuring stuff that demonstrates whether the organisation is getting closer to the strategic goals.

Action: Check your measures are in line with the strategic vision for the company and that the right people were involved in coming up with them.

3. Focus on outcome-based measures

Outcome-based measures are those that reflect what was achieved on the project in terms of deliverables and change. They are different from the project management measures of time, cost and scope. (Note: those are still useful, but they aren’t the only thing you should be tracking.)

Projects exist to deliver change. That’s what is important: the impact that the work has on the organisation.

Action: Review the measures you are using to track your project and see which of them are outcome-based. Is that enough?

4. Review measures regularly – and with the right people

Once your measures are set up, keep them under review. Make sure to get input from a wide range of stakeholders: those who are collecting data for the measure, those who are using the measure for decision-making, the PMO and the project team, along with anyone else who has a stake in the outcome.

Reflect on whether the numbers or results are telling you what you thought they would. Do they provide accurate, reliable data that can be applied to different projects in a meaningful way? If not, why? Perhaps the measures need tweaking to make sure they properly serve their purpose.

Action: Schedule time to reflect on the usefulness of the measures you have in place.

5. Define success

Finally, define what success looks like. What parameters for your metrics represent a ‘good’ score? Below what level would you need management intervention? What are the red flags that would push the project into an Amber or Red status?

Capturing data is one thing – but being able to use it is something else entirely. In addition, some of the more ‘fluid’ desirable outcomes like cultural change or customer satisfaction are harder to grasp with a single number. You might need to combine several metrics to get a full picture of the current performance levels.

Action: Review the parameters that trigger action for your measures.

PMI and PwC did a survey on metrics and measurement on projects. You can read it online: Measuring What Matters.

I had a conversation about measurements recently with a PMO leader who wanted to understand how best to talk about what her team did. So are there any insights in the report that will help you if you are in a similar situation?

I think there are. Here are four of the key things that are worth looking at.

1. Measure more things

The survey points out that the top 10% of PMOs measure more than the average PMO. Quite a lot more: they tend to have 10 measures on average compared to only 7 in lower performing environments. Of those 7, generally the researchers found that 4 were linked to the classic project management iron triangle of time, cost, quality, scope.

Those metrics don’t really tell you anything about how the project landed in the organisation or whether the right project was done at all. Success in the eyes of strategic integration and execution is not often to do with whether you hit a budget baseline – especially if that budget changed during the project and was rebaselined to your new forecast anyway.

2. Make measures more effective

The research points out that there are two considerations for effective measures:

• Making sure you are using the right measures
• Implementing them appropriately.

The right measures for one project might not work on another project. Again, we see the fact the project management needs to be tailored in order to get the best results, and the PMO reporting is no different.

Some measures are worth capturing regardless: were you on time, did you deliver what was expected? But outcome-related measures like customer satisfaction, risk management and operational impact probably need to be tweaked for each project.

If you can’t tweak the measure itself, at least let project managers choose from a list of metrics so they can select how to report on the project in the most appropriate way.

3. Involve the right stakeholders

The PMI research shows that only 63% of organisations with a PMO use the PMO team as part of the development for metrics. Which begs the question: who is setting project measurements if it isn’t the PMO?

It seems to be middle management, as only 39% involve the C-suite execs. I think we can conclude that project sponsors or senior leaders decide what success will look like and what should be managed without linking into strategy or what’s actually practical and possible to measure as good practice (via the PMO).

When you adopt metrics on your projects, question where they came from. Are they part of an organisational standard created by the PMO with appropriate stakeholder input or did your boss just tell you to do it?

4. Use more tech-enabled solutions

The final takeaway from the PMI study into PMO metrics is about increasing the use of tech to measure effectively. Many online tools help you measure the impact. Whether you have advanced strategy execution tools or a simple online survey, tech opens doors to be able to repeatably and reliably collect data to help us assess success.

Many PPM tools collate data by virtue of the fact that we use them everyday for project management and scheduling. You can extract data about timelines, resourcing and budgets that helps inform future projects. However, it seems like budget is a barrier to investing in the tech that will help us do our jobs better. If you’re still stuck on spreadsheets, keep lobbying the powers that be for a better solution!

The Standard for Program Management (Fourth Edition) talks about how to track program financial metrics once your financial management plan is up and running. I thought it would be worth comparing the guidance to what I’ve done as a program manager to see how I measure up – and you can compare your own practice to what’s in the Standard too.

Program financial management, as a refresher, is defined in the Standard as:

Activities related to identifying the program’s financial sources and resources, integrating the budgets of the program components, developing the overall budget for the program, and controlling costs during the program.

Once you’ve got the program going, your work as a program manager shifts to tracking the money and making sure you are on track.

Spoiler alert: I’ve never used earned value to do this in real life, although I’m well aware of the benefits of doing so on projects and programs.

I think the techniques you use for tracking very much rely on your organisational culture and maturity levels, and I’ve not worked anywhere where EV is considered part of the way things were done. If you’ve got experience working in an EV environment, let me know how that goes in the comments.

The program manager’s role shifts to monitoring spending and controlling spending, ensuring what is being paid out is in line with the budget. In my experience, as a program manager, I’ve had a fair amount of latitude to move money between ‘pots’ (or projects) to ensure the overall goals of the program are met. And I have to say, I’ve enjoyed being able to make those decisions.

What I haven’t enjoyed is the financial scrutiny. I know we need governance on programs, and I’m all for it, but sitting in a meeting having to present the numbers has always been uncomfortable for me. Not because I don’t believe in the numbers, but because I’m normally presenting to people with an accounting background and honestly they could dance rings around me if they wanted to pick holes in my maths. So I have to put extra effort into making sure I can justify how numbers are put together.

My top tip is to make sure you keep detailed records of how you came to land on certain figures. For example, on a program I’m working on at the moment, we track committed spend, forecast and then actual “out-the-door” spend. But there are a couple of other strands within the program that are accounted for separately (don’t ask, it’s just the way it works best) so I have to make sure I’m clear as to what’s in and what’s out of the numbers so I can justify them every month and make sure we are reporting to the PMO on a consistent basis.

Because trust me, if I didn’t, I’d forget from month to month what the basis of calculation was and report something that wasn’t internally consistent and that I couldn’t justify reliably. Which would be bad.

Governance serves a purpose: it makes sure that a program is operating within approved cost limits and challenges programs that are forecasted to go out of those budget targets. Then the organisation can decide if it wants to continue with the program or not.

I’ve luckily never worked on a program that has been cancelled because of financial issues – but I imagine that is largely luck and the kind of programs I have been involved with rather than any skilful cost management on my part.

My experience of program cost management has been very similar to managing large project budgets: the skills are the same, and business acumen comes into play too. I think that having the bigger picture and goals in mind helps. What do you think?

In my next article, I’ll look at some typical financial management activities as outlined in the Standard and talk a bit more about those.

Someone emailed me the other day asking about how to use percent complete to track progress on their project schedule. It’s not the worst way to measure performance, but as I’ve got more experienced at putting schedules together, and the work I do is more uncertain, I’ve got less interested in using percent complete.

It means very little (at least, the way we were using it – which was basically a guess to feed into a schedule that was also mainly guessing given the level of complexity and uncertainty, and changes every week).

So I started thinking about schedule performance tracking – and there are plenty more ways to measure your progress than sticking to percent complete.

The infographic below shares some of the ways I know to measure your performance. You wouldn’t want to use them all on the same project necessarily, but it’s good to have options. Which ones do you use?

There’s a video here about schedule performance tracking measures if you would like some more information.