The Money Files

Last month I looked at a range of alternative metrics for assessing success. One of the comments on that article asked how, on reflection, had I seen these or other alternative metrics implemented effectively. It also asked whether there were challenges that might arise integrating alternative metrics into existing project management frameworks (which I’ll look at in a different article).

So, if you want to implement a more nuanced approach to measuring project success, how do you put this into practice? Below are some ideas. This topic is one that we could probably speak (or write) about at length, but hopefully the examples and ideas I share will give you some starting points for incorporating a range of measures into your project management practice.

1. Customer satisfaction

This is an area where I have quite a lot of examples to share. In my book, Customer-Centric Project Management, my co-author and I describe an extensive exercise to set up customer satisfaction tracking on projects, with internal customers.

I’ve written about customer satisfaction measurement on this blog before (start with this article: Lessons about project metrics). Basically, find out what is important to people, then track that regularly and plot your scores as part of regular reviews.

You don’t have to use an ‘official’ CSAT mechanism, or pay for electronic survey tools. Just ask people as a minimum.

2. Innovation level

I haven’t used an innovation score regularly throughout a project, but it has been included in the way we rate projects and prioritize which ones should be done as the requests come into the pipeline.

Implement an innovation score yourself on a simple Likert scale (1-5 or similar). Ideally, you don’t want the measure to be subjective, so set some criteria e.g.:

• Number of tools used where the business has less than 1 year of experience using them
• Years of experience of team implementing (as someone could have gained a lot of relevant experience in a different role)
• Contractor level of experience rated on a scale of how many implementations they have done.

Or similar – you really only need a couple of measures to make up your innovation score. Use the innovation score as part of the decision criteria for whether a project should be taken forward or not.

3. Resource utilization

Resource utilization reports are something I’ve only actively used on a couple of projects, because mostly we either haven’t had the tools to extract the data (because we aren’t doing timesheets or detailed estimates), or because staff have been 100% assigned to the project so we don’t have to worry about them splitting their time across projects. We would still have to worry about them being scheduled to do too many tasks in a week and being over booked, or under utilized, but when the team is full time somehow that’s easier to manage as you can see what they are doing and we speak every day.

Use the reports to check exactly that: is the team going to struggle to meet its commitments? And if it is, because staff are over-scheduled, what are you going to do about it?

Even the basic reports from Microsoft Project will give you utilization data, so take a look at what you already have within your tool suite and if you find it useful, use it.

4. Risk mitigation effectiveness

The final one I’m going to talk about today is tracking risk mitigation effectiveness. I mentioned in the article that you could use AI-powered insights to establish the effectiveness of the risk management activities undertaken, but in reality I imagine that takes quite a lot of effort to set up.

Another way to do this would be at project closure, or during lessons learned conversations, whenever these take place in your project lifecycle.

Look at what the risk was and what was done to mitigate it. Score the risk based on the effectiveness of the action:

• 1: Risk management activity was 100% successful and risk was closed
• 2: Risk management activity was successful but residual risk was worth £x
• 3: and so on.

Using residual risk (and specifically, a financial value of residual risk) is a way to establish how effective your management action was – or at least, it’s one way to create data that allows you to assess that.

Hopefully that gives you some pointers for measuring project success through different routes, with some more concrete examples of how to get started.

I’m working with organisations who are putting a lot of thought into benefit management at the moment – more thought than I’ve ever seen in past years. I know benefits management has always been something to consider on projects, but how many project/leaders/organisations actually truly do it?

I’m sure the larger projects are geared up for benefits tracking, but smaller initiatives? Those that might only deliver a few thousand pounds/dollars of benefit per year? Are those being tracked?

I think the trend is changing, and I think there is more focus on benefits tracking, particularly in programmes. Where there is a large programme of work, even small projects get to contribute.

So how do you do it?

The first step is to work out what benefits your project is going to deliver. For example, that could be:

• Cost saving
• Increased sales/revenue

In fact, benefits could be lots of things (shorter cycle times, more calls answered, higher customer satisfaction) but most often they can be reduced to money. Shorter cycle times should mean more projects delivered in a year, so more value achieved, or more widgets through the process, so more sales. More calls answered should convert to more sales, so more cash. Higher customer satisfaction results in more repeat purchases and more testimonials that drive new buyers, so more cash.

Even carbon savings and sustainability targets can often be represented as financial benefits. Carbon has a financial figure associated with it, so you can translate carbon reduction into money.

The trouble comes with trying to get stakeholders to agree on how these benefits will be calculated.

More calls in, for example, might not translate to more sales for several months, depending on your business timeframes and processes. You will need some rules and assumptions around how benefit numbers are going to be worked out where the maths is not straightforward.

For example: if we outsource a service, we can reduce headcount in function and probably make a saving on the cost to serve (budgets are part of the outsourcing decision, after all). That’s a straightforward cash saving: those individuals are no longer on our payroll (even if they were moved across to the outsourced party – the UK has TUPE rules for this).

But do we count that saving as a ‘pure’ saving, or do we have to take off the cost of buying the outsourced service? How long can we claim the service for before it becomes BAU? What happens when we need to add a head or two back in, where do those costs go?

We need assumptions to make benefit calculations work, like what’s the average cost of a sale (so if we answer more calls we should convert x% and make y% extra money as a result).

This first step of creating a model on which to base benefit calculations is the hardest part, in my opinion. There are experts to consult, finances teams to engage, data to gather so we have something to use as the baseline, and lots of maths and spreadsheets to test out so we can model what the benefits might look like.

Getting agreement on how the benefits will be worked out is hard. You need everyone to agree, and more importantly, to understand how it all goes together so they can repeatedly work out the benefit using the same calculation, month after month for as long as you decided to track it for.

That’s no small undertaking, especially for organisations starting out. Especially as many projects have multiple strands and multiple things contribute to the benefits.

What’s the journey like for benefits tracking in your organisation? Let us know in the comments what the biggest challenge is for you – I’m interested to see if you agree with me that it’s the setting up and creating the assumptions and rules for calculation, or whether there is something else that is the hard part.

The sustainability agenda is something all aspects of business are thinking about, and project management is no exception. We all need to be working towards creating a more sustainable future, and making the right choices on our projects is definitely one way to do that.

The first step is to make sure sustainability criteria are included in your documentation templates. You can add sustainable and carbon footprint criteria to your business case templates.

Here are some suggestions for how you can consider reducing the carbon footprint on your projects.

Create a benchmark

Many companies these days have teams dedicated to energy management and sustainability, so your first point of call should be to check in to see what initiatives are already underway. If your carbon efforts are going to be officially recognised, you need to use official methods of calculating the reduction, with full transparency and a robust, proven measurement process in place.

Who in the team will be responsible for tracking and measuring carbon savings, and do you need any additional processes in place to make it happen?

A full approach like that might be overkill – if there is no expectation of having to save carbon, you might not want to go down the route of proving that you are in a measurable, transparent way. However, you can still follow the rest of the steps to brainstorm carbon-saving ideas with the team and build in carbon-friendly ways of working.

Review project tasks

Take a look at the work you are going to be doing on this project. Which tasks are going to generate a carbon calculation that can be measured? What tasks are going to generate carbon and what can you do differently?

Identify carbon reduction activities

From the work you are doing and the tasks identified, take a look at what extra actions you could to reduce the carbon impact. For example, are you planning to print a lot of leaflets or documentation? What could be delivered via a QR code or provided in digital format? What would still need to be created in paper format for a selection of users who would not be able to access digital content?

What journeys are going to be undertaken on the project? Perhaps there are some workshops that can be held virtually and training delivered online instead of in a classroom. There are many advantages to getting together in person, but you might be able to identify some elements of the project that would work remotely.

You can also look at the supply chain. Can you ask whether suppliers can deliver in a more sustainable way, such as with electric vehicles, or in recyclable packaging. What would that do to the budget, if anything?

Add tasks to the plan

Add your carbon reduction actions to your project plan. You might need to take specific actions, you might need to be more mindful of the way in which existing tasks are delivered.

Make sure each action has an identified owner.

Track and monitor progress towards your goal

Include carbon data in your reporting. If you are tracking according to your organisation’s metrics, then you might have some help from specialist teams who will know how best to report the data.

Otherwise, add some narrative to your reporting to allow you to share your success with others.

A lot of being smart with carbon on your project is just asking yourself the question: what can we do and are we doing things in the best possible way? Being aware of the challenges and helping others see that sustainability is a focus for you is key to keeping it on the agenda and making it part of the accepted ways of working.

We’ve all got metrics we use to assess project success: cycle time, earned value and so on.

As the year ends, maybe it’s time to look at some other measures we could use that might be a bit more… dare I say… interesting? Below, I’ve suggested 7 alternative metrics you could put in place (some easily, some would take more thought and set up) to look at what project performance really means in the round.

1. Customer satisfaction (CSAT)

You might find CSAT in use across other teams. Why not implement it for project management customers too? Even if you work in-house, you will have internal customers. Trust me, they have an opinion on the project management service you provide. Why not check in with them directly and ask for it?

You don’t need a formal CSAT tool. Set some survey questions and set up a form to ask stakeholders their views, and then collate the results.

2. Time to Value (TTV)

Cycle time is worth knowing, but does the end of your cycle always end in value delivered? A different way of thinking about it would be time to value: how quickly the project delivers tangible value per feature, or perhaps overall.

This metric comes with the added challenge of having to define value: but that could be a very useful exercise for stakeholders!

3. Innovation level

Could you create your own innovation index? There are already indices in use like the Global Innovation Index, but that’s probably overkill for our projects. Consider how innovative the product/deliverables are and the method used to implement them.

4. Resource utilization

Here’s one you can probably get from your project management software but I don’t see it on reports very often. What could you take from a utilization report? Metrics are only helpful if there is something you can use them for, like decision support. In this case, it would be making sure the team is adequately resourced, so you really want to be looking forward not backward. Although historical data is useful too to see if there is a trend towards over or under staffing.

5. Change adoption rate

Could you create a metric that looks at how quickly the organization is adopting new changes? If you work with a change manager, they might have some ideas about how to implement this. Any new process changes or anything that requires training could be included, even if your measure was only based on smiley faces!

6. Sustainability impact

Your procurement team might already have a sustainability index based on their work with vendors and a sustainable supply chain. If you have an energy team, they might have measures you can pull into your projects too. For example, how much carbon saving your project is creating, or how much waste is recycled from different locations.

7. Risk mitigation effectiveness

We mitigate risks, but are those actions really useful? We could draw on AI-powered insights by plugging in risk mitigation activities across a selection of risks and the outcomes. (Or you could work through this manually). I’m not sure how you’d assess the usefulness of the mitigation strategy: maybe on a scale of 1-5? Then you could see which actions had the biggest impact in reducing the risk.

There are lots of ways to measure project performance, and no one wants to be creating reports and tracking metrics for the sake of it. However, it might be worth looking at whether your current suite of metrics truly give you the complete, holistic picture of performance, because we all know it goes beyond time, cost and quality.

I was inspired to do the Generative AI for Project Managers course (which is currently free, and definitely worth looking at on the PMI website) and from there, to sign up for a ChatGPT account. In parallel to that, I was asked to deliver a webinar on AI-proofing your career (spoiler alert: AI takes away the parts of the job that I didn’t like so much anyway, and frees up more time for you to shine in human relationships), so I seem to have embedded myself in the world of AI recently!

I thought I would put ChatGPT to the test by asking it a question I get asked frequently:

What are the top 5 challenges affecting project managers in the world today?

This is an edited version of what it said. It gave me 246 words, and I’ve added my own thoughts after each point to add a human-in-loop spin on the feedback. The GenAI course explains human-in-loop if you have not come across this term before.

(Sidenote: I feel weird calling it ‘it’ when you can have a conversation with it. I think I would like to give my ChatGPT bot a name, I just don’t know what!)

As of my last knowledge update in September 2021, the following were some of the top challenges affecting project managers worldwide:

1. Resource Constraints:
   • Limited availability of skilled resources, both in terms of human capital and specialized tools or technologies, can significantly impact project timelines and deliverables.

Elizabeth says: So far so good. This seems to be a perennial problem across the ages. It’s not particularly new, but I’m sure it is true. Every project manager I speak to seems to have issues with resourcing and capacity planning.

2. Scope Creep and Change Management:
   • Managing changes to project scope and requirements can be a major challenge.

Elizabeth says: Again, this is not new news. It’s not the modern, fresh take I was hoping for. Perhaps there isn’t one: we’re stuck with the challenge we’ve always had, which is reassuring and disappointing at the same time!

3. Remote and Distributed Teams:
   • The shift to remote work, accelerated by the COVID-19 pandemic, has posed challenges in team collaboration, communication, and ensuring productivity across geographically dispersed team members.

Elizabeth says: I’m sensing a theme here. These are all true, but as project managers, we’ve been overcoming the challenge of remote and distributed teams for years, even before the pandemic.

4. Risk Management and Uncertainty:
   • Identifying, assessing, and mitigating risks is critical to project success.

Elizabeth says: I agree that risk management and working in uncertainty is a challenge, but we have whole project approaches designed to minimise the impact of that and maximise user engagement through short deliverable sprints. Yes, the world is an uncertain place, but again, project managers have been looking out for the things that will bite us in the backside for years. That’s the job!

5. Stakeholder Management:
   • Balancing the needs and expectations of various stakeholders, including clients, sponsors, team members, and end-users, is crucial.

Elizabeth says: I would not have put stakeholder management as one of the challenges for project managers. In fact, it’s one of the fun parts! I do understand that it is challenging to manage competing priorities, but again, our processes are set up to identify those and work through them.

Then ChatGPT goes on to add:

Please note that the landscape of project management may have evolved since my last knowledge update. Factors like emerging technologies, evolving business models, and global events (such as the ongoing COVID-19 pandemic) can introduce new challenges or shift the emphasis on existing ones. It's recommended to stay updated with current industry reports, trends, and insights to understand the most pressing challenges faced by project managers in 2023.

Elizabeth says: I agree that it is important to stay up to date with the pressing challenges, and the best way to do that is to stay connected to the industry. Conferences like the PMI Summit and publications like project management magazines provide timely insights into what is important. For example, sustainability isn’t on the ChatGPT list and I would have thought that would have been a concern back in 2021.

I haven’t yet paid for an upgraded account, so perhaps a paid account would have better outcomes and give me a different take on what the current challenges are, so there’s that to consider.

Meanwhile, I’m curious! Do you agree with these challenges? Do you think there are others more pressing for us as project managers? And what do you think of the output of ChatGPT if you’ve used it? Let me know in the comments below!