Can you prove that your PMO has improved project delivery?

 

Some project management offices (PMO) are like Rodney Dangerfield – they don’t get no respect. While there are many causes for a PMO to be shut down, the inability to demonstrate their value proposition is one of the more common reasons.

So how can a PMO prove that there has been an improvement in project delivery?

To answer this question, we need to identify one or more metrics which will be used to represent project delivery capability. A commonly used metric these days is time to market which could be calculated as the duration from the start of project investment to the first delivery of customer-facing value.

You might think that it would be a simple matter of calculating the average time to market based on a sample of pre-PMO and post-PMO projects, but this is not statistically defensible. The sample size used to determine the average values might not be sufficient to prove that the difference is statistically significant. If variation has remained the same or has increased, even if the average time to market has dropped, portfolio-level outcomes won’t have improved.

Time to call your friendly neighbourhood statistician!

It might not make sense lumping all projects together for these calculations. For example, one might reasonably expect that a $10,000 project will usually take less time to deliver value than a $1,000,000 project. Project size and complexity influence timelines, so you might wish to stratify your population into a few distinct project tiers.

The next step is to determine the minimum sample size to prove that a difference is statistically significant. Statistical analysis packages such as Minitab enable you to calculate sample size based on the statistical test you will be running, the difference you’d like to see, and an estimate of the standard deviation of the population. For example, let’s say that we’d like to prove a reduction of one month in time to market, and the estimate of the population standard deviation is also one month. Minitab will calculate a minimum sample size of 18 projects. Unless we have at least 18 projects in both the before and after samples for each project tier, the difference in averages can’t be stated to be statistically significant.

Assuming you have sufficient data to support statistical testing, the two statistical tests you can run for each project tier are a 2-sample t test and a 2 variances test. The first will help you decide if the difference between the averages for the before and after samples is statistically significant or not, and the second determines if there is a statistically significant difference in the variation of the two samples. Ideally, after running the tests you will see a reduction in both the average time to market and the variation in the post-PMO sample. This won’t prove causality – there could have been other factors which more directly caused the improvement, but barring any obvious alternative influencers, you can state with confidence that things have improved since your PMO was established.

A key assumption underlying these tests is that before and after sample time to market data is normally or close to normally distributed – this can be confirmed using an Anderson Darling normality test. If it turns out that the sample is significantly non-normal, other tests would need to be used to statistically prove an improvement.

Disraeli was accurate when he said “There are three types of lies — lies, damn lies, and statistics”, but used appropriately, statistical testing can support the case for a PMO’s continued existence.

(Note: this article was originally written and published by me in December 2015 on my personal blog, kbondale.wordpress.com)