Benchmarking, Investment Analysis and Front-End Planning

In the first of my series on Investment Analysis, I open with an introduction to the subject based on the material I presented at a a series of three seminars for the UK Chapter PMI during 2018 and 2019.

Imagine you are leading a project and your instinct tells you there is a flaw in the business case; is there something missing in the investment analysis.  What do you do?

Back in the mid 1990’s , I was part of a project negotiating team looking to do a deal between my company and a National Oil Company.  

Involved were representatives of the National Oil Company – local sheikh’s ; their advisors and from my company senior business and project executives.  We were in a grand house in London.  This was the house that senior executives would use for big meetings – they even had a butler and chef in the kitchen.

I was relatively inexperienced and responsible for the project analysis and business case. One of the first things I learnt about investment analysis which is fundamental is the importance of the assumptions that are being made. My analysis included data and assumptions on all the key variables and showed that one of the biggest uncertainties was the logistic cost of moving the finished product.  Would there be a fixed tariff – $/tonne or would further investment $m be required in port facilities for distributing the product. 

I could see that a high freight cost or significant additional capital cost would make the project much less economic.  I pointed this out to my senior company executive and he asked me to present the economic case and draw attention to the key assumptions including the logistics uncertainties.

I remember being in the oak panelled board room sitting at a large oval table, with the smell of fine wood panelling .  Round the able there were the sheiks, their expert advisors and the company executives.  The discussions were intense and it came to the shared economic model that I had prepared.  I felt nervous because I had big news to present – and maybe a deal breaker.

Our chief business executive, a gnarly, grizzly sort of character turned to me and said, ‘Tim, please present the project economics’.  

Everyone went quiet – I outlined the economics – then I highlighted the impact of the uncertainty on the logistics assumptions – the consequence on the project value of a significant $/tonne charge and/or a multi $m investment cost for jetties and loading facilities.  I sensed the interest and tension in the room – I paused and looked at my boss and he said very firmly ‘carry on Tim’  – the stage was truly mine and I was away – the adrenaline flowing.   I was then chairing a discussion of the project value and the key drivers –  there was a lot of heated debate about the facts – AND the logistic costs were accepted as a key variable –  to manage the uncertainty there was an undertaking from the National Oil Company that these would not impact the project economics – they would cover these arrangements.  

 And later that day the deal was done,,,,,

The learning for me was that having a shared understanding and clarity on the project assumptions – the facts – helped drive a good deal.  For me all projects need that clarity up front and the PM would do well to have that understanding as well.  

Having been given the stage by a senior executive my career never looked back – It was an opportunity to lead that I grasped and become a company subject matter expert in the matter of investment analysis – and here I am today to share that expertise with you.

I have discussed the subject of Investment Analysis with many Project Managers and there is a view that the project economics are not normally the responsibility of the Project Manager; but I would argue that it would be helpful for PM’s to understand the drivers of value that underpin a project.

Over the next few weeks of this blog my aim is to give you a greater awareness of the key factors affecting the business case and give you greater confidence in questioning and challenging the decisions that have the biggest impact on the business case both before the project is sanctioned and when the project is in flight.

What is a Green Belt, a qualification in Six Sigma, Continuous Improvement (CI) - someone told me it is really a white belt and you have had a roll in the grass.

Where does benchmarking fit in, well it does in three places;  firstly in the definition of a problem - a reference for improvement; secondly in identifying solution for a problem and thirdly in tracking improvements by comparing and sharing.

My story of my green belt project as a Commercial Manager in a Corporate IT function - responsible for overall budgeting and forecasting, quarterly and annual.  The problem was that forecasts were inaccurate and normally over -  the errors in quarterly forecast ranged from 30-70% out, normally underspend and the range of full year forecast up to 15% out, also normally underspend.  The impact was lost opportunity to reassign funds and significant management and direct report time in investigating and explaining variances.

I reviewed the process and identified and number of improvements - recognising one root cause was budgeting and forecasting behaviour by leaders and another the quality and consistency of the financial data.

The agreed approach was one of ‘No surprises’ with portfolio forecasting accuracy demanded at +/- 3%. This required 

1. Leadership engagement  and ownership ; transparent financials, deliverables and status.

2, Portfolio judgement ; Project contingencies and assumptions around scheduling and business engagement made transparent at the portfolio level

3. Feedback on forecasting performance ; understanding root causes of defects.

4. Single point of truth and source for data ; with consistency and thoroughness provided by the person involved.

The improvements required more leadership attention to forecasting activity and resulting spend and honesty in bidding for budget against requirements.  This was coupled with accurate tracking and reporting against a control chart.

By benchmarking across the teams with control charts over time it became apparent some groups were better than others.  There were two leaders who were at the centre of this result; both very effective and charismatic IT directors.

The story is of a leadership meeting and the moment that the different behaviours came to the fore.  One IT directors approach was to micromanage the costs and he achieved almost too good result well within the 3% target,  The other IT directors approach was to play the game but still budget with a reserve up his sleeve which came home to roost towards the end of the forecasting year.  These were extremes in the response and I remember the moment these came face to face in the leadership meeting,   After a moment of tension, in an environment of trust - there was realisation and it was agreed that the next round of forecasts would be tempered to allow some flexibility but with clarity and with a significant challenge to one IT director to really change his budgeting approach.

Benchmarking in the context of a Continuous Improvement (CI) green belt /six sigma project helped create the solution and also supported the sharing, learning and the environment of trust as part of cementing the improvements.

I was challenged by the Data Analytics community to give a presentation on the Next Generation of Benchmarking I have been making enquiries into the matter. This note summarises the nextgen possibilities that build on my Subject Matter Expert presentation and series of blogs on Benchmarking - one of my passions, particularly in the matter of projects.

There are three areas that data analytics can take benchmarking to the next level.

1. Deeper and deeper insights
2. Very early intervention
3. Instantaneous advice

In my example of benchmarking of project practices - data analytics can provide deeper and deeper insights.

In project creation and delivery -‘doing the right project and doing the project right’  in the matter of real time advice there is clearly scope, based on the deeper and deeper insights in all areas of benchmarking  With each of my examples, I believe there is the possibility of deeper and deeper insights and each of the following three examples of benchmarking the I delve into; 

- Front end planning performance - leading to data to enhance the PDRI index and workshops 

- Elements of Team alignment -  leading to data on team functionality 

- Sources of complexity - leading to data on risks.

Insights like these could support very early intervention and or instantaneous advice.

The challenge of Next Gen Benchmarking has been highlighted by the IPA (Independent Projects Analysis) in a paper they have published .  There are three barriers

1. Data competency - half of the organizations interviewed had no data analysis people at all.
2. Data processes - many companies have a process, but it requires the project team to gather and record data manually at the end of the project
3. Data infrastructure - few companies have automated systems for transfer of data between systems - accounting, scheduling and cost management. 

A another useful reference the British Standards Institute (BSI) have published a paper on Digital Transformation.  It majors on driving the necessary change through Data Standards  In the document they quote that ’47% of construction managers are still using manual systems for data collection’.

When it comes to the yin and yan of benchmarking - where trust is required to work across organisational boundaries - internally and externally; the opportunity to leverage data analytics the challenge to deal with is amplified.  

Trust covers three areas

1. To pool data

2. In the quality of the data

3. in the insights that are provided

I am sure I have just scratched the surface but am convinced that there is a powerful Next Generation for Benchmarking and call on you all to join in and share your examples and experience.  I am putting together a detailed review of the subject and I will prepare a series of blog to publish during the summer.

For my third and final professional example, I am using the UK Governments Infrastructure and Projects Authority guidance document Benchmarking – Best Practices. I was privileged to be asked to provide a ‘critical friend’ review of the document before it was issued in March of 2019.  The document describes Benchmarking as the process of improving cost and performance against similar projects.

Benefits of Benchmarking by Project Stage.  The IPA document states that benchmarking is useful at three key project stages.

1. During policy creation to select the right project
2. During detailed delivery conversations ; providing challenge and building confidence
3. At the end of the project to assess performance, share learning and build continuous improvement into the system.

There are five C’s in this summary; policy creation, conversations; challenge; confidence and continuous improvement which cover the range of benefits that benchmarking provides

Infrastructure Benchmarking Process – The document provides guidance on the detailed process for the effective use of benchmarking; particularly in the matter for major infrastructure projects – rail and road – although I would argue that it could be applied to other types of project as well.  At the core of the document there are seven steps to using effective benchmarking. To help explain the process I have developed an an acronym OCTAVIA – it is a Latin name  – and means eighth-born baby.

Step 1: Confirm the project Objectives and set the metrics – get clarity on the basis for benchmarking

Step 2: Break the project into major Components for benchmarking –  choose the benchmarks wisely

Step 3: Develop Templates for data gathering – create a common language and a forum of expertise

Step 4: Scope sources and gather data – a critical success factor is the Acquisition of data

Step 5: Validate and re-base the data – a further critical success factor is the quality of data

Step 6: Produce and test the benchmark figure – use Images and visual presentation

Step 7: Review and repeat, if necessary, before using data for benchmarking – Assess impact

In support of the process the IPA have created a Benchmarking Team to support the Transforming Infrastructure Performance (TIP) programme.  The Benchmarking guide is also aligned to the TIES – Transport Infrastructure Efficiency Strategy – a Department for Transport (DfT) sponsored activity involving Transport for London (Tfl), Traffic England, High Speed 2 (HS2), Network Rail and Crossrail. Members of TIES have already worked together to produce a benchmarking forum for tunnelling which an excellent example of the benchmarking guidance in action.

Tunnel example – cost – The worked example produced by the benchmarking team at the IPA working with a forum of tunnellers representing a number of organisations; Crossrail; High Speed 2; Heathrow; Network Rail, Transport for London, Highways England; Thames Tideway Tunnel; National Grid and Scottish Water,  The study was based on Tunnelling data across a range of infrastructure types.  Of the 25 major tunnelling projects identified which have been recently completed in the UK, valid data points were collected for 16 of them.

The chart used has cost per unit face area against tunnel length – this is a standardised metric, based on face area where from my earlier pipeline example the standardised metric was based on diameter.

On analysis by the tunnelling benchmarking forum it was apparent that the data separated distinctly into transport tunnels and utility tunnels. The former were larger, in the range of 5.5m to 11.5m diameter but most fell between 6.8m and 8.1m, while utility tunnels were in the range of 2.m to 8.8m but most fell between 3.4m and 5.2m.

The principal drivers of cost across both groups are, perhaps unsurprisingly, length and diameter with some useful correlation of data points to form useful trend lines on the chart.  This gives the data for either a benchmark against the trend or can help identity specific projects to benchmark against.

Tunnel example – schedule  The second example is for schedule and the assessment of tunnel performance as a production rate – metre/week. Once again this gives either a benchmark against the trend or can help identify specific projects to benchmark against.

As a final point there are two lenses available for benchmarking here – what I would describe as the yin and yan of benchmarking - internal and external.

The point I want to draw out is the blending of Internal and External Benchmarking - these ideally interact for maximum benefit.  Internal benchmarking is within an organisation such as across government or a major company;  external benchmarking is with organisations outside your own.  External benchmarking is normally harder to do as you need three things;  legal considerations around Non-Disclosure Agreements (NDA) ; and Intellectual Property IP – and trust in the sharing of information in order to get maximum benefit.

However, my experience of a major oil and gas company, means that there can also be internal silos to deal with – we had upstream, downstream, petrochemicals and alternative energy with internal boundaries to deal with – primarily the issue of trust and different approaches – so benchmarking certainly facilitated sharing and learning.  This is a similar situation across government but with a shared benchmarking mindset it should be possible to overcome differences across government agencies – Tfl, Traffic England, HS2, Network Rail, Crossrail all involved in the TIES effort.

To conclude – I’ve explained benchmarking, used one personal example – park running and three professional examples – pipelines, best practices and the IPA document top show illuminate benchmarking is the use of data to facilitate continuous improvement, to support target setting and creating a learning culture.

I believe passionately that using a  benchmark gives an invaluable external perspective ; can remove ‘tunnel vision’ and prevent  ‘groupthink’ .  as a measure a benchmark provides a discussion point, a reasonable target and a fair comparison for a project.

My call for action is to use data and benchmark to help achieving professional and personal goals.

For my second professional example of benchmarking in projects I am drawing on the context of major oil and gas pipelines.  Specifically pipelines from Azerbaijan to Europe.  I was involved in facilitating a series of front end planning workshops; advising on project best practices and investment analysis in support of the Finance Memorandum being presented for the Financial Investment Decision (FID).

My personal story of facilitating the front end planning workshops was a great example of how the benchmarking process can make a huge difference to the project outcomes.  I led the workshops in the main project team offices in London.  The key representatives of the team (Project Manager, Project Services Manager, Lead Engineers, Operations Representative, Environmental and Safety Leads) were all present in a meeting room overlooking Paddington Station. The atmosphere was tense but positive – we were assessing readiness and looking for gaps. My role was to listen and lead – particularly to listen for the quietest voices – and there were several moments when I probed for more information and we identified some critical items that needed more attention to reduce the risk of delays.  I’m happy to say the project moved smoothly into detailed design and is in effective operation today.

This front end planning result became part of a benchmarking supplement; supporting the Financial case used to seek approval for the Investment in the pipeline project.  To illustrate this I have examples of three benchmarks that were presented.

Front End Planning  – The first benchmark was the Project Definition Rating Index (PDRI) and the results of the front end planning workshops.  The benchmark showed the PDRI for the portfolio of oil and gas pipelines from Azerbaijan through Georgia to Turkey and the progression through the planning phase to Final Investment Decision to an acceptable level of definition.  The comment included in the finance submission stated: “The completed FEL Self Assessment using CII Project Definition Rating Index (PDRI) tool. The overall score of 240/1000 (Lower score is better) indicates the project is well positioned for entering Execute Stage. This score is within the range for projects approaching final sanction”

Schedule –  The second benchmark for the pipelines (metres/spread day) had the project seeking sanction compared to company references. These comparative benchmarks are based of reference projects, assessed for their comparability by the project team and validated by the PMO/Central team.  The comment included in the finance submission stated: “The pipeline has a low Metres/Day lay rate due to the terrain, brownfield working and numerous work locations. The line to be replaced is not continuous, requiring the move around of the total spread numerous times.“ This explained why the project had a slower projected rate than the comparative benchmarks

Cost –  The third benchmark was cost per inch-km; an industry standard metric for benchmarking of pipelines.  The  project seeking sanction was benchmarked against company references. The comment included in the finance submission stated:  “The pipeline cost is higher than the benchmarks; due to more complex work scope, brownfield working, interrupted work sequence and terrain challenges.“This explained why the project has a more expensive projected rate than the comparative benchmarks.

These three benchmarks were core to the benchmarking supplement in support of the Investment case and I believe, represent an example of best practice in using benchmarking.

I am sure you will have your own professional examples of project benchmarks, please share any that you think would be of interest to this community.  For my next blog on the subject of benchmarking  I will share my third professional example of benchmarking at work – in the context of major infrastructure projects.