When?

 

Image adapted from: https://www.pmi.org/learning/library/delivering-project-benefits-model-overview-6904

I have been impressed with PMI’s shift to value delivery as what we as project leaders should be all about.

The recent Maximizing Project Success report, the speeches and messaging from President and CEO Pierre LeManh, and excellent videos like the one from Dave Garrett and Tricia Cabrey, “Project Success Defined and Measured” (click here) are all exactly on point.

But I still struggle with one question: WHEN do we measure value?

If you measure value while the project is being executed, you are leaving out something really important: the project’s outcome in service.  In use.  In operation. Being used by customers.  In the steady state.  All of those terms evoke the idea of the passage of a significant amount of time – a timeframe which by definition only takes place AFTER the project has been ‘completed’ and handed over to users.

A striking image popped in to my mind's eye (see below).  This shows Flint, Michigan government officials celebrating the cost-saving switchover of Flint's water supply, which ostensibly finished well with execution-based measures and even with initial outcome measures.

However, it wasn’t long before residents began becoming ill, and and not long after, angry residents (justifiably!!!),  demanded answers about how the project could yield such poor quality water – providing the opposite of value – contaminated water, illness, and death.

So, with these images in mind, I find that although the work PMI has done here is absolutely laudable, I note that there is a need to take this further – literally further in time, and to make some recommendations or guidelines about when value is measured.

I know that PMI's philosophy has (thankfully) asserted that project leaders must be involved earlier in the process of a project launch - during the 'rationale' stage.  I also know (from directly supervising about 150 project managers over the past three decades) that the personality of a project manager is focused on “get it done” and “now let’s move on to the next project”, not “let’s stay on and see how the deliverable works for a couple of months”. 

So it’s hard to get necessarily short-term, focused project managers to become long-term, holistic-thinking project leaders.  But if we are to truly measure a “Net Project Success Score”, shouldn’t there be some significant effort placed in the project’s product in the longer term, where “success” happens?

I had a conversation with ChatGPT about this.  I gave it my opinion about measuring value during and after a project’s outcome, and that exchange, which I think you will find illustrative, is below:

Your perspective has strong merit, especially considering that value realization often occurs after project completion, when deliverables are in use and benefits begin to accumulate. Here's how your argument holds up, along with some nuances to consider:

Why Measuring NPSS Post-Completion Makes Sense:

1. Value is Realized Over Time:
   • Many projects, especially in IT, infrastructure, or business transformation, don't show their true value until after implementation. Early NPSS measurements might capture surface-level satisfaction but miss the long-term impact.
   • For example, a new CRM system might launch smoothly, but the real value—improved customer engagement and sales growth—only becomes apparent months later.
2. Avoids Misleading Early Feedback:
   • During project execution, stakeholders may focus on immediate challenges (e.g., delays, budget overruns) rather than long-term benefits. Measuring NPSS mid-project risks skewing perceptions toward short-term frustrations rather than ultimate value.
   • It’s similar to judging a book halfway through—it might get better, but an early review locks in an incomplete assessment.
3. Aligns with Benefits Realization Management (BRM):
   • PMI’s own Benefits Realization framework supports the idea that project success is about achieving intended benefits, which often occur post-project.
   • Measuring NPSS after stakeholders have experienced the benefits better aligns with this philosophy.

Potential Counterpoints to Consider:

1. Stakeholder Perception Shifts:
   • By waiting until post-completion, there’s a risk that emotions fade or stakeholders' memories shift. Immediate post-project NPSS might capture stronger impressions, while a delayed NPSS might be more rational but less emotional.
2. Long-Term Tracking Approach:
   • Some thought leaders suggest a phased NPSS:
     • At Completion: Initial feedback on perceived value.
     • 6-12 Months Later: A follow-up NPSS to assess if the anticipated benefits materialized.
   • This method balances initial sentiment with real-world impact.
3. Complex Projects & Change Management:
   • For projects involving organizational change, early NPSS checkpoints can help gauge adoption and engagement. While they don’t measure final value, they provide insight into whether the project is on track to deliver it.

Your argument is solid—value is indeed a cumulative outcome, and measuring NPSS too early risks evaluating an incomplete picture. A post-implementation NPSS, possibly with a follow-up after benefits have been realized, seems like a balanced approach. It would focus on long-term value while still capturing immediate stakeholder reactions.

Next I asked the question many of you are thinking… WHEN does this make sense, and won’t it vary wildly amongst different practice areas like infrastructure compared to new game development?  The answer from ChatGPT:

You're correct that the optimal timing for conducting a Net Project Success Score (NPSS) assessment can vary significantly depending on the industry and specific project type. While there's no one-size-fits-all guideline, several factors can help determine the most appropriate timing for an NPSS evaluation:

1. Nature of the Project Deliverables:

• Immediate Impact Projects: For projects where deliverables provide immediate value upon completion—such as software deployments or event planning—conducting an NPSS assessment shortly after project handover can effectively capture initial stakeholder satisfaction.
• Delayed Impact Projects: In cases where benefits materialize over time, like infrastructure developments or organizational change initiatives, it's prudent to schedule the NPSS evaluation after a period that allows stakeholders to experience the project's full impact.

2. Industry-Specific Considerations:

• Technology Sector: Given the rapid evolution in technology, projects often require quick feedback. An NPSS assessment immediately post-implementation can help in making timely adjustments.
• Construction and Infrastructure: These projects might necessitate a longer interval before assessment to account for factors like settling periods or user adaptation.

3. Project Duration and Complexity:

• Short-Term Projects: For projects with brief timelines, a single NPSS assessment upon completion may suffice.
• Long-Term or Complex Projects: Ongoing NPSS evaluations at key milestones can provide continuous insights, allowing for adjustments and ensuring alignment with stakeholder expectations.

General Guidelines for Timing NPSS Assessments:

• Initial Assessment: Conduct an NPSS evaluation immediately after project completion to gauge immediate reactions and identify any pressing issues.
• Follow-Up Assessment: Schedule subsequent evaluations after stakeholders have had sufficient time to realize and assess the project's benefits. The exact timing should be tailored to the project's context and industry norms.

Ultimately, the goal is to balance obtaining timely feedback with allowing enough time for the project's value to manifest fully. Engaging with stakeholders to understand their expectations and experiences can further inform the optimal timing for NPSS assessments.

Finally, to come full-circle, I asked ChatGPT to consider the Flint, Michigan water cutover project as an example.  After a few back-and-forth exchanges, here is what it provided:

The Flint Water Crisis is indeed a profound example that underscores the critical importance of when and how to measure project value—and the dangers of focusing solely on short-term success metrics.

Key Lessons on Measuring Value:

1. Execution Success ≠ Value Delivered
   • In Flint’s case, the cutover project was celebrated for its cost savings and timely execution. From a traditional project management perspective—scope, time, budget—it might have been labeled a "success" initially.
   • BUT, the actual value—clean, safe drinking water—was not delivered. Worse, the project introduced disbenefits that were catastrophic: lead poisoning, illness, and long-term health impacts.
2. The Time Lag Between Completion and Realized Value
   • This highlights why measuring value immediately after project handover can be misleading. The negative outcomes (lead contamination, health issues) only became visible after residents used the new water supply for weeks or months.
   • Had an NPSS-style evaluation been conducted only at project handover, it might have shown high satisfaction among stakeholders (hence the celebration in the first image). But waiting until real benefits (or harms) became apparent would have revealed the truth.
3. The Importance of Measuring Disbenefits
   • Projects like Flint show how crucial it is to include disbenefits in success metrics.
   • Disbenefits here weren’t just technical (contamination) but reputational—city officials lost public trust, and the crisis became a national scandal.

How This Shapes TTV (Time to Value) and NPSS Measurement Thinking:

1. Post-Implementation Evaluation is Essential:
   • Value assessments shouldn’t end at cutover or project completion. Follow-up evaluations, especially for projects affecting public health and safety, are crucial.
   • In Flint, water quality should have been monitored continuously after the switch, with stakeholder feedback (residents' concerns about water taste, color, health symptoms) incorporated into evaluations.
2. Incorporate Long-Term Impact in Value Measurement:
   • Some projects require longitudinal studies—especially those involving health, safety, or environmental impacts. A multi-phase NPSS could be used:
     • Phase 1: Immediately post-implementation (technical delivery and initial stakeholder feedback).
     • Phase 2: 6-12 months later (to assess actual, experienced value).
     • Phase 3: Long-term (years later, for impacts on health, economy, or environment).
3. Stakeholder-Centric Evaluation:
   • Measuring success only from an executive or managerial lens (e.g., cost savings) ignores the experiences of end-users—in this case, Flint residents.
   • In NPSS terms, this emphasizes surveying the right stakeholders and focusing on perceived value, not just technical performance.

In Conclusion:

The Flint example is a cautionary tale about the dangers of prematurely declaring success and failing to measure true value over time. It highlights the importance of post-implementation evaluations, considering disbenefits, and ensuring that success metrics reflect the real-world impact on stakeholders—not just project execution metrics.

If NPSS had been applied over time, including stakeholder feedback post-cutover, the issues could have been identified sooner, potentially preventing harm.

If nothing else, this is significant food for thought about how we truly do value delivery and how we MEASURE value delivery, with a strong admonition about having it ONLY be during the project.

I value (pun intended) your thoughts on this, I know this can be controversial and difficult topic for us as project leaders.  Please respond with  your comments.  I believe a healthy conversation between humans – aided by side conversations with AI – is important.