Place yourself back in front of the Standing Review Board – you MUST address the final of the six required judgement criteria for your project.  There are, of course, many other items you’ll need to address, but this is the last of the minimal set.

THE GOAL OF RISK MANAGEMENT

• To manage risk in a holistic and coherent manner across the Agency
  • Agency strategic goals explicitly drive risk management activities at all levels
  • All risk types and their interactions are considered collectively during decision-making
  • Implementation of risk management in the context of complex institutional relationships (programs, projects, centers, contractors, …)
• To better match the stakeholder expectations and the “true” resources required to address the risks to achieve those expectations
  • Better comprehension of the risk that a Better comprehension of the risk that a decision decision-maker is accepting when is accepting when making commitments to stakeholders
  • Having an integrated perspective of risks when analyzing competing alternatives
• To better establish close ties between the selected alternative and the requirements derived from it
  • Derivation of achievable requirements through systematic characterization of uncertainties

ADEQUACY OF THE RISK MANAGEMENT APPROACH

NASA takes risk management VERY seriously.  In this blog, I’ve reduced the scope and detail of the complete NASA risk management approach to be applicable to a wide range of different industries and applications.  NASA’s Risk Management program provides a unified structure that applies to all agency activities to ensure that risk management decisions are delegated and/or elevated to the appropriate level.   The full Risk-Informed-Decision-Making handbook is 128 pages long, and that’s just one of the references I’m using.  My goal is to give the reader a “taste” of what each PM must know about risk management – a lot!

Risk Management includes opportunity management — recognizing that spaceflight is an inherently risky endeavor and that the proper attitude towards risk management is to reach an optimal balance between minimizing the potential for loss while maximizing the potential for gain (opportunity).

All forms of Risk Management consist of two main and joined processes:

1. A Risk (and opportunity) Informed Decision Making (RIDM) addresses informed selection of decision alternatives to assure effective approaches to achieving objectives.
2. Continuous Risk (and Opportunity) Management (CRM) addresses implementation of the selected alternative to assure that requirements are met.

The Project Manager is required to be totally conversant on the adequacy of their project’s risk management approach including:

• Risk-management plans and processes including:
  • Risk Informed Decision Making (RIDM)
  • Continuous Risk Management (CRM)
• Open and accepted risks
• Risk assessments
• Risk mitigation plans
• Resources for managing/mitigating risks.

NASA’s Definition of Risk

The definition of risk used is very like what is described in the PMBOK® guide as an output of “Identify Risks” and placed in the risk register (PMBOK® 11.2.3.1).  A risk is defined by: “EVENT may occur causing IMPACT, or If CAUSE exists, EVENT may occur leading to EFFECT.”

NASA defines this as a “Triplet”

• The scenario(s) leading to degraded performance with respect to one or more performance measures
• The likelihood(s) (qualitative or quantitative) of those scenarios.
• The consequences that would result if those scenarios were to occur.

Also, in agreement with the PMBOK® guide, the purpose of this type of risk definition is to be able to “sift” the high-probability, low-consequence risks from the low-probability, high-consequence risks.

THE NASA RISK MANAGEMENT PROCESS

NASA Risk Management processes are based on both Continuous Risk Management (CRM), which stresses the management of risk during implementation - and - Risk-Informed Decision Making (RIDM) which is concerned with analysis of important or direction-setting decisions.

Continuous Risk Management (CRM)

1 – Identify:  Search for and locate risks before they become problems or opportunities.  This is the process of transforming uncertainties and issues about a project into distinct (tangible) risks that can be described and measured.

2 – Analyze: Converts risk data into decision-making information. The process of examining the risks in detail to determine the extent of the risks, how they relate to each other, and which ones are the most important

3 – Plan: Translates risk information into decisions and mitigating (or enhancing) actions.  This part of the process deals with deciding what, if anything, should be done about a risk or set of related risks

4 – Track:   Answers the questions:  Are the risk indicators and actions plan followed?  This is the process in which risk status data are acquired, compiled, and reported

5 – Control: To make informed, timely, and effective decisions regarding risks and their mitigation or enhancement plans.  During this process the project team examines the tracking status reports for identified project risks and decides what actions to take based on the reported data

6 - Communicate & Document:  Provides information and feedback to the project on the risk activities, current risks, and emerging risks.   It is this process in which risk information is conveyed between all project stakeholders.

Risk Informed Decision Making (RIDM)

RIDM helps ensure that decisions between alternatives are conducted with an awareness of the risks associated with each.  This is done to help prevent late design changes which are often drivers of risk, cost overruns, schedule delays, and even cancellation.  Also, it has been found that most project cost-saving opportunities occur in the definition, planning, and early design phases of a project.

The RIDM process attempts to respond to some of the primary issues that have derailed programs in the past:

1. The mismatch between stakeholder expectations and the resources required to address the risks to achieve their expectations
2. The miscomprehension of the risk that a decision-maker is accepting when making commitments to stakeholders
3. The miscommunication in considering the respective risks associated with competing alternatives

The RIDM process acknowledges the role that human judgment plays in decisions, and that technical information cannot be the sole basis for decision making. This is not only because of inevitable gaps in the technical information, but also because decision making is an inherently subjective, values-based enterprise. 

RIDM is typically appropriate for decisions that have one or more of the following characteristics:

• High Stakes — High stakes are involved in the decision, such as significant costs, significant potential safety impacts, or the importance of meeting the project objectives.
• Complexity — The actual ramifications of alternatives are difficult to understand without detailed analysis.
• Uncertainty — Uncertainty in key inputs creates substantial uncertainty in the outcome of the decision alternatives and points to risks that may need to be managed.
• Multiple Attributes — Greater numbers of attributes cause a greater need for formal analysis.
• Diversity of Stakeholders — Extra attention is warranted to clarify objectives performance measures when the set of stakeholders reflects a diversity of values, preferences, and perspectives.

Throughout the RIDM process, interactions take place between the stakeholders, the risk analyst, the subject matter experts (SMEs), the Technical Authorities, and the decision-maker to ensure that the knowledge is properly integrated and communicated into the deliberations that inform the decision.

The RIDM Process

You can download a free copy of the RIDM process handbook at:  http://ow.ly/TCWH306qAq9

Part 1: Identification of Alternatives

Objectives are decomposed into an individual issue that is significant to some or all the stakeholders. In general, a performance measure has a “direction of goodness” that indicates the direction of increasingly beneficial performance measure values.

Considered are:

• Safety (e.g., avoidance of injury, fatality, or destruction of key assets)
• Technical (e.g., thrust or output, amount of observational data acquired)
• Cost (e.g., execution within allocated cost)
• Schedule (e.g., meeting milestones)

Part 2: Risk Analysis of Alternatives

In Risk Analysis of Alternatives, the performance measures of each alternative are quantified. 

It is incumbent on risk analyst to model each significant possible outcome, accounting for its probability of occurrence, in terms of the scenarios that produce it.  This produces a distribution of outcomes for each alternative, as characterized by probability density functions over the performance measures.  The depth of analysis needs to agree with the stakes and complexity of the decision situations being addressed.

Avoiding Decision Traps During Analysis

• Anchoring —The tendency of decision-makers to give extra weight to the first information they receive. It is related to a tendency for people to reason in terms of changes from a “baseline” and to formulate that baseline quickly and sometimes baselessly.

• Status Quo Bias — There is a tendency to want to preserve the status quo in weighing decision alternatives. Early designs of “horseless carriages” were strongly based on horse-drawn buggies, despite being not-optimal for engine-powered vehicles. There is also the tendency for managers to believe that if things go wrong with a decision, they are more likely to be punished for having acted vs. having allowed the status quo to continue.
• Sunk-Cost — The tendency to throw good money after bad: to try to recoup losses by continuing a course of action, even when the rational decision would be to walk away, based on the current state of knowledge. This bias is seen to operate in the perpetuation of projects that are floundering, to the point where additional investment diverts resources that would be better spent elsewhere.
• Confirmation Bias — The tendency to give greater weight to evidence that confirms our prior views.
• Framing — A class of biases that relate to the human tendency to respond to how a question is framed, regardless of the objective content of the question.  
• Overconfidence — The widespread tendency to underestimate the uncertainty that is inherent in the current state of knowledge. While most “experts” will acknowledge the presence of uncertainty in their assessments, they tend to do a poor job of estimating confidence intervals. This is particularly true for decisions that are challenging to implement, as many decisions at NASA are. In the face of multiple sources of uncertainty, people tend to pay attention to the few with which they have the most experience, and neglect others. It is also particularly true for highly unlikely events, where there is limited data available to inform expert judgment.
• Recallability — The tendency of people to be strongly influenced by experiences or events that are easier for them to recall, even if an analysis of that experience would yield a different answer. This means that dramatic or extreme events may play an unwarrantedly large role in decision making based on experience.

Part 3, Risk-Informed Alternative Selection

There are several approaches to selecting an alternative.  Deliberation takes place among the stakeholders and the decision-maker, and the decision-maker either culls the set of alternatives and asks for further scrutiny of the remaining alternatives OR selects an alternative for implementation OR asks for new alternatives.

Deliberation and decision making might take place in several venues over time.  The rationale for the  

• The risk deemed acceptable for each performance measure;
• The risk information and Risk Analysis of an Alternative Uncertain Conditions Performance Measure 1 Performance Measure n Probabilistically - Determined Outcomes Funding Environment Technology Development Limited Data Operating Environment Etc.
• Performance measures depicted for a single alternative Design, Test & Production Processes
  • Safety Risk …
  • Technical Risk
  • Cost Risk
  • Schedule Risk
• The pros and cons of each contending decision alternative, as discussed during the deliberations.

References

The following items are referenced in the text of this document:

1. NASA/SP-2010-576 – NASA Risk-informed Decision Making Handbook
2. Continuous Risk Management Guidebook – Software Engineering Institute, Carnegie Mellon University, 1996.
3. NPR 7120.5 – NASA Procedural Requirement, NASA Program and Project Management Processes and Requirements
4. NPR 8000.4 - NASA Procedural Requirement, Risk Management Procedural Requirements

THE FOURTH OF SIX CRITERIA THAT A NASA PROJECT MANAGER MUST KNOW.

This blog discusses the fourth of six project criteria every Project Manager is held responsible for – and must clearly define for their project(s).  I’m taking these one at a time now, since they get a bit into the more technical side of Project Management.

“You, as a project manager, are called on to make some key decisions, but you are also riding on the top of the 95 percent of the good decisions that were made by the people you delegated to. So, it is a team activity and how you treat and manage those people … makes the real difference.”

 – Human Research Facility Project Manager, JSC

You need to imagine that we’re still doing a stand-up review in front of a LOT of smart people as described in my previous blogs.   People that have managed complex projects for years.   These are people that are experts in risk analysis as well as academics that understand the mission of the project also of course – your bosses are there.   This is a big deal!  It’s your opportunity to show how good a project manager you are – and – to learn things from the audience.  It’s a *very* interactive meeting.

Criteria 4. Adequacy of integrated cost and schedule estimate and funding strategy

I don’t want to get too nerdy here.  But the bottom line is that careful estimates are made, integrated with risk management and tracked with earned value. 

4A: Cost and schedule control plans

Yes, we’re talking about full-blown Earned Value Management!  (see https://evm.nasa.gov/)  Typically, effort uncertainty is modeled using a three-point estimate at the activity or a summary (Work Package) level. The lowest estimated value represents the low extreme of uncertainty, the middle value represents the “most likely” value of the cost or duration, and the high value represents the high extreme of uncertainty.  These estimates are linked to the identified risks for the project to establish a reasonable cost and perhaps schedule reserves.  

“Throughout the execution of the project, the Project Manager shall ensure that the results of all analysis based on EVM are linked to the Risk Management Plan of the Project. Any cost and/or schedule risks being managed by the Project Manger should rely on the results of EVM analysis to track, manage, and mitigate risks.” - NPR 9501.3

You can have reserves, but they must be smartly estimated, reviewed and disclosed.  “Undisclosed reserve” is a bad thing.  Reserves (both cost and schedule) can be handled different ways, but you MUST be consistent in the way they are managed and presented. Below are a few options that are offered as guidance.  

“The PM can make reserve numbers available within the project so all project team members know what reserves are, or the PM can keep reserve numbers quiet.  For example, a PM on a recent successful science mission allowed science instrument teams to have insight into how much reserve he had allocated for each instrument, but reserve was held at the project level. This openness allowed everyone to see the situation but also provided oversight and control” - NPR 9501.3

4B: Basis of Estimate (BoE)

A bottom-up (from the lowest level WBS elements) analysis is a common way to approach this.  Documenting the basis of estimate is often invaluable in the latter phases of the project.  The estimate’s focus should be on:

• Clarity of the objectives
• Thoroughness in the state of the technical and management plan
• Complexity of technology needs

This is exactly what is described in the PMBOK® Guide’s description of a Bottom-Up Estimate “Estimate Activity Resources” in paragraph 6.4.2.4

4C: Cost and schedule estimates consistent with project requirements assumptions, risks, and margins

Probably one of the most interesting item prepared and reviewed is the Joint Cost and Schedule Confidence Level (JCL) of the project.   This isn’t a PMBOK® Guide topic, but it certainly builds on what is addressed in the guide.

NASA’s Project Management requires that projects develop probabilistic risk-informed analyses of cost and schedule estimates to obtain a quantitative measure of the likelihood that the estimate will be met.  Risk analysis provides an analytical basis for establishing defensible cost estimates for likely project risks.  This analysis must be continuously reviewed and updated as more data become available.  A risk analysis, consists of answering the following questions:

• What can happen?
• How likely is it that it will happen?
• If it does happen, what are the consequences?

The cost analysis considers:

• Possible risks, threats, liens, uncertainties, mitigation strategies, and opportunities must be explicitly quantified, including the following:
  • Their probability of occurring
  • Their estimated cost and/or schedule consequences

Risk analysis utilizes modeling, analysis, and evaluation and contains various types of uncertainty. In general, these uncertainties may be attributable to several factors that include

• The statistical nature of data
• The insufficient understanding of physical and biological phenomena
• Unpredictable events (e.g., natural, biological, and human behavior).
• Impacts of cost and schedule performance to date

4D: Joint Cost and Schedule Confidence Level

Joint Cost and Schedule Confidence Level (JCL) is a process that combines a project’s cost, schedule, and risk into a complete picture.  The probability that the project cost will be equal to or less than the targeted cost and that schedule will be equal to or less than the targeted schedule date.   This helps inform management of the likelihood of a project’s success.

Why Do a JCL?

JCL analysis provides a cohesive and holistic picture of the project’s ability to achieve cost and schedule goals by integrating technical, cost, schedule, and risk data.  The project’s JCL can show the impacts of risk to a project as well as highlight the relationship between cost and schedule. This relationship can be extremely important in situations with constrained budgets. A complete JCL analysis also facilitates transparency with stakeholders on expectations and probabilities of meeting those expectations.

What is the official definition of JCL?

JCL is:

• The probability that cost will be equal or less than the targeted cost AND schedule will be equal or less then the targeted schedule date
• A process and product that helps inform management the likelihood of a projects’ programmatic success
• A process that combines a projects’ cost, schedule, and risk into a complete picture

JCL is not a:

• Specific methodology (e.g. resource loaded schedule)
• Product from a specific tool (e.g. @RISK)

The Four Key JCL Inputs:

1. DEVELOP A SUMMARY ANALYSIS SCHEDULE
   • Build a logic network of activities.
   • Utilizing a summary analysis schedule can significantly improve the process.

2. LOAD COST ONTO THE SCHEDULE ACTIVITIES
   • Map cost to the schedule.
   • Cost data can be summarized by a work breakdown structure (WBS) to aid with mapping.

3. INCORPORATE RISK LIST
   • Quantify likelihood of occurrence and impact.
   • Map risks to the appropriate activities.

4. CONDUCT UNCERTAINTY ANALYSIS
   • Apply uncertainty to the schedule and cost.  (The results of a quantitative analysis)

CALCULATE AND VIEW RESULTS

To calculate a JCL, the project combines its cost, schedule, and risk into a single model that can generate a probabilistic assessment of the level of confidence of achieving a specific cost-schedule goal.  Programs must be baselined at a 70 percent probability that the projects will be completed at or below the estimated cost and at or before the projected schedule.

The JCL scatterplot is a standard XY chart with schedule on the X-axis and cost on the Y-axis. Each point is a result of the simulation calculation representing one cost and schedule pair. The JCL calculation is based on the number of results in the target area for both cost and schedule and is expressed as a percentage of all simulation results displayed on the scatterplot.

Establishing a cost and schedule target (black dotted lines) divides the scatterplot into two areas. One area contains results that are at or beneath the target (shown in green). The other area contains results that exceed the target (shown in blue).

The yellow points and frontier line represent all results from the simulation that meet a desired Joint Confidence Level. Multiple points from the simulation may meet the JCL target. Each of the yellow points would establish a new target area large enough to meet the desired JCL

The Joint Cost and Schedule Confidence Level model describes the:

• Basis for base schedule duration and logic
• Basis for baseline cost estimates
• Risks included and basis for probability and consequences
• Risks excluded and why
• Description of the JCL method use

Analyze the scatterplot, run sensitivities, and refine!

Facts and Myths About JCL

MYTH: JCL analysis requires expensive software tools.

FACT: NASA has JCL tools available at no cost to the projects.

MYTH: A JCL requires a detailed resource-loaded schedule.

FACT: Completing a JCL requires only costs, not labor categories and rates.

MYTH: A JCL must be based on a detailed integrated master schedule (IMS).

FACT:  Summary and analysis schedules are preferred!

A full-blown Project Joint Cost and Schedule Confidence Level

The First Three Of Six Criteria That Each NASA Project Manager Must Know

“There is one other thing we should discuss, and that is the difference between management and leadership.  Management is figuring out how to get the job done.  Leadership is deciding what needs to be done and inspiring the team to get where you want to go.”  

- Program Manager, JSC

In my last blog, I discussed Project Oversight (a missing chapter in the PMBOK guide?). And, I left you with the intimidating “rule” from the Project and Program Handbook: 

“THE PROJECT MANAGER IS RESPONSIBLE FOR PLANNING AND SUPPORTING THESE (Project) REVIEWS.”

The Project Manager must present six assessment criteria for consideration and review. They are large meetings and once scheduled are never delayed, and never excused. The reviews will happen, and the Project Manager is responsible for the majority of the information to be presented. There is flexibility as to the timing, number and information to be stressed in each review.

I strongly believe that the six project criteria the NASA PM is responsible for addressing - makes sense for all industries and all projects – regardless of size or complexity.  They aren’t a part of NASA practice because they were NASA projects.  They are a part of NASA practice – because they are good common sense.  And, they should apply for nearly every company. 

From a NASA standard (7120D): “There are three reasons for conducting Independent Life Cycle Reviews: first, we want the program/project to receive independent assurance that they are doing the right thing; second, NASA senior management needs to understand that the program/project is on the right track, is performing according to plan, and that externally-imposed impediments to its success are being removed; and third, the Agency needs to provide our external stakeholders assurance we are doing the right thing.”

These reviews are also what I like to refer to these reviews as “PM school.”  It’s where each Project Manager learns what is expected of them, how powerful a position they actually have and best of all – help from more experienced managers, academic experts and other stakeholders.  

PROJECT PHASES

At the top level, program and project life cycles are divided into three phases: Concept, Formulation and Implementation.   The formal project reviews don’t start until the project passes a conceptual study review - (MCR or Mission Concept Review) and enters the formulation stage where requirements are further developed, refined and reviewed.   While these are slightly different words than PMI uses in the PMBOK® and other documents, the concepts are very similar – if not identical.

Above is a high-level diagram of the project life-cycle, I’ll describe the components of this life-cycle model in greater detail in a later blog.  But for now, I’m focusing on the “PM School” or project reviews.  As stated in the NASA standard, “These reviews are essential elements of conducting, managing, evaluating, and approving space flight projects.” 

WHO ATTENDS THESE REVIEWS?

The short answer is: stakeholders.  But it’s a carefully selected mix of stakeholders.  One interesting aspect is the inclusion of a “The Standing Review Board.”  They are a group of independent experts whose role is to “assess and evaluate project activities, advise projects and report their evaluations to the responsible organizations.”  They are responsible for conducting independent reviews of projects and providing objective, expert judgments to both the Project Manager and to upper management.

I can tell you from personal experience, they are smart, quick-witted, incisive and not hesitant to point out what they think is a problem in a Project Manager’s efforts or plans.   The goal is not to punish the project manager, but to ensure that everything that can be reasonably done to insure project success IS being done. 

Reviews are a big deal, a stressful time for a Project Manager, and a truly wonderful learning experience. You can absorb the thinking of experts from many fields – learn what they are looking for, learn what worries them, and continually improve your project management skills.

As I promised at the start, we’ll cover the first 3 of the 6 important areas covered during the reviews.  Each of these can be taken to a MUCH, MUCH deeper dive by the review group.  I’m only listing the high level view of each.

1. Alignment with Agency strategic goals – does this project “fit” within the overall strategic goals set by the agency? 

• Project requirements and constraints – Have the requirements that the project is planned to meet been well defined?  Are they complete, consistent, traceable to an overall mission or need, are they unambiguous (facts not opinions), what is each requirements relative importance and can they be verified once completed?
• Mission needs and success criteria
• Allocation of program requirements to projects
• Proactive management of changes in project scope and shortfalls

2. Adequacy of management approach - Do you have a project management approach that makes sense and the group believes will work? 

• Project authorization
• Management framework and plans
• Acquisition strategies
• Internal and external agreements

3. Adequacy of technical approach and success criteria

• Flow down of project requirements to systems/subsystems architecture and design
• Operations concepts that respond to and satisfy the requirements and mission needs

DOES THE PROJECT MANGER GET YELLED AT?

NO!  The goal is not to attack the work the Project Team has done.  The goal is to make sure that everyone is informed of the status and plans of the project.  Sometimes, mistakes will be discovered and that’s a good thing!  That’s what the review is all about.

It's about trying to do things right the first time, to uncover all of the potential roadblocks and keeping everyone informed.