Place yourself back in front of the Standing Review Board – you MUST address the final of the six required judgement criteria for your project.  There are, of course, many other items you’ll need to address, but this is the last of the minimal set.

THE GOAL OF RISK MANAGEMENT

• To manage risk in a holistic and coherent manner across the Agency
  • Agency strategic goals explicitly drive risk management activities at all levels
  • All risk types and their interactions are considered collectively during decision-making
  • Implementation of risk management in the context of complex institutional relationships (programs, projects, centers, contractors, …)
• To better match the stakeholder expectations and the “true” resources required to address the risks to achieve those expectations
  • Better comprehension of the risk that a Better comprehension of the risk that a decision decision-maker is accepting when is accepting when making commitments to stakeholders
  • Having an integrated perspective of risks when analyzing competing alternatives
• To better establish close ties between the selected alternative and the requirements derived from it
  • Derivation of achievable requirements through systematic characterization of uncertainties

ADEQUACY OF THE RISK MANAGEMENT APPROACH

NASA takes risk management VERY seriously.  In this blog, I’ve reduced the scope and detail of the complete NASA risk management approach to be applicable to a wide range of different industries and applications.  NASA’s Risk Management program provides a unified structure that applies to all agency activities to ensure that risk management decisions are delegated and/or elevated to the appropriate level.   The full Risk-Informed-Decision-Making handbook is 128 pages long, and that’s just one of the references I’m using.  My goal is to give the reader a “taste” of what each PM must know about risk management – a lot!

Risk Management includes opportunity management — recognizing that spaceflight is an inherently risky endeavor and that the proper attitude towards risk management is to reach an optimal balance between minimizing the potential for loss while maximizing the potential for gain (opportunity).

All forms of Risk Management consist of two main and joined processes:

1. A Risk (and opportunity) Informed Decision Making (RIDM) addresses informed selection of decision alternatives to assure effective approaches to achieving objectives.
2. Continuous Risk (and Opportunity) Management (CRM) addresses implementation of the selected alternative to assure that requirements are met.

The Project Manager is required to be totally conversant on the adequacy of their project’s risk management approach including:

• Risk-management plans and processes including:
  • Risk Informed Decision Making (RIDM)
  • Continuous Risk Management (CRM)
• Open and accepted risks
• Risk assessments
• Risk mitigation plans
• Resources for managing/mitigating risks.

NASA’s Definition of Risk

The definition of risk used is very like what is described in the PMBOK® guide as an output of “Identify Risks” and placed in the risk register (PMBOK® 11.2.3.1).  A risk is defined by: “EVENT may occur causing IMPACT, or If CAUSE exists, EVENT may occur leading to EFFECT.”

NASA defines this as a “Triplet”

• The scenario(s) leading to degraded performance with respect to one or more performance measures
• The likelihood(s) (qualitative or quantitative) of those scenarios.
• The consequences that would result if those scenarios were to occur.

Also, in agreement with the PMBOK® guide, the purpose of this type of risk definition is to be able to “sift” the high-probability, low-consequence risks from the low-probability, high-consequence risks.

THE NASA RISK MANAGEMENT PROCESS

NASA Risk Management processes are based on both Continuous Risk Management (CRM), which stresses the management of risk during implementation - and - Risk-Informed Decision Making (RIDM) which is concerned with analysis of important or direction-setting decisions.

Continuous Risk Management (CRM)

1 – Identify:  Search for and locate risks before they become problems or opportunities.  This is the process of transforming uncertainties and issues about a project into distinct (tangible) risks that can be described and measured.

2 – Analyze: Converts risk data into decision-making information. The process of examining the risks in detail to determine the extent of the risks, how they relate to each other, and which ones are the most important

3 – Plan: Translates risk information into decisions and mitigating (or enhancing) actions.  This part of the process deals with deciding what, if anything, should be done about a risk or set of related risks

4 – Track:   Answers the questions:  Are the risk indicators and actions plan followed?  This is the process in which risk status data are acquired, compiled, and reported

5 – Control: To make informed, timely, and effective decisions regarding risks and their mitigation or enhancement plans.  During this process the project team examines the tracking status reports for identified project risks and decides what actions to take based on the reported data

6 - Communicate & Document:  Provides information and feedback to the project on the risk activities, current risks, and emerging risks.   It is this process in which risk information is conveyed between all project stakeholders.

Risk Informed Decision Making (RIDM)

RIDM helps ensure that decisions between alternatives are conducted with an awareness of the risks associated with each.  This is done to help prevent late design changes which are often drivers of risk, cost overruns, schedule delays, and even cancellation.  Also, it has been found that most project cost-saving opportunities occur in the definition, planning, and early design phases of a project.

The RIDM process attempts to respond to some of the primary issues that have derailed programs in the past:

1. The mismatch between stakeholder expectations and the resources required to address the risks to achieve their expectations
2. The miscomprehension of the risk that a decision-maker is accepting when making commitments to stakeholders
3. The miscommunication in considering the respective risks associated with competing alternatives

The RIDM process acknowledges the role that human judgment plays in decisions, and that technical information cannot be the sole basis for decision making. This is not only because of inevitable gaps in the technical information, but also because decision making is an inherently subjective, values-based enterprise. 

RIDM is typically appropriate for decisions that have one or more of the following characteristics:

• High Stakes — High stakes are involved in the decision, such as significant costs, significant potential safety impacts, or the importance of meeting the project objectives.
• Complexity — The actual ramifications of alternatives are difficult to understand without detailed analysis.
• Uncertainty — Uncertainty in key inputs creates substantial uncertainty in the outcome of the decision alternatives and points to risks that may need to be managed.
• Multiple Attributes — Greater numbers of attributes cause a greater need for formal analysis.
• Diversity of Stakeholders — Extra attention is warranted to clarify objectives performance measures when the set of stakeholders reflects a diversity of values, preferences, and perspectives.

Throughout the RIDM process, interactions take place between the stakeholders, the risk analyst, the subject matter experts (SMEs), the Technical Authorities, and the decision-maker to ensure that the knowledge is properly integrated and communicated into the deliberations that inform the decision.

The RIDM Process

You can download a free copy of the RIDM process handbook at:  http://ow.ly/TCWH306qAq9

Part 1: Identification of Alternatives

Objectives are decomposed into an individual issue that is significant to some or all the stakeholders. In general, a performance measure has a “direction of goodness” that indicates the direction of increasingly beneficial performance measure values.

Considered are:

• Safety (e.g., avoidance of injury, fatality, or destruction of key assets)
• Technical (e.g., thrust or output, amount of observational data acquired)
• Cost (e.g., execution within allocated cost)
• Schedule (e.g., meeting milestones)

Part 2: Risk Analysis of Alternatives

In Risk Analysis of Alternatives, the performance measures of each alternative are quantified. 

It is incumbent on risk analyst to model each significant possible outcome, accounting for its probability of occurrence, in terms of the scenarios that produce it.  This produces a distribution of outcomes for each alternative, as characterized by probability density functions over the performance measures.  The depth of analysis needs to agree with the stakes and complexity of the decision situations being addressed.

Avoiding Decision Traps During Analysis

• Anchoring —The tendency of decision-makers to give extra weight to the first information they receive. It is related to a tendency for people to reason in terms of changes from a “baseline” and to formulate that baseline quickly and sometimes baselessly.

• Status Quo Bias — There is a tendency to want to preserve the status quo in weighing decision alternatives. Early designs of “horseless carriages” were strongly based on horse-drawn buggies, despite being not-optimal for engine-powered vehicles. There is also the tendency for managers to believe that if things go wrong with a decision, they are more likely to be punished for having acted vs. having allowed the status quo to continue.
• Sunk-Cost — The tendency to throw good money after bad: to try to recoup losses by continuing a course of action, even when the rational decision would be to walk away, based on the current state of knowledge. This bias is seen to operate in the perpetuation of projects that are floundering, to the point where additional investment diverts resources that would be better spent elsewhere.
• Confirmation Bias — The tendency to give greater weight to evidence that confirms our prior views.
• Framing — A class of biases that relate to the human tendency to respond to how a question is framed, regardless of the objective content of the question.  
• Overconfidence — The widespread tendency to underestimate the uncertainty that is inherent in the current state of knowledge. While most “experts” will acknowledge the presence of uncertainty in their assessments, they tend to do a poor job of estimating confidence intervals. This is particularly true for decisions that are challenging to implement, as many decisions at NASA are. In the face of multiple sources of uncertainty, people tend to pay attention to the few with which they have the most experience, and neglect others. It is also particularly true for highly unlikely events, where there is limited data available to inform expert judgment.
• Recallability — The tendency of people to be strongly influenced by experiences or events that are easier for them to recall, even if an analysis of that experience would yield a different answer. This means that dramatic or extreme events may play an unwarrantedly large role in decision making based on experience.

Part 3, Risk-Informed Alternative Selection

There are several approaches to selecting an alternative.  Deliberation takes place among the stakeholders and the decision-maker, and the decision-maker either culls the set of alternatives and asks for further scrutiny of the remaining alternatives OR selects an alternative for implementation OR asks for new alternatives.

Deliberation and decision making might take place in several venues over time.  The rationale for the  

• The risk deemed acceptable for each performance measure;
• The risk information and Risk Analysis of an Alternative Uncertain Conditions Performance Measure 1 Performance Measure n Probabilistically - Determined Outcomes Funding Environment Technology Development Limited Data Operating Environment Etc.
• Performance measures depicted for a single alternative Design, Test & Production Processes
  • Safety Risk …
  • Technical Risk
  • Cost Risk
  • Schedule Risk
• The pros and cons of each contending decision alternative, as discussed during the deliberations.

References

The following items are referenced in the text of this document:

1. NASA/SP-2010-576 – NASA Risk-informed Decision Making Handbook
2. Continuous Risk Management Guidebook – Software Engineering Institute, Carnegie Mellon University, 1996.
3. NPR 7120.5 – NASA Procedural Requirement, NASA Program and Project Management Processes and Requirements
4. NPR 8000.4 - NASA Procedural Requirement, Risk Management Procedural Requirements

The First Three Of Six Criteria That Each NASA Project Manager Must Know

“There is one other thing we should discuss, and that is the difference between management and leadership.  Management is figuring out how to get the job done.  Leadership is deciding what needs to be done and inspiring the team to get where you want to go.”  

- Program Manager, JSC

In my last blog, I discussed Project Oversight (a missing chapter in the PMBOK guide?). And, I left you with the intimidating “rule” from the Project and Program Handbook: 

“THE PROJECT MANAGER IS RESPONSIBLE FOR PLANNING AND SUPPORTING THESE (Project) REVIEWS.”

The Project Manager must present six assessment criteria for consideration and review. They are large meetings and once scheduled are never delayed, and never excused. The reviews will happen, and the Project Manager is responsible for the majority of the information to be presented. There is flexibility as to the timing, number and information to be stressed in each review.

I strongly believe that the six project criteria the NASA PM is responsible for addressing - makes sense for all industries and all projects – regardless of size or complexity.  They aren’t a part of NASA practice because they were NASA projects.  They are a part of NASA practice – because they are good common sense.  And, they should apply for nearly every company. 

From a NASA standard (7120D): “There are three reasons for conducting Independent Life Cycle Reviews: first, we want the program/project to receive independent assurance that they are doing the right thing; second, NASA senior management needs to understand that the program/project is on the right track, is performing according to plan, and that externally-imposed impediments to its success are being removed; and third, the Agency needs to provide our external stakeholders assurance we are doing the right thing.”

These reviews are also what I like to refer to these reviews as “PM school.”  It’s where each Project Manager learns what is expected of them, how powerful a position they actually have and best of all – help from more experienced managers, academic experts and other stakeholders.  

PROJECT PHASES

At the top level, program and project life cycles are divided into three phases: Concept, Formulation and Implementation.   The formal project reviews don’t start until the project passes a conceptual study review - (MCR or Mission Concept Review) and enters the formulation stage where requirements are further developed, refined and reviewed.   While these are slightly different words than PMI uses in the PMBOK® and other documents, the concepts are very similar – if not identical.

Above is a high-level diagram of the project life-cycle, I’ll describe the components of this life-cycle model in greater detail in a later blog.  But for now, I’m focusing on the “PM School” or project reviews.  As stated in the NASA standard, “These reviews are essential elements of conducting, managing, evaluating, and approving space flight projects.” 

WHO ATTENDS THESE REVIEWS?

The short answer is: stakeholders.  But it’s a carefully selected mix of stakeholders.  One interesting aspect is the inclusion of a “The Standing Review Board.”  They are a group of independent experts whose role is to “assess and evaluate project activities, advise projects and report their evaluations to the responsible organizations.”  They are responsible for conducting independent reviews of projects and providing objective, expert judgments to both the Project Manager and to upper management.

I can tell you from personal experience, they are smart, quick-witted, incisive and not hesitant to point out what they think is a problem in a Project Manager’s efforts or plans.   The goal is not to punish the project manager, but to ensure that everything that can be reasonably done to insure project success IS being done. 

Reviews are a big deal, a stressful time for a Project Manager, and a truly wonderful learning experience. You can absorb the thinking of experts from many fields – learn what they are looking for, learn what worries them, and continually improve your project management skills.

As I promised at the start, we’ll cover the first 3 of the 6 important areas covered during the reviews.  Each of these can be taken to a MUCH, MUCH deeper dive by the review group.  I’m only listing the high level view of each.

1. Alignment with Agency strategic goals – does this project “fit” within the overall strategic goals set by the agency? 

• Project requirements and constraints – Have the requirements that the project is planned to meet been well defined?  Are they complete, consistent, traceable to an overall mission or need, are they unambiguous (facts not opinions), what is each requirements relative importance and can they be verified once completed?
• Mission needs and success criteria
• Allocation of program requirements to projects
• Proactive management of changes in project scope and shortfalls

2. Adequacy of management approach - Do you have a project management approach that makes sense and the group believes will work? 

• Project authorization
• Management framework and plans
• Acquisition strategies
• Internal and external agreements

3. Adequacy of technical approach and success criteria

• Flow down of project requirements to systems/subsystems architecture and design
• Operations concepts that respond to and satisfy the requirements and mission needs

DOES THE PROJECT MANGER GET YELLED AT?

NO!  The goal is not to attack the work the Project Team has done.  The goal is to make sure that everyone is informed of the status and plans of the project.  Sometimes, mistakes will be discovered and that’s a good thing!  That’s what the review is all about.

It's about trying to do things right the first time, to uncover all of the potential roadblocks and keeping everyone informed.