The Smallest Thing Can Cause Major Problems, and take a long time to find & fix

Life Lessons I learned as a Project Manager at NASA. 

These are things that my mind drifts to occasionally.   They are real, they happened to me while managing projects, but had a larger life-changing effect.   They affected my personality, my outlook on life and the way my brain works.

A couple I’ve already blogged about, but in the light of “how we managed projects.”  These blogs are a series of how managing projects at NASA changed my life.  I’m sure the same thing has happened to you.  You learn from managing projects and those lessons apply to more than your next project.  They go into your brain and are “compiled” into your personality

BACKGROUND

The primary caution and warning system is designed to warn the crew of conditions that may adversely affect orbiter operations. The system consists of hardware and electronics that provide the crew with both visual and aural cues when a system exceeds predefined operating limits.

The primary system's visual cues consist of four master alarm lights, a 40-light array on panel F7 and a 120-light array on panel R13.  The caution and warning system interfaces with the auxiliary power units, data processing system, environmental control and life support system, electrical power system, flight control system, guidance and navigation, hydraulics, main propulsion system, reaction control system, orbital maneuvering system and payloads.   

THE PROBLEM

One day one of the indicators on the caution and warning panel was “stuck on.”  I can’t remember which one it was, but it didn’t take long to discover that it SHOULDN’T be on.  It was a false indication of a problem.  Think of a Project Management dashboard that shows a problem – that doesn’t really exist.  ANNOYING.  And, in this case it was VERY annoying to the crew.  Everyone wants to fly in something that has all its parts working.  I know I do!  The false indication seemed to suggest there was something in the Avionics or Glass displays that was wrong.  That’s how I got involved. 

What could cause a false indication? 

The shuttle used MANY MIL-standard 61 pin connectors for the cables.  A GREAT many of them.   They are very rugged, reliable and only semi-difficult to work with.  And the electrical systems on the orbiter are complex with lengthy runs, packed cables and difficult to get to.  Unplugging a cable is something like standing on your head while trying to unscrew a garden hose.   Meanwhile schedules were slipping, deadlines being missed, testing delayed, EVERYONE was aware of the problem and asked me about it all the time.  It moved to the top of the “squawk” list.  Again, I was encountering a “Significant Emotional Event.” 

Finding the problem took  a long, long time.  Scope here, meter there, inspect visually here.  That doesn’t sound too difficult but take a look at the wiring we had.  And there’s lots of connectors!!  There was a problem in there someplace.  The wires were neatly laced, the pins were crimped on and the 61 pin connectors were all in the right places.   Every time a lacing was cut, it had to be re-done and re-inspected.

My Project:  THE LIGHT WAS ON.  FIX IT!

The charter: Manage a group of about 12 people to scientifically and methodically look for the problem.  

We laid out a plan, devised ways to “split the problem” succcussively into halves until we could isolate it. 

This was a very boring task.  Not thrilling, not doing engineering,  not math…  But it was a problem and it had to be fixed.   After about a week I found it.

IT WAS A BENT PIN

The life lesson learned from this rather simple but important project was that sometimes it’s the littlest thing that can set your project back and ruin your schedule.  But you can’t give up – you must “press ahead.”   I think this is  true of nearly every project.

In other words: “Notice the small things. The rewards are inversely proportional.” -  Liz Vassey

By the way, I can still tie the NASA standard lacing knot.  I use it for everything from tying up bags of leaves to fixing my lawn mower.

Here's what I consider to be the "standard" lacing knot -- a single (not running) lacing knot. 

Place yourself back in front of the Standing Review Board – you MUST address the final of the six required judgement criteria for your project.  There are, of course, many other items you’ll need to address, but this is the last of the minimal set.

THE GOAL OF RISK MANAGEMENT

• To manage risk in a holistic and coherent manner across the Agency
  • Agency strategic goals explicitly drive risk management activities at all levels
  • All risk types and their interactions are considered collectively during decision-making
  • Implementation of risk management in the context of complex institutional relationships (programs, projects, centers, contractors, …)
• To better match the stakeholder expectations and the “true” resources required to address the risks to achieve those expectations
  • Better comprehension of the risk that a Better comprehension of the risk that a decision decision-maker is accepting when is accepting when making commitments to stakeholders
  • Having an integrated perspective of risks when analyzing competing alternatives
• To better establish close ties between the selected alternative and the requirements derived from it
  • Derivation of achievable requirements through systematic characterization of uncertainties

ADEQUACY OF THE RISK MANAGEMENT APPROACH

NASA takes risk management VERY seriously.  In this blog, I’ve reduced the scope and detail of the complete NASA risk management approach to be applicable to a wide range of different industries and applications.  NASA’s Risk Management program provides a unified structure that applies to all agency activities to ensure that risk management decisions are delegated and/or elevated to the appropriate level.   The full Risk-Informed-Decision-Making handbook is 128 pages long, and that’s just one of the references I’m using.  My goal is to give the reader a “taste” of what each PM must know about risk management – a lot!

Risk Management includes opportunity management — recognizing that spaceflight is an inherently risky endeavor and that the proper attitude towards risk management is to reach an optimal balance between minimizing the potential for loss while maximizing the potential for gain (opportunity).

All forms of Risk Management consist of two main and joined processes:

1. A Risk (and opportunity) Informed Decision Making (RIDM) addresses informed selection of decision alternatives to assure effective approaches to achieving objectives.
2. Continuous Risk (and Opportunity) Management (CRM) addresses implementation of the selected alternative to assure that requirements are met.

The Project Manager is required to be totally conversant on the adequacy of their project’s risk management approach including:

• Risk-management plans and processes including:
  • Risk Informed Decision Making (RIDM)
  • Continuous Risk Management (CRM)
• Open and accepted risks
• Risk assessments
• Risk mitigation plans
• Resources for managing/mitigating risks.

NASA’s Definition of Risk

The definition of risk used is very like what is described in the PMBOK® guide as an output of “Identify Risks” and placed in the risk register (PMBOK® 11.2.3.1).  A risk is defined by: “EVENT may occur causing IMPACT, or If CAUSE exists, EVENT may occur leading to EFFECT.”

NASA defines this as a “Triplet”

• The scenario(s) leading to degraded performance with respect to one or more performance measures
• The likelihood(s) (qualitative or quantitative) of those scenarios.
• The consequences that would result if those scenarios were to occur.

Also, in agreement with the PMBOK® guide, the purpose of this type of risk definition is to be able to “sift” the high-probability, low-consequence risks from the low-probability, high-consequence risks.

THE NASA RISK MANAGEMENT PROCESS

NASA Risk Management processes are based on both Continuous Risk Management (CRM), which stresses the management of risk during implementation - and - Risk-Informed Decision Making (RIDM) which is concerned with analysis of important or direction-setting decisions.

Continuous Risk Management (CRM)

1 – Identify:  Search for and locate risks before they become problems or opportunities.  This is the process of transforming uncertainties and issues about a project into distinct (tangible) risks that can be described and measured.

2 – Analyze: Converts risk data into decision-making information. The process of examining the risks in detail to determine the extent of the risks, how they relate to each other, and which ones are the most important

3 – Plan: Translates risk information into decisions and mitigating (or enhancing) actions.  This part of the process deals with deciding what, if anything, should be done about a risk or set of related risks

4 – Track:   Answers the questions:  Are the risk indicators and actions plan followed?  This is the process in which risk status data are acquired, compiled, and reported

5 – Control: To make informed, timely, and effective decisions regarding risks and their mitigation or enhancement plans.  During this process the project team examines the tracking status reports for identified project risks and decides what actions to take based on the reported data

6 - Communicate & Document:  Provides information and feedback to the project on the risk activities, current risks, and emerging risks.   It is this process in which risk information is conveyed between all project stakeholders.

Risk Informed Decision Making (RIDM)

RIDM helps ensure that decisions between alternatives are conducted with an awareness of the risks associated with each.  This is done to help prevent late design changes which are often drivers of risk, cost overruns, schedule delays, and even cancellation.  Also, it has been found that most project cost-saving opportunities occur in the definition, planning, and early design phases of a project.

The RIDM process attempts to respond to some of the primary issues that have derailed programs in the past:

1. The mismatch between stakeholder expectations and the resources required to address the risks to achieve their expectations
2. The miscomprehension of the risk that a decision-maker is accepting when making commitments to stakeholders
3. The miscommunication in considering the respective risks associated with competing alternatives

The RIDM process acknowledges the role that human judgment plays in decisions, and that technical information cannot be the sole basis for decision making. This is not only because of inevitable gaps in the technical information, but also because decision making is an inherently subjective, values-based enterprise. 

RIDM is typically appropriate for decisions that have one or more of the following characteristics:

• High Stakes — High stakes are involved in the decision, such as significant costs, significant potential safety impacts, or the importance of meeting the project objectives.
• Complexity — The actual ramifications of alternatives are difficult to understand without detailed analysis.
• Uncertainty — Uncertainty in key inputs creates substantial uncertainty in the outcome of the decision alternatives and points to risks that may need to be managed.
• Multiple Attributes — Greater numbers of attributes cause a greater need for formal analysis.
• Diversity of Stakeholders — Extra attention is warranted to clarify objectives performance measures when the set of stakeholders reflects a diversity of values, preferences, and perspectives.

Throughout the RIDM process, interactions take place between the stakeholders, the risk analyst, the subject matter experts (SMEs), the Technical Authorities, and the decision-maker to ensure that the knowledge is properly integrated and communicated into the deliberations that inform the decision.

The RIDM Process

You can download a free copy of the RIDM process handbook at:  http://ow.ly/TCWH306qAq9

Part 1: Identification of Alternatives

Objectives are decomposed into an individual issue that is significant to some or all the stakeholders. In general, a performance measure has a “direction of goodness” that indicates the direction of increasingly beneficial performance measure values.

Considered are:

• Safety (e.g., avoidance of injury, fatality, or destruction of key assets)
• Technical (e.g., thrust or output, amount of observational data acquired)
• Cost (e.g., execution within allocated cost)
• Schedule (e.g., meeting milestones)

Part 2: Risk Analysis of Alternatives

In Risk Analysis of Alternatives, the performance measures of each alternative are quantified. 

It is incumbent on risk analyst to model each significant possible outcome, accounting for its probability of occurrence, in terms of the scenarios that produce it.  This produces a distribution of outcomes for each alternative, as characterized by probability density functions over the performance measures.  The depth of analysis needs to agree with the stakes and complexity of the decision situations being addressed.

Avoiding Decision Traps During Analysis

• Anchoring —The tendency of decision-makers to give extra weight to the first information they receive. It is related to a tendency for people to reason in terms of changes from a “baseline” and to formulate that baseline quickly and sometimes baselessly.

• Status Quo Bias — There is a tendency to want to preserve the status quo in weighing decision alternatives. Early designs of “horseless carriages” were strongly based on horse-drawn buggies, despite being not-optimal for engine-powered vehicles. There is also the tendency for managers to believe that if things go wrong with a decision, they are more likely to be punished for having acted vs. having allowed the status quo to continue.
• Sunk-Cost — The tendency to throw good money after bad: to try to recoup losses by continuing a course of action, even when the rational decision would be to walk away, based on the current state of knowledge. This bias is seen to operate in the perpetuation of projects that are floundering, to the point where additional investment diverts resources that would be better spent elsewhere.
• Confirmation Bias — The tendency to give greater weight to evidence that confirms our prior views.
• Framing — A class of biases that relate to the human tendency to respond to how a question is framed, regardless of the objective content of the question.  
• Overconfidence — The widespread tendency to underestimate the uncertainty that is inherent in the current state of knowledge. While most “experts” will acknowledge the presence of uncertainty in their assessments, they tend to do a poor job of estimating confidence intervals. This is particularly true for decisions that are challenging to implement, as many decisions at NASA are. In the face of multiple sources of uncertainty, people tend to pay attention to the few with which they have the most experience, and neglect others. It is also particularly true for highly unlikely events, where there is limited data available to inform expert judgment.
• Recallability — The tendency of people to be strongly influenced by experiences or events that are easier for them to recall, even if an analysis of that experience would yield a different answer. This means that dramatic or extreme events may play an unwarrantedly large role in decision making based on experience.

Part 3, Risk-Informed Alternative Selection

There are several approaches to selecting an alternative.  Deliberation takes place among the stakeholders and the decision-maker, and the decision-maker either culls the set of alternatives and asks for further scrutiny of the remaining alternatives OR selects an alternative for implementation OR asks for new alternatives.

Deliberation and decision making might take place in several venues over time.  The rationale for the  

• The risk deemed acceptable for each performance measure;
• The risk information and Risk Analysis of an Alternative Uncertain Conditions Performance Measure 1 Performance Measure n Probabilistically - Determined Outcomes Funding Environment Technology Development Limited Data Operating Environment Etc.
• Performance measures depicted for a single alternative Design, Test & Production Processes
  • Safety Risk …
  • Technical Risk
  • Cost Risk
  • Schedule Risk
• The pros and cons of each contending decision alternative, as discussed during the deliberations.

References

The following items are referenced in the text of this document:

1. NASA/SP-2010-576 – NASA Risk-informed Decision Making Handbook
2. Continuous Risk Management Guidebook – Software Engineering Institute, Carnegie Mellon University, 1996.
3. NPR 7120.5 – NASA Procedural Requirement, NASA Program and Project Management Processes and Requirements
4. NPR 8000.4 - NASA Procedural Requirement, Risk Management Procedural Requirements

The Fifth of Six Criteria That Each NASA Project Manager Must Know

5A: ADEQUACY AND AVAILABILITY OF RESOURCES OTHER THAN BUDGET

This is often examined during the Project Reviews with the simple question: “Do you have a handle on the resources you’ll need?”  Without the correct resources, available at the correct time, a project will certainly encounter trouble or even fail.  This critical need is examined by the review committee, often with recommendations being made. By now you are probably asking:  What happened to the budget?  Isn’t money a concern?  That was part of an earlier blog that discussed the Joint Cost and Schedule Confidence Level (JCL) This is the probability that cost will be equal to or less than the targeted cost, and schedule will be equal to or less than the targeted schedule date.  During the conduct of a project review – very little is missed! 

“Project teams should embrace the external reviews. External reviews allow the project to think about all the tough questions they’re going to be asked and give them time to plug the holes. Brainstorm possible questions with the team to make sure they are covered. Next, projects should determine what the decision points are and what decision trees should be used for addressing these points. The project manager should assign actions and revisit them prior to the review, using this as a preparation for the review.”

— JPL Programs and Projects Manager

Of course, the resources required for a project of course include more than people.  I’ve made this mistake a few times, and believe me, I hope I won’t do it again.  NASA projects require not only highly skilled people, but Commercial-Off-The-Shelf (COTS) components and perhaps unique, complex, one-of-a-kind items created by a vendor, or internal NASA shops.   These are all resources that are needed to complete the project.  It’s part of your duty to make sure you understand what you’ll need, when you’ll need it and what to do if it’s not available.  So, each PM is judged on their having a documented understanding of the total resources required to complete the scope of their Project

Quoting the NASA standard: “Adequacy and availability of resources other than other than budget are essential elements of successful project functionality, implementation and operation. These resources include: workforce, fabrication, assembly, test facilities and equipment, test beds, ground support equipment, launch sites, communication networks, and mission operation centers. They can be either government or privately held resources.”

Each Project Manager must have a handle on all resources and where the need for that resource came from – in other words, what requirement is driving the need?  This in-depth and personal understanding includes the planning, projected or current availability of components and staffing, competency and stability of staffing, required infrastructure, and the industrial base/supplier chain requirements.  There’s a lot of investigation, deep-thinking and planning required to create the needed comfort-factor that “requirements other than budget” are completely understood.?

Essentially the review board is looking for:

• Planning, availability, competency and stability of staffing, infrastructure

• The industrial base/supplier chain requirements

• Planning, availability, competency and stability of staffing, and infrastructure requirements.

Resource Dashboard

The standard reporting system for the Review Board is a three-level metric scale, i.e., successful (green), partially successful (yellow), or unsuccessful (red). This is sometimes referred to as a stop-light assessment.

For judging Resource Adequacy:

                Successful: (Green Status): All key implementation facilities have been identified and are available to support near term (5-year) missions.  This includes the availability, competency and Stability of staffing, essential infrastructure and additional resources are adequate for remaining lifecycle phases.

                Partially Successful: (Yellow Status):  All key resources, may not be identified to support near term (5-year) missions, known resources may not be available when needed, external resource needs are notional. Preliminary staffing and essential infrastructure requirements have been identified and documented; preliminary sources have been identified.

                Unsuccessful: (Red Status): Needed resources and/or facilities are not identified, availability of either internal or external resources are unknown.  Staffing resource needs are clearly inadequate.

The Review Board’s assessment will consider not only the adequacy of the proposed and acquired resources, but also alternatives that might reduce cost or risk, or perhaps improve the performance of associated life-cycle activities. As with the other assessments, the Review Board must understand the margins and constraints for the project especially as it relates to current and planned workforce loading

“Clearly, having a good system philosophy and well-transmitted expectations makes a big difference in how they do their jobs.”

– Project Manager, JSC

Overall Resource Acquisition Strategy: As early as possible in planning, all project types begin to define theirs acquisition strategy. The Acquisition Strategy is the plan or approach for using NASA’s acquisition authorities to achieve the project’s mission.   This includes recommendations from make/buy analyses, the recommendations from vendor competition analyses, proposed partnerships and contributions, proposed infrastructure use and needs, budget, and other considerations.

This documented strategy addresses the project’s initial plans for obtaining the systems, research, services, construction, and supplies that it needs to fulfill its mission, including any known procurement(s); the availability of the industrial base capability and supply chain needed to design, develop, produce, and support the project and its planned projects; identifying risks associated with single source or critical suppliers; and attendant mitigation plans

5b: A Technology Development Plan

This plan should describe the technology assessment, development, management, and acquisition strategies needed to achieve the project’s objectives.  It describes how the project will assess its technology development requirements, including how the project will evaluate the feasibility, availability, readiness, cost, risk, and benefit of the new technologies. It describes how the project will identify opportunities for leveraging ongoing technology efforts, including technology developed on other NASA projects or programs, at other governmental agencies, or in industry.

The Technology Development Plan also identifies the supply chain needed to manufacture the technology and any costs and risks associated with the transition from development to the manufacturing and production phases.  To accomplish these rather in-depth and detailed goals, the Technology Development Plan typically:

• Describes the project’s strategy for ensuring that there are alternative development paths available if/when technologies do not mature as expected

• Describes how the project will remove technology gaps, including

  •  Maturation, validation, and insertion plans

  •  Performance measurement at quantifiable milestones

  •  Off-ramp decision gates (i.e., the point during development where the project assesses whether the technology is maturing adequately and, if not, decides to terminate continued technology development); and required

• Describes briefly how the project will ensure that all planned technology exchanges, contracts, and partnership agreements comply with all laws and regulations regarding export control and the transfer of sensitive and proprietary information

• Describes the project’s technology utilization and commercialization plan in accordance with the requirements of NPD 7500.2, NASA Innovative Partnerships Project and NPR 7500.1, NASA Technology Commercialization Process

• Describes how the project will transition technologies from the development stage to manufacturing, production, and insertion into the end system

• Identifies any potential costs and risks associated with the transition to manufacturing, production, and insertion; and documents appropriate mitigation plans for the identified risks.

A Technology Development Plan is  a big deal…

Here is a “Technology Development Plan Appendix” (PDF download @ http://go.nasa.gov/2fhx8CO)  published this year for the ‘Exoplanet Exploration Program’ by the Jet Propulsion Laboratory at Cal Tech.