Prepared to Launch: Growing up PM at NASA

THE FOURTH OF SIX CRITERIA THAT A NASA PROJECT MANAGER MUST KNOW.

This blog discusses the fourth of six project criteria every Project Manager is held responsible for – and must clearly define for their project(s).  I’m taking these one at a time now, since they get a bit into the more technical side of Project Management.

“You, as a project manager, are called on to make some key decisions, but you are also riding on the top of the 95 percent of the good decisions that were made by the people you delegated to. So, it is a team activity and how you treat and manage those people … makes the real difference.”

 – Human Research Facility Project Manager, JSC

You need to imagine that we’re still doing a stand-up review in front of a LOT of smart people as described in my previous blogs.   People that have managed complex projects for years.   These are people that are experts in risk analysis as well as academics that understand the mission of the project also of course – your bosses are there.   This is a big deal!  It’s your opportunity to show how good a project manager you are – and – to learn things from the audience.  It’s a *very* interactive meeting.

Criteria 4. Adequacy of integrated cost and schedule estimate and funding strategy

I don’t want to get too nerdy here.  But the bottom line is that careful estimates are made, integrated with risk management and tracked with earned value. 

4A: Cost and schedule control plans

Yes, we’re talking about full-blown Earned Value Management!  (see https://evm.nasa.gov/)  Typically, effort uncertainty is modeled using a three-point estimate at the activity or a summary (Work Package) level. The lowest estimated value represents the low extreme of uncertainty, the middle value represents the “most likely” value of the cost or duration, and the high value represents the high extreme of uncertainty.  These estimates are linked to the identified risks for the project to establish a reasonable cost and perhaps schedule reserves.  

“Throughout the execution of the project, the Project Manager shall ensure that the results of all analysis based on EVM are linked to the Risk Management Plan of the Project. Any cost and/or schedule risks being managed by the Project Manger should rely on the results of EVM analysis to track, manage, and mitigate risks.” - NPR 9501.3

You can have reserves, but they must be smartly estimated, reviewed and disclosed.  “Undisclosed reserve” is a bad thing.  Reserves (both cost and schedule) can be handled different ways, but you MUST be consistent in the way they are managed and presented. Below are a few options that are offered as guidance.  

“The PM can make reserve numbers available within the project so all project team members know what reserves are, or the PM can keep reserve numbers quiet.  For example, a PM on a recent successful science mission allowed science instrument teams to have insight into how much reserve he had allocated for each instrument, but reserve was held at the project level. This openness allowed everyone to see the situation but also provided oversight and control” - NPR 9501.3

4B: Basis of Estimate (BoE)

A bottom-up (from the lowest level WBS elements) analysis is a common way to approach this.  Documenting the basis of estimate is often invaluable in the latter phases of the project.  The estimate’s focus should be on:

• Clarity of the objectives
• Thoroughness in the state of the technical and management plan
• Complexity of technology needs

This is exactly what is described in the PMBOK® Guide’s description of a Bottom-Up Estimate “Estimate Activity Resources” in paragraph 6.4.2.4

4C: Cost and schedule estimates consistent with project requirements assumptions, risks, and margins

Probably one of the most interesting item prepared and reviewed is the Joint Cost and Schedule Confidence Level (JCL) of the project.   This isn’t a PMBOK® Guide topic, but it certainly builds on what is addressed in the guide.

NASA’s Project Management requires that projects develop probabilistic risk-informed analyses of cost and schedule estimates to obtain a quantitative measure of the likelihood that the estimate will be met.  Risk analysis provides an analytical basis for establishing defensible cost estimates for likely project risks.  This analysis must be continuously reviewed and updated as more data become available.  A risk analysis, consists of answering the following questions:

• What can happen?
• How likely is it that it will happen?
• If it does happen, what are the consequences?

The cost analysis considers:

• Possible risks, threats, liens, uncertainties, mitigation strategies, and opportunities must be explicitly quantified, including the following:
  • Their probability of occurring
  • Their estimated cost and/or schedule consequences

Risk analysis utilizes modeling, analysis, and evaluation and contains various types of uncertainty. In general, these uncertainties may be attributable to several factors that include

• The statistical nature of data
• The insufficient understanding of physical and biological phenomena
• Unpredictable events (e.g., natural, biological, and human behavior).
• Impacts of cost and schedule performance to date

4D: Joint Cost and Schedule Confidence Level

Joint Cost and Schedule Confidence Level (JCL) is a process that combines a project’s cost, schedule, and risk into a complete picture.  The probability that the project cost will be equal to or less than the targeted cost and that schedule will be equal to or less than the targeted schedule date.   This helps inform management of the likelihood of a project’s success.

Why Do a JCL?

JCL analysis provides a cohesive and holistic picture of the project’s ability to achieve cost and schedule goals by integrating technical, cost, schedule, and risk data.  The project’s JCL can show the impacts of risk to a project as well as highlight the relationship between cost and schedule. This relationship can be extremely important in situations with constrained budgets. A complete JCL analysis also facilitates transparency with stakeholders on expectations and probabilities of meeting those expectations.

What is the official definition of JCL?

JCL is:

• The probability that cost will be equal or less than the targeted cost AND schedule will be equal or less then the targeted schedule date
• A process and product that helps inform management the likelihood of a projects’ programmatic success
• A process that combines a projects’ cost, schedule, and risk into a complete picture

JCL is not a:

• Specific methodology (e.g. resource loaded schedule)
• Product from a specific tool (e.g. @RISK)

The Four Key JCL Inputs:

1. DEVELOP A SUMMARY ANALYSIS SCHEDULE
   • Build a logic network of activities.
   • Utilizing a summary analysis schedule can significantly improve the process.

2. LOAD COST ONTO THE SCHEDULE ACTIVITIES
   • Map cost to the schedule.
   • Cost data can be summarized by a work breakdown structure (WBS) to aid with mapping.

3. INCORPORATE RISK LIST
   • Quantify likelihood of occurrence and impact.
   • Map risks to the appropriate activities.

4. CONDUCT UNCERTAINTY ANALYSIS
   • Apply uncertainty to the schedule and cost.  (The results of a quantitative analysis)

CALCULATE AND VIEW RESULTS

To calculate a JCL, the project combines its cost, schedule, and risk into a single model that can generate a probabilistic assessment of the level of confidence of achieving a specific cost-schedule goal.  Programs must be baselined at a 70 percent probability that the projects will be completed at or below the estimated cost and at or before the projected schedule.

The JCL scatterplot is a standard XY chart with schedule on the X-axis and cost on the Y-axis. Each point is a result of the simulation calculation representing one cost and schedule pair. The JCL calculation is based on the number of results in the target area for both cost and schedule and is expressed as a percentage of all simulation results displayed on the scatterplot.

Establishing a cost and schedule target (black dotted lines) divides the scatterplot into two areas. One area contains results that are at or beneath the target (shown in green). The other area contains results that exceed the target (shown in blue).

The yellow points and frontier line represent all results from the simulation that meet a desired Joint Confidence Level. Multiple points from the simulation may meet the JCL target. Each of the yellow points would establish a new target area large enough to meet the desired JCL

The Joint Cost and Schedule Confidence Level model describes the:

• Basis for base schedule duration and logic
• Basis for baseline cost estimates
• Risks included and basis for probability and consequences
• Risks excluded and why
• Description of the JCL method use

Analyze the scatterplot, run sensitivities, and refine!

Facts and Myths About JCL

MYTH: JCL analysis requires expensive software tools.

FACT: NASA has JCL tools available at no cost to the projects.

MYTH: A JCL requires a detailed resource-loaded schedule.

FACT: Completing a JCL requires only costs, not labor categories and rates.

MYTH: A JCL must be based on a detailed integrated master schedule (IMS).

FACT:  Summary and analysis schedules are preferred!

A full-blown Project Joint Cost and Schedule Confidence Level

“Projects are the means by which NASA explores space, expands scientific knowledge, and performs research on behalf of the nation” – NASA Space Flight Program and Project Management Handbook.  http://go.nasa.gov/1QPkK8Z (Free download)

Project Categorization

Not all projects at NASA are huge.  Many are actually very small.  As the handbook says, projects “Vary in scope and complexity and require varying levels of oversight (notice the key word OVERSIGHT).  They are assigned a category based upon

1. The project life-cycle cost estimate
2. The inclusion of radioactive material
3. If the system being developed is for human space flight

Personally, I never worked with a product of a project that had radioactive material included.  But the other two categories – I have grown up in, work on and managed.  

How are Projects Prioritized?

The first and absolute primary consideration is if the product of the project will be used for HUMAN spaceflight.  That criteria out rules all others.  Then the life-cycle costs are considered.  It’s not the total in-house cost of the project that determines the degree of oversite, it’s the cost of design, acquisition of components, assembly, then putting the project’s product in the field and maintaining it.    A PM’s first project will be small.  But thrilling none the less.  A young PM is subjected to all of the scrutiny and oversight that the most critical projects are.

The Value of Oversight (Standardized Reviews)

As the NASA project standard states: “These reviews are essential elements of conducting, managing, evaluating and approving space flight projects.”  They are a combination of a marvelous learning environment and a way to ensure that each component of the project plan has been reviewed, understood and exposed to scrutiny.

To make sure all aspects of your project are reviewed, a “Standing Review Board” is brought in.  This is a group of independent experts who assess and evaluate project activities, advise project managers and report their evaluations to the responsible organizations.  They conduct independent reviews of a project and providing objective, expert judgments.   

I freely admit that at first, it’s very scary to stand up in front of an auditorium of 100 plus experts and review your project’s “Magic 6” criteria.  But after a while it becomes an adventure.   I’ve told many people, that at first these standardized reviews feel like “Throw stones at Dave” time.  But it wasn’t! Everyone was being constructive, lending their expertise and trying to help all of us succeed.  With a large, experienced and expert audience, you can get concerns expressed that you would NEVER, EVER have thought of as a project team. 

What is the value of these reviews?

The reviews provide the project (team) with an independent assessment of their plans, goals, techniques and risk assessment.  EVERY AREA is open for inspection. 

They provide NASA senior management with an understanding of whether:

• The project is on track to meet objectives
• The project is performing according to plan
• Impediments to project success are addressed

The PMBOK Guide

This type of standardized review is missing from the PMBOK guide (V5), but I wish they weren’t.  A disciplined system of peer reviews, system reviews and reviews by experts may seem expensive, but are a great stakeholder communication tool, a project team communication tool and team-building exercise as well as insuring you’re doing the job to the best of your ability.

NASA doesn't use the word "Portfolio" as PMI does, but divides large efforts into "Mission Directorates." There are:

- Aeronautics Research

• NASA's Aeronautics Research Mission Directorate works to solve the challenges that still exist in our nation's air transportation system: air traffic congestion, safety and environmental impacts.

- Human Exploration and Operations Mission Directorate

• The Human Exploration and Operations Mission Directorate provides the Agency with leadership and management of NASA space operations related to human exploration in and beyond Low-Earth orbit.  (MY PERSONAL FAVORITE!)

- Science

• NASA's Science Mission Directorate and the nation's science community use space observatories to conduct scientific studies of the Earth from space to visit and return samples from other bodies in the solar system, and to peer out into our Galaxy and beyond.

- Space Technology

• The Space Technology Mission Directorate is responsible for developing the crosscutting, pioneering, new technologies and capabilities needed to achieve NASA's current and future missions.

Below the directorates are Programs (such as the Shuttle), then Projects. The projects vary greatly in size and complexity and require different levels of oversight. Each project is assigned a category based upon:

PROJECT AND PROGRAM OVERSIGHT

Oversight came in the form of *REQUIRED* standardized reviews, peer reviews and many others. Most interesting to Project Management are the project life-cycle reviews. These reviews are essential elements of conducting, managing, evaluating, and approving space flight projects 

These reviews were HUGE affairs. Hundreds of attendees would appear to listen to the humble Project Manager present their status. The attendees included a group of independent experts who assess and evaluate project activities, advise projects and report their evaluations to the responsible organizations. They are responsible for conducting independent reviews of a project and providing objective, expert judgments.

These reviews could become very stressful for the PM. I've known people to "crack" under the pressure. But the purpose was not to stress the Project Manager it was to explore every area of the project and point out weaknesses, strengths, risks and more. I think of it as "PM School." After years of standing up to an auditorium-sized audience getting questions from experts in their fields, a PM becomes VERY GOOD at what they do. 

THE PROJECT MANAGER IS RESPONSIBLE FOR PLANNING AND SUPPORTING THESE REVIEWS.

And, must present six assessment criteria for consideration and review. 

More on that later. 

-- Dave