LIFE LESSONS: Learned as a Project Manager at NASA #2

 

These  stories are  are real, they happened to me while managing projects at NASA and were a life-changing event.   They affected my personality, my outlook on life and the way my brain works.

A couple I’ve already blogged about, but in the light of “how we managed projects.”  These blogs are a series of how managing projects at NASA changed my life.  I’m sure the same thing has happened to you.  You learn from managing projects and those lessons apply to more than your next project.  They go into your brain and are “compiled” into your personality

BACKGROUND

Early in the Shuttle Program, (the third launch or STS-3) an important mission was for the astronauts to fly Space Shuttle Columbia for its longest stay in space thus far (7 days).  In preparation for that mission, the astronauts planned to spend 40 hours – without stop - in the Shuttle Mission Simulator (SMS) practicing on-orbit operations.

The very large simulator was a mixture of flight hardware and commercial equipment.  The commercial equipment (several very large main frames and 20 or so “Front End Processors” – we’d call them servers today) had the job of creating the environment to make the flight computers believe they were, indeed on-orbit. 

The primary objectives of the mission were to continue testing the "Canadarm" Remote Manipulator System (RMS), and to carry out extensive thermal testing of Columbia by exposing its tail, nose and top to the Sun for varying periods of time.   And, we were running an important and useful simulation using the SMS.  The simulation tested the flight software and hardware and helped the crew establish their procedures, processes and help generally “ring out” the system. There other purposes as well.  Experiments included a plasma diagnostic package, a mono-disperse latex reactor experiment, the first study of insects in space, and many more.  

The Crew in the Simulator During the 7 Day Trial

Everyone had a role in the simulation, Mission Control, Remote tracking stations, experts – I was told there were 800 people world-wide participating in the 40-hour simulation.  Many were “on the loop” or plugged into the audio network we’ve all heard many times on TV.  I had a headset and could hear the network communications.

THE PROBLEM

We wanted to run the simulation for the full 7 days.  But, the computer systems failed just before 40 hours each time.  Not good!  The clever single flight computer voting scheme, didn’t work, the system simply came to a complete HALT – always at about the same time.  Maybe 38 hours, maybe 39 hours…   Emotions flared, crew members were not amused and would leave the building and everyone would yell about something.  This happened about 4 or five hours. Imagine 800 + angry people, all with a headset!

The work-around the first few times was to pause the simulation, set the event “clocks” to 40+ hours and continue.  This worried several of us a great deal.   Why couldn’t get past 40 hours?  Was it the simulator or the flight hardware, the complex simulated environment we invented (the main frames and servers).

AN APPROACH

I built a system (cleverly called “Test data”) so I could monitor the traffic from our main frames to the flight computers.  The idea was to catch whatever was causing the problem.  My little system had probes all over, wires running around to each of the 12+ (large) computers, a disk drive, monitor and a nice hidden spot in the corner of the SMS complex.    For days, I watched the data flow from the main frame to the flight computers. 

After a few more failed simulations, I was convinced I spotted the problem.  (not to get too nerdy, but the main frames were 60-bit single precision words and the servers were 32 bit words.  We had designed hardware to do the “stack and pack” to and from 32 bits and 60 bit words.   I looked like it failed after 30+ hours, but only once!  Once was enough to crash the system.  A real needle in a haystack.

I wanted to stop the simulation right then so I could try and see what caused it.  So…. I did.   I pulled emergency stop.    800 people went off line because young Dave shut down everything. 

GOOD NEWS / BAD NEWS

The good news is that I captured the data and could point to the cause (heat!).  We fixed it easily and within a day or so.   The bad news is that this was one of many of what we used to call “A significant Emotional Event” for me.   I was instantly very, very unpopular with a LOT of folks.  Remember, everyone was already on edge when I pulled the stop.  I imagine people around the world (all the tracking stations) were using my name in short sentences. 

THE AFTERMATH

About a week later, my wife and I were shopping for a new vacuum cleaner at Sears, and one of the crew (pictured above) came up to me and said that I did the right thing. “It was a decisive and bold move well worth the risk.  Thanks Dave!   Later, I was given a hand-made award by the astronaut office for what I did.   It’s very crudely made, and fading now, but it’s my favorite award.  Signed by the crew and John Young (head astronaut at the time).

AWARD FOR REACHING 60 HOURS, 6 MINUTES AND 12 SECONDS.  :) 

THE LIFE LESSON LEARNED

If you believe something is the right thing to do, and the time is right – DO IT.  Don’t wait, don’t hesitate, don’t worry about repercussions.  However, think about your action ahead of time, make sure you’ve considered other options, make sure you know what you’re doing, but DO IT! Have the courage of your convictions.