Nexus of Evil

There are a small group of individuals who spend their days and nights trying to find the weak points in NASA’s human spaceflight program.  This diabolical and insidious team has penetrated the most secure sectors of the space agency and they gather their information from the inside.  I personally have confronted this organization and can attest to their manipulative and devious behavior. 

 

My fellow Flight Directors have termed them “the dirty slime ball [expletive deleted]s.”  Their name:  the integrated training team.  Their leader:  Sim sup.

 

Training and simulations have been an integral part of America’s human spaceflight program from the very beginning.  We like to say that we train like we fly, and it is pretty close in many ways.  The job of the training team is to ensure that the astronauts and the flight controllers are prepared for any eventuality.  Not only if things go as planned, but what to do if something goes wrong.  The trainer’s relish their role.

 

When the astronauts are in the simulator, it is as close to the space flight experience as we can make it.   Microgravity can’t be replicated, but almost everything else can be.  When the flight control team is in the Mission Control Center, the data coming in looks just the same whether it is coming from a real life shuttle (or ISS) or from a simulator.

 

The sim team is lead by the Sim Sup (pronounced like “soup”) which stands for Simulation Supervisor.  In the ISS world, they have adopted the moniker STL for Station Training Lead, but the job is the same.  The Sim Sup and his team of trainers think very hard about lessons that the astronauts and flight control team needs to learn.  A lot of these are cataloged and are de rigueur.  Leaks, circuit breaker pops, engines that quit, radios and other electronic gear that flakes out; all of these and many more are standard issue failure scenarios.  A moderately well trained team should be able to handle any single failure without breaking a sweat.  The sim team looks for the optimum combination of problems that lead the flight team to the edge of failure.

 

No kobayashi maru scenarios, though.   Mission operations management stands by the credo “Failure is not an option.”  There is always a way out.  Kirk would be proud.

 

That doesn’t mean the scenarios aren’t tough, however.  During one memorable shuttle launch simulation, I counted 47 different malfunctions that the simulation team inserted into the run in the space of 10 minutes.  When I asked sim sup what was the point of that run, he replied:  “Flight, just wanted the team to learn to prioritize between problems that could kill ya now and stuff that could wait until later.”  Thanks a lot sim sup. 

 

More often than not, the cases were highly cerebral, and it frequently seemed like playing an elaborate chess game with the sim team. 

 

Whatever the flight plan and the objectives, Sim Sup was certain to put together a scenario that would make the team question their assumptions and plans.  That was the point; not just failure response, but is the plan a good one.

 

There is a long history of simulations causing the team to build a better plan that in fact saves the day.  The last landing simulations before the flight of Apollo 11 inserted some LM computer failures which caused the team to abort the landing.  The DPS officer went back to the office determined to avoid that outcome.  When the real LM computer started spitting out alarm codes during the real first lunar landing, DPS was prepared. 

 

Similarly, during an Apollo 12 simulation, the training case required the LM to be used as a lifeboat for a crippled CSM.  This lead to a series of studies and plans about how to improve that capability.  Those plans became the center of the Apollo 13 response.

 

I learned early on never to tell Sim Sup that his case was non-credible.  Every time I complained about some failure scenario, sure enough something like it would come close on the next shuttle flight.  But we were ready.

 

And not all cases were introduced through the computer models running over in the simulator building.  Once Sim Sup snuck out to the MCC and handed the EECOM a note “you are having a heart attack.”  The resulting theatrics by the EECOM and his next door neighbor EGIL caused another flight controller on the other side of the room to call 911.  The EMTs were not amused to find out that they had been scrambled out of the fire station due to a simulation.  MOD management said no more simulated heart attacks in the MCC.

 

Another flight was preparing for an October launch shortly before a Presidential election.  The Sim team called the Flight Director and told him that a candidate was at a campaign stop and wanted to talk with the crew.  That caused a flurry.  But wait; a month or so later, during the actual flight, just a couple of weeks before the election, the phone rang and, guess what?  A certain candidate wanted to talk to the crew while he was at a campaign stop! 

 

There must be a million stories about the complex interlocking training cases that the sim team inflicted on the flight team.  But the key remains that assumptions were questioned, better plans were made, and the team was better prepared for real spaceflight and the problems that Murphy would throw our way.

 

I’ve been reading a lot recently about the financial meltdown and the “quants” that had become so influential in business circles.  They could not believe that their computer models of the financial industry were flawed.  But they were.  I wonder if the financial sector could benefit from Sim Sup? 

 

How valuable would it be to have a Sim Sup for life decisions?  Somebody who could string out the scenario so we got to see how our choices play out.  We could all use that on a personal level; maybe we could use that on a national level. 

Project Management and Innovation

NASA puts on an outstanding training event every year for aspiring Project Managers.  About 1200 folks, both teachers and students, rendezvoused last week near the Kennedy Space Center for an intense two days of classes and panels on how to be a successful project manager.

The fundamentals of project management were firmly reinforced:  have a good plan, stick to requirements, control costs, provide schedule margin in high risk areas, etc.

It got me to thinking about how Project Management 101 could be seen as a barrier to innovation. 

When building something that has never been built before, innovation is critically important.   Innovation at all stages of a project is vital for the end product to be cost effective and carry out its intended function.  But what kind of innovation, and when, and how much?  There is the rub. 

In the barriers to innovation video that I referenced in an earlier blog post, one of the “evil supervisor” stops to innovative ideas was flatly state:  “there is no requirement for this”.   Anybody who has been to project management 101 knows that requirements creep has killed many a worthy project. 

Having a better idea, adding just one more function, tweaking the design through just one more iteration — all these things are wonderful, marvelous, the very lifebreath of a successful project — right up until the point where they kill the project by driving it way over budget, way behind schedule, or into an endless technology development cycle. 

Need a down to earth example?  Ok, but don’t spread this one around or it will get me in real trouble!  My wife came to me several  months ago with the requirement to replace the carpet in our dining room.  Well, the stuff is 20 years old and looks pretty ratty.  So I agreed; we decided on a budget, went shopping at the carpet store.  Our project was to replace the carpet, within a budget, certainly within a schedule (before next Thanksgiving!). 

Then a new requirement popped out:  before changing the carpet the walls should be painted.  Certainly makes sense; fresh paint was needed.  Nobody in their right mind replaces carpet first and paints later.  But adding this new requirement meant that the schedule  stretched out and the budget increased!  But there is more!  It only makes sense to replace the drapes, too.  One shouldn’t put old, dusty drapes back up when the paint is fresh and the carpet is new!  So another new requirement has been added, costs go up, schedule gets stretched out . . . and in the meantime the carpet we liked got discontinued by the factory.  Now, new carpet must be picked, at a higher price . . . .

Congress passed an act a number of years ago which decreed that a project more than a certain percentage over budget or behind schedule should be cancelled.  That is where we are with the dining room.  Got to descope the requirements and try again.

But on the other hand, without appropriate innovation and upgrades, projects may succeed in building something less than what we could.  Something that costs too much to operate, for example, or fails to have an important feature that wasn’t included because of an oversight.

Summarily dismissing any new idea because “there is no requirement for this” is clearly wrong.  Nobody gets the requirements perfectly right the first time,  no matter how hard you try.

So, the art of project management includes listening to proposed innovators and thoroughly evaluating their ideas.  Unfortunately a lot of good ideas get left in the trash.  Not because they were not good ideas, but because at some point, somebody has to draw the line and say this much is good enough; we can’t afford any more. 

That is a conversation that is hard to have.  But it is important.

————————————————

Akin’s laws of spacecraft design:

 #4. Your best design efforts will inevitably wind up being useless in the final design. Learn to live with the disappointment.

#13. Design is based on requirements. There’s no justification for designing something one bit “better” than the requirements dictate.

Yep, I think Dr. Akin is pretty smart.