Nexus of Evil

There are a small group of individuals who spend their days and nights trying to find the weak points in NASA’s human spaceflight program.  This diabolical and insidious team has penetrated the most secure sectors of the space agency and they gather their information from the inside.  I personally have confronted this organization and can attest to their manipulative and devious behavior. 


My fellow Flight Directors have termed them “the dirty slime ball [expletive deleted]s.”  Their name:  the integrated training team.  Their leader:  Sim sup.


Training and simulations have been an integral part of America’s human spaceflight program from the very beginning.  We like to say that we train like we fly, and it is pretty close in many ways.  The job of the training team is to ensure that the astronauts and the flight controllers are prepared for any eventuality.  Not only if things go as planned, but what to do if something goes wrong.  The trainer’s relish their role.


When the astronauts are in the simulator, it is as close to the space flight experience as we can make it.   Microgravity can’t be replicated, but almost everything else can be.  When the flight control team is in the Mission Control Center, the data coming in looks just the same whether it is coming from a real life shuttle (or ISS) or from a simulator.


The sim team is lead by the Sim Sup (pronounced like “soup”) which stands for Simulation Supervisor.  In the ISS world, they have adopted the moniker STL for Station Training Lead, but the job is the same.  The Sim Sup and his team of trainers think very hard about lessons that the astronauts and flight control team needs to learn.  A lot of these are cataloged and are de rigueur.  Leaks, circuit breaker pops, engines that quit, radios and other electronic gear that flakes out; all of these and many more are standard issue failure scenarios.  A moderately well trained team should be able to handle any single failure without breaking a sweat.  The sim team looks for the optimum combination of problems that lead the flight team to the edge of failure.


No kobayashi maru scenarios, though.   Mission operations management stands by the credo “Failure is not an option.”  There is always a way out.  Kirk would be proud.


That doesn’t mean the scenarios aren’t tough, however.  During one memorable shuttle launch simulation, I counted 47 different malfunctions that the simulation team inserted into the run in the space of 10 minutes.  When I asked sim sup what was the point of that run, he replied:  “Flight, just wanted the team to learn to prioritize between problems that could kill ya now and stuff that could wait until later.”  Thanks a lot sim sup. 


More often than not, the cases were highly cerebral, and it frequently seemed like playing an elaborate chess game with the sim team. 


Whatever the flight plan and the objectives, Sim Sup was certain to put together a scenario that would make the team question their assumptions and plans.  That was the point; not just failure response, but is the plan a good one.


There is a long history of simulations causing the team to build a better plan that in fact saves the day.  The last landing simulations before the flight of Apollo 11 inserted some LM computer failures which caused the team to abort the landing.  The DPS officer went back to the office determined to avoid that outcome.  When the real LM computer started spitting out alarm codes during the real first lunar landing, DPS was prepared. 


Similarly, during an Apollo 12 simulation, the training case required the LM to be used as a lifeboat for a crippled CSM.  This lead to a series of studies and plans about how to improve that capability.  Those plans became the center of the Apollo 13 response.


I learned early on never to tell Sim Sup that his case was non-credible.  Every time I complained about some failure scenario, sure enough something like it would come close on the next shuttle flight.  But we were ready.


And not all cases were introduced through the computer models running over in the simulator building.  Once Sim Sup snuck out to the MCC and handed the EECOM a note “you are having a heart attack.”  The resulting theatrics by the EECOM and his next door neighbor EGIL caused another flight controller on the other side of the room to call 911.  The EMTs were not amused to find out that they had been scrambled out of the fire station due to a simulation.  MOD management said no more simulated heart attacks in the MCC.


Another flight was preparing for an October launch shortly before a Presidential election.  The Sim team called the Flight Director and told him that a candidate was at a campaign stop and wanted to talk with the crew.  That caused a flurry.  But wait; a month or so later, during the actual flight, just a couple of weeks before the election, the phone rang and, guess what?  A certain candidate wanted to talk to the crew while he was at a campaign stop! 


There must be a million stories about the complex interlocking training cases that the sim team inflicted on the flight team.  But the key remains that assumptions were questioned, better plans were made, and the team was better prepared for real spaceflight and the problems that Murphy would throw our way.


I’ve been reading a lot recently about the financial meltdown and the “quants” that had become so influential in business circles.  They could not believe that their computer models of the financial industry were flawed.  But they were.  I wonder if the financial sector could benefit from Sim Sup? 


How valuable would it be to have a Sim Sup for life decisions?  Somebody who could string out the scenario so we got to see how our choices play out.  We could all use that on a personal level; maybe we could use that on a national level. 

12 thoughts on “Nexus of Evil”

  1. If it is too strict and prevent everything that can go wrong, it stops everything! Everything is risky. A watermelon rolling downhill is easy, but if you happen to get down there long, began to complicate.

  2. I think about all the jobs Nasa, its contractors, and spin off technology companys have created. You cant clinb a cliff without the same approach. Knowing your equipment, and training for numerous scenerios. Or you may die. Infact much equipment are somehow as a result of Nasa’s Great Acheivments. Thank You. I sure hope Nasa gets what they need to do even greater things. Dont forget the Sim Sups.
    Oh yes, I would’nt be the ameteur astronomer I am without Nasa.
    Never miss the Shuttle or ISS going over. Coming up on 2 billion miles ISS. Lets get that heavy lifter going, and get going!
    Best Regards to Wayne Hale
    Jerry Rowe Marrowstone Island, Wa.

  3. If we hade Sim Sup for all life decisions that would be called The Matrix would it not.

    Years ago I was learning to fly helicopters. I was always initially amazed that we would spend more time on training for something going wrong rather than just flying normally. Anyway one day something did go wrong when I was flying one of my early solo flights. The engine decided to partly fail while I was turning onto the base leg for landing. The helicopter started to shake badly, I thought initially it was the tail rotator about to fail or fall off. There was no panic, no thought of imminent death, my training simply kicked in and ran through all the procedures I could quickly think of and I landed ASAP at the Airport I was doing circuits at safely. Turns out one of the spark plugs on 1 cylinder had failed although its duel backup plug still was working but this plug failure was enough to make the piston engine run rough and shake the helicopter.

    Needless to say after I got home several hours later I then reflected on why we spent so much time training on something going wrong and the what if thoughts started. I did not sleep well that night. I was very glad my Sim Sups had drummed into my head all they could on something going wrong because that day I needed it. I was also very glad their training had helped me to remain calm and analyse the problem, rather than just panic and freeze. Good instructors, teachers, bosses are worth their weight in Gold.

    The problem with the Financial Crisis is simply too much debt, and the wrong kind of debt as per the Great Depression. History is repeating itself and history is a great teacher. No computer model or Sim Sup is needed to answer that one.

  4. As NASA is not going anywhere come Shuttle retirement this year, the simulation teams should be sent into the big banks to find out how prepared they really are.

    I expect there are plenty of training teams in critical services across Federal, State and local governments that could learn much from the NASA sim teams.

  5. Wayne, thanks for the great read. Have you considered writing a book? These stories should be shared with a wider audience.


  6. Many places could do with a SimSup type role. Too often in my own job I’ve been asked to put together testing cases and processes only to have them thrown out as soon as the next major panic hits. When I’ve pointed out that we actually do have a process to follow I get some odd looks and do despair..

    Regarding that heart attack scenario – wasn’t that part of the Apollo training? I recall reading about it in Gene Kranz book. If I recall right his backup took over despite EECOM writhing on the floor. Job done, mission saved and simulation passed.

    I too call for a Wayne Hale book or short story collection!

  7. Hi Wayne,
    You said: “Similarly, during an Apollo 12 simulation, the training case required the LM to be used as a lifeboat for a crippled CSM. This lead to a series of studies and plans about how to improve that capability. Those plans became the center of the Apollo 13 response.”

    As I related in my autobio, the failure of loss of both cryo H2 tanks (loss of all power) occurred during an Apollo 10 sim. The failure was waved off by the FD as not realistic. Other than that your staement is accurate.

  8. Hello Wayne,

    I have not met you yet, but I have known of you for years. I worked in Bldg. 5 for 25 plus years. The old OAS simulator to the SMS FB/MB/GNS simulators. I worked with them for all those years. I was an SMS Systems Test Engineer/Simulation Training Engeneer for about 20 of those years. I proudly did my job to get the Intg. base of the day ready for “RUN” when Sim Sup gave the word. I worked with Tim Terry and Bob Cabana on the STS88 mission trraining and then Brian Duffy and Gail Barnett on STS92. I would have been involved with STS107 and Columbias crew, but I move to a new position at NSS and STS107 was given to some one else.

    I was an honor and priveldge to work for you and the Flight and Training Teams in variuos capacities over the years. Thank you for your part in the Sims and Debriefs too. It all has come out to be to the benifit of the human race. I too liked the Star Trek notables you included too. I think our generation had Gene Rodenberry and Star Trek to help forward the Space Exploration efforts and make us want to reach to the Stars.


  9. Wayne,

    Having worked on the computers and hardware supporting the Space Shuttle and ISS simulators for the last 29 years, it is very heartwarming to hear how useful our efforts have been. I have always felt lucky and proud to be part of the manned space flight program, and your stories about our contributions to its advancement is icing on the cake.

    Thank you very much for sharing your views from the other side with us.


Comments are closed.