Getting Myself Fired

In 1985 I was a Propulsion Systems Officer in the Space Shuttle Mission Control team.  I was responsible for the reaction control system that was absolutely vital to orient the space shuttle outside the atmosphere, and for the orbital maneuvering system which provides the final push to get the orbiter into orbit and the deorbit burn to come home.  These liquid rocket systems are a mechanical engineer’s delight:  lots of plumbing, valves, some smoke and fire, knowledge of orbital mechanics required, thermal control, crew interaction, and software.  We had a great team and I was proud to be part of most of the early shuttle missions.  But it was time to make a move to supervisor, and in the spring of the year I was selected to be the leader of the INCO group.

INCO stands for Integrated Communications — that’s just about what you think:  radios, recorders, instrumentation, television.  That discipline is an Electrical Engineer’s delight.  Not mine.  I took exactly one EE course in college and nearly flunked it.  But the big bosses said it was OK, I would be a supervisor who knew all the processes and procedures for Mission Control.  I didn’t need to understand the technical stuff, they told me, that was what the staff was for.

That was a lie.

Very quickly I found out that understanding the basics of radios and digital electronics was absolutely mandatory for supervising the INCO team. 

Oh, and the INCOs were responsible for the coffee pots for the MCC — but that story will have to wait for another post.

I decided that I would have to go through the process to be certified as an INCO if I were to lead this group.  This is not easy!  The INCO team is made up of the shift leader in the FCR (who you see on TV) – he is the guy who owns the title “INCO” and responds directly to the Flight Director; then in the “back room” are the support staff:  RF COMM, and INST.  The entry level position was INST.  The Instrumentation Officer is responsible for the onboard telemetry, the signal conditioners, the engineering recorders, etc. 

I understood nothing about any of this stuff.  But the INCO folks recognized that they would have to teach me their job if I were going to be an adequate leader, so they all pitched in.  I read the books, went to the lectures, observed the operations in the MCC, did all my homework.  Then I was ready to start working in the MCC for the integrated simulations.  With the astronaut crew in the space shuttle simulator in building 5, and an entire Mission Control team in building 30, these sessions were intense.  The Simulation Supervisor and his training team were diabolically clever in developing training lessons where interlocking malfunctions could appear insurmountable – but which a good crew and MCC team could overcome.

On my final day as the supervisor of the INCO section, I participated in an integrated Ascent simulation.  We would practice launching the shuttle over and over and over again, with the clock picking up about 2 minutes before liftoff, and as soon as the shuttle cleared the tower malfunction after malfunction appearing in short order.  Most of the time the crew survived.  Sometimes the shuttle even made it to orbit.  But it was intense.  And back in the office, all the Flight Control management is listening to the comm loops to hear how well the team is doing.

It was an artifact of the system that when the simulator starts at T-2, not all the communications system is in the right configuration.  In real life, the INCO team has many hours to command all the various components to the optimum conditions for launch, but in an integrated Ascent sim, there are two minutes to get everything configured properly onboard.  This meant that the INCO, the RF COMM, and the INST were all banging away on their command keyboards furiously to get all the commands sent to the (simulated) shuttle before lift off.  In those ancient days (well before PCs and point&click logic), the consoles had the Multifunction Command and Display Keyboard.  Basically this was a bunch of pushbuttons which had the hexadecimal alphabet on the keys plus one larger key marked “Command Execute.”  You had to know the hexadecimal code for the command you wanted to send; have the dexterity to type it in correctly; confirm on the computer display the code was entered properly; then hit Command Execute for the big mainframe computer on the ground floor of the MCC to send the command.  A fraction of a second later the command would be received at the shuttle (real or simulated) and if everything lined up properly Things Would Happen.  The Right Thing, you hoped.

So with each run, my job was to start the MADS recorder – capturing the “ancillary” data for post flight analysis.  As soon as the simulator went to run at T-2 minutes, I would carefully type in the hexadecimal command for MADS recorder start, verify that code appeared properly on my computer screen, and push the command execute button. 

Ascent simulations are not very interesting to the INST operator because Sim Sup generally targets the bigger systems — main engines fail, external tanks leak, fire breaks out in the cockpit, stuff like that.   Ascent runs take only about 15 minutes, then you debrief, turn the simulator around, and start again.  Many times.  After all day, I got pretty good at starting the MADS recorder.  Ticky tickety tick, execute.  Next run:  ticky tickety, tick, execute.  And repeat. 

On the last run of the day, I punched in the numbers by rote, disregarded the computer screen and hit the execute key.  “WHY DID THE FM TRANSMITTER JUST TURN OFF” echoed in my headset.  “INST – YOU SENT THE WRONG COMMAND!”  Uh oh.  Just one little keystroke wrong.  I was the goat.  

The debrief was not fun. 

When I got back to my office, there was a note on my door from the Division Chief:  “Come see me”. 

As I said, that was my last day as an INCO.  Back in the PROP section the next morning. 

Moral of the story:  Treat each command as if it were your last.  It could be.

 

12 thoughts on “Getting Myself Fired”

  1. Wayne, thanks for the interesting insight into your experiences with NASA. As a citizen-fan of the space program, I got to see you primarily in PAO press briefings, but I enjoy reading your blog postings. It’s interesting to read your perspective via specific glimpses of your career of service.

  2. Hello Wayne

    I used to work in IT (now retired to watch, in part, all things NASA…). Back in the early 1990’s I started a new IT job which in part involved administration of a UNIX based server. I had a number of years experience on UNIX systems and had written many standard UNIX programs to help administer the system (eg deleting tmp files). 2 days after starting my new job I wrote a standard UNIX program to do some basic house keeping on this UNIX server I was responsible for. These programs had worked on all other UNIX servers I had worked on without any problems. On this particular UNIX server it run a propriety brand of UNIX – so I thought I better manually test all my programs first instead of just setting them to run automatically overnight. So when it came to test the script designed to delete tmp files it started to delete every file except tmp files, by the time I realised what was happening (only a few seconds) enough files had disappeared to bring down a database system on the server and half of the operating system. Turns out this version of UNIX had a major flaw in its Operating System that had never been identified and took the manufacture some months to work out. The script I had written was correct but this version of UNIX operating system interpreted it as the exact opposite. Needless to say the next few hours I was in a cold sweat while restoring the deleted files (with the aid of the UNIX server Company emergency call out IT person). So while I had pushed all the right “buttons” they did unexpected things. After reading your article I now wonder if 18 years ago if I had automatically set those “standard” programs without testing (and coming in next morning to find a totally deleted server) if I would have “got myself fired”.

    Cheers

    P. Savio

  3. Wayne,

    Interesting reading. I did not know that you have been selected as INCO section chief before I knew you as section chief of Prop. I was the younger of the “blue-suiters” you had. Thanks for the inspiring blog and thanks for the inspiration you provided directly some 20 years ago.

  4. Wayne:
    You had me worried for a minute there.
    I know nothing about the launch business, nothing about rockets, nothing about building payloads. There’s lots I don’t know, but like you I started reading, listening and found myself inspired by those who pitched in to help me learn. My job, like yours, do the best I can on everything I touch. I agree, it matters every day. You are lucky you inspired people along the way.

  5. “Integrated Communications”…?
    I always thought that it stood for Instrumentation and Communications. Live and learn, I suppose.
    For a time, you were playing in my world, and so I shall share a story…

    In early 1997, Westinghouse sent me to Hartsville, SC, to help commission a pair of 501D5A gas combustion turbines rated at 125 MW each. This was an $80M investment by Carolina Power and Light.

    The WDPF distributed control system had a chess game as part of its program. It was meant as a familiarization tool for the operators.

    One morning, we had both units on the grid, and fully loaded. I had gotten good enough at the chess game to finally have it on the ropes, and when I put it into “check”, both turbines inexplicably shut down!

    Curiously, no alarm condition was logged on the computer, and so a search for the reason began. I didn’t tell anyone that I had beaten the computer at chess, and that “Hal” exacted his revenge by shutting the units down, but 12 years later, I suppose that any statute of limitations should have expired by now…

    Speaking of starting things…I once had a dream where I was a guest at Mission Control, and my responsibility was starting a backup mechanical MET clock.
    I found out by watching NASA TV that they actually used to have a backup mechanical clock…

  6. Thanks for all your insights Wayne, and letting an outsider take a glimpse of the inner sanctum.

    Taking a break from a job search, which is the result of being fired for sending the equivalent of a “wrong command”. At least the person in charge saw it that way. Wish the rest of the world was as cut and dry as a hex word keyed in a console.

  7. You’re fired alright!! You forgot about the Data Comm position Wayne (being one myself – in the distant past – my feelings are just slightly hurt 😉 ).

    Data Comms were orginally part of the manning scheme for STS 1. The Data Comm was the entry level position of the INCO Group. If you recall – the Data Comm was responsible for operating the Ops and Payload data engineering Recorders, the FM RF downlink, amongst other things. They were in a sense the right-arm of the RF COMM (I was ‘abused’ by most since I was a previous ‘Command’ ground systems controller, and I had a better than normal ‘Data Comm’ understanding of the Orbiter Comm and Instrumentation system. From the Data Comm position, one progressed on to INST, and then to RF COMM before becoming an INCO. You never did note that the INCO sat immediately to the left of the Flight Director (as well – the CAPCOM sat immediately to the Flight Directors right). If there was any importance or relevance to that seating arrangement – I would not know off hand….

    Historically – there was a position under INCO in the SKYLAB days to dump SKYLAB data also, but it was not called ‘Data Comm’ as a dedicated position.

  8. The moral of all this is test, test, test – even if the test is only an eyeball test – before committing yourself.

    I love war stories. Experience is such a great teacher.

  9. Another wonderful story, Wayne. Keep ’em coming. This one has more morals (or, perhaps, buried wisdom) than you listed.

    Reminded me of the time that I, the lowly SMS RNDZ Instructor, used the ‘Hand of God’ button to undock Mir (one of those quick-implementation models) in the midst of a Joint International integrated simulation…before double-checking that the SMS Systems Instructor had ‘informed’ the tricky ELCSS model that the hatches were closed. WHHOOOOSSSHHHH. HONK! HONK! HONK! And I’m sure you remember, total cabin depress always crashed the entire simulator…

  10. Dear Wayne,

    A bit off topic:
    Did you get yourself fired? I am kind of addicted to your stories, so 20 days without a new one is hard … ;-).

  11. Hi Mr. Hale,

    Are you planning to write any more articles? I have really been enjoying these and hope this isn’t the end. No need to post this.

    Thanks!

  12. Wayne,

    It’s been almost a month since your last blog entry. Is everything okay? You haven’t gone and gotten yourself fired again, have you?

    Just wanted to let you know that you are missed and hope all is well.

    Regards
    A Friend

Comments are closed.