Multi-Tasking: Day 4

First, a note from the Flight Director…my apologies for not keeping up with my writing.  I’ve been extremely busy with flight duties, and am just now getting on top of things again.  This post was written in draft form a couple of days ago, but just now shaped up for posting.  I am enthused by all of your comments and encouragement!  Thank you for all the great notes.  I hope you enjoy this next installment.  More to come soon!

-Paul

The first week of the LCROSS mission plan is ambitious, even without problems to work through.  On Day 3 when we started realizing we had some thermal issues, it was a little daunting.  But now that I think about it, our team had met similar challenges prior to launch.  Over the LCROSS development, our team has had to work around a lot of surprises, under a very tight schedule and budget.  So, in a sense, development was training us for flight in ways we could not have anticipated.

At NASA Ames, after Day 3, Shift A (my shift) transitioned from “execution” to “planning”.  As part of our staffing plan, Shift B exchanged roles with us to take TCM 3, Star Field Cal  (Day 4) and Lunar Swingby (Day 5) execution.  The execution shifts end up being fully committed to commanding maneuvers and other spacecraft activities, so when we started having our cold thruster issue, our Shift A became the de facto shift for coming up with solutions to the problem.  You see, LCROSS was designed to manage itself thermally if we maintained its orientation correctly relative to the sun.  The trouble is that our thrusters are getting too cold, even when we’re sun-pointed. 

Operationally, this means until we design a good thermal control mode, we have a high-maintenance spacecraft.  Our goal was to find a control approach that would enable our spacecraft to manage itself again.  At first we had to change the orientation of the spacecraft every 2-3 hours, sometimes more.  Mixed in with our other numerous activities, this was really difficult to manage.  In collaborations between NASA Ames and our partners at Northrop Grumman, we started experimenting with different orientations, warming one side, warming the other side, entering control deadbands to fire thrusters more often, etc.  No strategy worked for more than a few hours.  Then we found an orientation that was more effective than the rest – where we tipped the “top” of our spacecraft a little bit toward the sun (negative yaw angle).  The thrusters at the bottom of the spacecraft were warmed by heaters, and the ones at the top, whose heaters don’t activate very easily, were warmed by the sun.  On my Day 3 post, we tipped 10 degrees to the sun (-10 deg in yaw), and things looked great.  I came back on Day 4, and found to my disappointment that the thruster temperatures fell too low after several hours.  But, this approach was a great discovery – maybe if we pushed it a little further, things would balance.

For Day 4, we had to improve our thermal performance, and also oversee the planners in creating commands for TCM 3 and Star Field Calibration.  We set to work.  Our first experiment was to nudge our attitude a little further in yaw, to -20 degrees.  Technically, this violated a flight rule that protects our spacecraft against thermal damage.  The LCROSS spacecraft was designed to fly with its solar array pointed directly towards the sun.  Outside the bounds set by our thermal flight rules, the concern was that perhaps the parts of the spacecraft newly exposed to the sun would overheat, and those newly shadowed from the sun would get too cold.  But confronted with an urgent problem, the flight team had to explore options that were outside of the nominal flight envelope.  Our hope was that the thermal subsystem was designed with some conservatism, and that it would allow us to off-point  from the sun just a little bit. 

So, we rotated to -20 degrees yaw, and watched, and waited.  The behavior we observed at -10 degrees on Day 3 improved, and magically, the bad effects we feared might bite us never emerged.  In the short term, the -20 degree yaw bias seemed to do the trick.  Thruster temperatures on the top of the spacecraft really stabilized, and those on the bottom continued to be warmed by their heaters.  From there, our engineering team wanted to explore the thermal sensitivity to these yaw changes, to understand how vulnerable we’d be if, for whatever reason we moved even further in yaw away from directly sun-pointed.  Would we get too hot, or too cold?  We needed to know, so we commanded the spacecraft just a bit further, to -25 degrees. 

Unfortunately we were so focused on thermal issues that we neglected to check the safety of this new attitude from other perspectives.  As soon as LCROSS began rotating, our Star Tracker (STA) “faulted” and dropped into its Standby Mode.  I knew immediately what we had done.  As we pointed our solar array down, the STA, mounted on the opposite site of the spacecraft, pointed more directly to the Earth.  The Earth is bright, and the STA cannot process star fields to determine our orientation if the sun, Earth or moon enter into its field-of-view.  So, our spacecraft reverted to relying only on its inertial reference unit (IRU) for forward-propagating our attitude.  No immediate threat.  Our IRU is quite capable of keeping track of the spacecraft orientation, so none of us were immediately worried.  However, I started worrying about TCM 3.  If, for whatever reason, we couldn’t get the STA re-initialized soon, I wanted to know whether it would suffice for accurately achieving the burn orientation for TCM 3.  As Flight Director, you always have to think ahead to the consequences of your actions.  I was disappointed that we hadn’t anticipated the entry of the Earth into the STA field-of-view, but we needed to proceed calmly and to right the wrong.

Our first action was to return to -15 degrees, to make sure the Earth would not interfere with the STA and to begin the STA recovery process (blinding is temporary, but requires a few minutes).  Of course by this time TCM 3 planning was going on, and the Flight Director on “planning” shifts has to help oversee the creation of new command sequences.  I missed the initial coordination meeting that nails down the precise set of activities we’d be performing later that night, but caught up with the process soon afterward.  Once I saw things were on track, I returned to the Mission Operations Control Room (MOCR) to continue working the STA issue. 

As it turns out, the STA recovery process is a little more involved from our standard Cruise configuration than from our initial Sun Point Mode from Day 1.  In Sun Point Mode, the process to initialize the STA takes only a few minutes.  In Stellar Inertial Mode, one has to re-initialize the spacecraft’s attitude estimator, a piece of software that merges the data from the IRU and STA into a single, optimal estimate of our current orientation.  Not a big deal, really, but time was beginning to get short.  As it turned out, with other spacecraft activities we’d had to perform, the team did not have time to re-initialize the STA before the Command Approval Meeting (CAM) for TCM 3 and Star Field Cal.  This meeting requires the presence of both outgoing and incoming teams, along with the entire planning and analysis teams, and of course our Science and Payload teams.  The meeting is important, since it is the venue where everyone gets their last view of the command sequences before execution, to make sure all is in order.  We suspended active spacecraft operations, and with a skeleton crew monitoring spacecraft health, we held the CAM.  In the CAM, armed with strong evidence that the burn could be safely and accurately conducted without the STA, we resolved that if time for STA recovery ran out prior to the burn, we’d continue with TCM 3 anyway and recover it later.

With the CAM completed, time was critical.  First, Shift A and the oncoming Shift B held a Shift Handover meeting to communicate all that had happened while B was gone, and to make sure Shift B was fully aware of the current state of LCROSS.  With little time to spare, Shift B took over operations in the MOCR, working towards getting the TCM 3 commands loaded in time.  I watched for a while, as Rusty, the Shift B Flight Director, moved to get everything done.  One concern was that our DSN site had had equipment problems on the Shift B from the night before, preventing commanding for a significant period.  Now we were over the same site, and we couldn’t afford an outage.  Things were going well and then bam!   Suddenly, there was a commanding outage, and things ground to a halt.  Never a dull moment!  Watching from the sidelines, I knew Rusty was in a real bind.  The station worked to resolve the issue with its ground equipment, and all of us in the MOCR started sweating.  But Shift B kept from panicking, and within minutes worked with DSN to get the problem resolved.  They really pulled TCM 3 back from the brink.  With only 10 minutes to spare (now too little time to recover the STA), Shift B got the commands into the spacecraft and the ball was rolling again!  I congratulated the team, and headed off to bed, confident that TCM 3 would go off without a hitch. 

It was a very satisfying moment, when the team really came together in a pinch to keep our operations on track.  Great job Shift B!

23 thoughts on “Multi-Tasking: Day 4”

  1. Very impressive operations teams you have there.

    Hats off to ya!

    Congratulations!

  2. What a nailbiter…

    I really appreciate that you talk about the less glamorous moments, too. What I mean is, there probably lies more glory in recovering the spacecraft from the impact of some speck of space dust than recovering it from accidentally self-created pains (like commanding the spacecraft to yaw so that the star tracker gets to see a planet for a change :-))

    This *is* rocket science, problems are complex, fixes may introduce errors (I’m in software development; no further explanation required), things do go wrong, Murphy’s Law sides with entropy, but the team pulls it off, and the crowd goes wild!

    Really, great job and enough material for an exiting movie.

  3. I am sure to put one of these babies on the moon requires alot of effotrs and work, keep on we�ll be there

  4. Thanks for keeping us updated.

    I’m curious about the creation of your command sequences. Are they strictly sequential time based actions? Or is there some amount of logic embedded in them?

  5. I am journalist based in India and I follow the Indian space programme. I find your candour about the problems that cropped up after the LCROSS launch and how those issues were tackled by you and your team thoroughly refreshing. It has served only to increase my respect at the professionalism all of you are displaying and has also provided me a better understanding of what a lunar mission — indeed any spacecraft mission — involves. I wish the Indian space programme would follow your example.

    Just one question. How was the negative yaw turn done — using momentum wheels?

    I look forward to reading more of your posts. Thanks for making time in what must be an absolutely crazy schedule of work and snatching some food and sleep to keep writing.

  6. kudos to all of you…..!

    Paul: hope all is well; I’m closely monitoring the mission from my laptop while on travel in Colombia.

  7. the pictures at the begining are beautifull, very nice pictures

  8. Thanks to God there are people like you to help and guide us…

  9. Thanks for sharing these moments with us. It’s nice to know what goes on behind the scenes.

    I understood, from what you wrote, that the star tracker is more important than the inertial reference unit. Am I right? And if so, why is STA more important than IRU?

    Best wishes.

  10. Excellent operations team indeed. well done to you we salute you!

  11. HELLO NASA, YOU DON’T HAVE THE RIGHT TO CAUSE DESTRUCTION TO A PART OF OUR ATMOSPERE THAT WAS HERE BEFORE YOU WERE EVEN THOUGHT OF. YOU DON’T KNOW WHAT KIND OF CHANGES IT MIGHT CAUSE US ON EARTH. SO WHAT IF THERE IS WATER ON THE MOON YOU DON’T LIVE THERE SO THEREFORE IT SHOULD NOT CONCERN YOU. THIS REALLY IS REDICULOUS. I ENJOY STAR GAZING AND LOOKING FOR THE BEAUTIFUL MOON AT NIGHT TIME AND IN THE DAY TIME. YOU ALREADY WALKED ON THE MOON. PLEASE STOP THE MADNESS BEFORE IT IS TOO LATE FOR THE WHOLE WORLD. NASA YOU DO NOT OWN THE MOON, AND YOU WANT TO CAUSE HARM TO IT. SUPOSE IT WERE TO RELEASE A DEADLY GAS OR MATERIAL IF YOU BLOW IT UP OR EVEN A PIECE OF IT OFF. IF THERE S WATER ON THE MOON IT MIGHT COME FROM EARTHS CONDESATION WE HAVE AN ABUNDANCE OF HUMIDITY. NOW I CAN UNDERSTAND YOU BLOWING UP A ASTEROID BEFORE IT HITS EARTH BECAUSE OF THE HARM IT COULD CAUSE US HUMANS AND ANIMALS THAT RESIDE ON THIS WONDERFUL MYSTERIOUS PLANET, BUT COME ON THE INOCENT MOON. PLEASE JUST LEAVE IT ALONE. I DON’T CARE IF I GET LAUGHED AT, BECAUSE I AM VERY STRICT WHEN IT COMES TO UNNESESSARY DESTRUCTION OR TORTURE FOR THAT MATTER.

  12. ok so it’s remarkable what your doing for the sake of science BUT what are we going to do if you hit it in the right spot and it cracks in half? will it stay in orbit broken into pieces? maybe it won’t happen this time but if you continue to drill and poke at it just like any rock your going to get movement and vibration that can cause disaster. How are you going to explain to my future gran kids that you blew up the moon?

  13. i think that you guys are makeing a mistake wy dont yall just send a team to the moon rather that hit it with a rocket i mean come on dont you guys think of the side effects what if something goes wrong i dont think that i am the only person who thinks that a HUGE WASTE OF MONEY!!! TAX PAYERS MONEY AT THAT the government has money to do that then why dont you all use it to help the economy of people that need it in sted why do yall do whaste money i think that what yall are doing is stupid cool but stupid look im just more worried about the effects of what will happen again what if something goes wrong

  14. The moon is a peaceful and quiet place, which we all like to see at night. I really believe in all the missions of NASA but I firmly disagree with hiting the moon. We should leave it alone because I will like for my sons and my grandsons to appreciate the moon in the future. I think that it is a great idea to be looking for water and other things in different places but, I repeat, I dont like the idea of touching the moon inappropiately. And at the end, after hiting the moon, no one found anything, not even a drop of water…
    Thank You

  15. I have a friend who believes the rocket that impacted the Moon has caused all the strange weather we are having now. She says it might have affected the orbit of the Moon.

  16. I think LCROSS is a guiness mission and that it will beifit all. Going back to the Moon is brillint.

  17. hello dis is neer from india..
    i m intrest on missions of any dangerous activities..
    if u get me a chance i wanna go for a dangerous mission on moon 🙂
    i can do everything .

  18. dear nasa,
    hi, my name is dan.. ever since i was a child I’ve wanted to be a part of the space program. i never really saw an opportunity to contribute untill this experiment, so i thought i’d post.
    first off, really cool experiment… I think it’s awesome that they’re looking for the “basic essentials” on the moon. water, obviously is a basic essential.
    what I wanted to share, was a theory to explain the presence of water. it come all be really rather simple and lucky, so i’m expressing my cautious optimism.. anyway, here it goes..

    DV Theory – An explanation for the discovery of water on the moon..

    My theory is based on the “one in a million” principle. there are thousands of craters on the moon. perhaps the crater we choose for this experiment was cause by the impact of a comet, or a significant piece of a comet. since we know that comets are made primary of ice (solid phase water) and dust, then certainly there would be quite a bit of water in that perticular area / crater.

    Anyway, enough said and I hope i’m wrong. I certainly hope there’s water up there, and more than just water.
    I believe our next step must be to “station up” the moon. We shouldn’t even consider mars, or anything until we can learn more about the nearest celestial neighbor.
    Anyway, much love and congrats to the teams who worked on this. Very cool stuff… Keep it up will ya, and give me a call if you’re looking for some help from passionate young people!!!

    Sincerely,

    Danny V

  19. I think that the moon is the answer in all our questions about the space!!!!!!WELL DONE, MODERN COPERNICUS!!!!!!!!!!

Comments are closed.