Real-Life Operations: Day 3

Written Saturday…

Before I begin, I want to let everyone know that our TCM 2 went off on-time tonight and just as planned (23:15 UTC, exactly 24 hours after TCM 1).  Another important goal met as we fine-tune our target trajectory leading up to Lunar Swingby.  Things continue to be a lot of fun, and we’re learning more each day we fly LCROSS. 

It is a rare event when a spacecraft launches and has no problems.  LCROSS is no different than other spacecraft in that regard, though it has certainly tended toward the problem-free end of the spectrum.  A Flight Team hopes for boredom in flight, because boredom can only happen when operating a perfect spacecraft.  The rosy picture I painted of LCROSS’s first two days of operations is largely accurate.  With so many possible failure scenarios to think of before launch, when our spacecraft displayed such good behavior in its first days, I breathed a huge sigh of relief.  We have discovered a few problems, and our team is really excited to work on them, because this is the challenging part, and exactly why we trained so hard.  I would be remiss if I glossed over the problems.  Another mini-novel for Day 3!

So, what are they?

1. Centaur Gas Leak

I tried to find the time to put a full, detailed posting on this prior to launch.  I never found the time.  I can’t wait any longer, since it’s important to my discussions.  This isn’t the kind of problem you can casually mention in a public e-mail, and then not clarify what you mean!  So here goes.

The Centaur upper stage has a distinguished history of successful launches and spacecraft deliveries.  Its missions typically last no more than a few hours – enough time to deliver a spacecraft payload into a target orbit, and then to shut down.  Well, LCROSS came along and proposed to use the Centaur as an impactor – a pretty out-of-the-box idea.  Great!… but at the time we didn’t know which rocket we’d be riding on and then once the Atlas was selected, very little information about Centaur long-duration behavior was known, due to the nature of its standard, short mission.  LCROSS really pushed the bounds on the Centaur design.  As an example, recall that LCROSS separates from Centaur only 10 hours or so before impact?  Well, the Centaur separation mechanism has to function 120 days after launch – unheard of in launch vehicle circles. 

We learned about Centaur’s potential to leak propellant gases out of several possible valves and vents after launch.  For most programs, this is not an issue, because by the time this happens, they’re separated from the Centaur, and unperturbed by the leaks.  The Centaur goes through all sorts of maneuvers to deplete itself of propellants.  Still, it is actually quite difficult to empty it completely. 

A couple of months before launch, we started getting a better picture of possible size of the leaks, what types of torques they might impart, and for how long during the mission they might occur.  The range was anywhere from virtually no effect, to mission-ending.  Basically, a constant leak could, in certain conditions, cause our LCROSS attitude control system to use all its propellant fighting the attitude disturbance.

This became one of LCROSS’s greatest risks, and the Flight Team spent most of its time in the month before launch preparing and testing special attitude control schemes to be able to mitigate the effects of such a leak, some of them significant departures from our normal operating configuration.  This had a lot of people worried, including us. 

So, what did we find in flight?  Well, we certainly display all of the symptoms of having a Centaur leak.  However, they are different than we expected, and overall a lot more benign.  First, following our pre-mission strategy, we have measured the disturbance the leak imparts by temporarily disabling thruster control and collecting attitude data to derive a torque magnitude and direction (assuming fixed mass properties).  Our torque seems to have changed direction more than once.  A discrete event, what we’re calling a “burp”, seems to have caused a dramatic improvement.  Second, we expected one of two scenarios – one where the leak would be very large but would decay quickly as gasses depleted, and another where the leak would have a mild effect, but would be long-acting.  The second was actually a more harmful case, since our attitude controller might have to fight it for months. What we actually seem to have is a benign combination – a relatively mild leak torque, but one that decays pretty rapidly.  We’ve seen noticeable decrease of the effect on each day, and already our spacecraft is almost acting like normal already.  I’ll bet by Lunar Swingby we’ll be done with this one.  I couldn’t have asked for a better leak outcome.

What did we do to fight this one?  Well, we used our pre-flight strategy to save propellant using some tricks of the trade.  First, we increased our minimum “pulse width” or thruster on-time for control, to improve the efficiency of each thruster pulse, thereby saving propellant.  We also tried firing thrusters in pairs rather than in fours.  As I alluded to in my summary of First Week Rehearsal, this ends up being more efficient, but also perturbs our orbit and makes our Navigation team unhappy.  But as it turned out, the efficiency had an unexpected consequence (see #2), and we couldn’t use it for long.  Thankfully, this leak ended up being quite benign. 

2. Cold Thrusters

This one has been a little more tricky. Our thrusters are mounted on four posts mounted circumferentially around the spacecraft, aligned with the axis of symmetry of the Centaur.  Each thruster has a pair of redundant heaters, a set of thermostats to control heating, and a thermistor, which is a temperature sensor that we read in telemetry to monitor thruster valve temperatures.  When the thrusters get cold (for example when they are shadowed from the sun behind other parts of the spacecraft), the thermostats are designed to activate the heaters to warm the thruster.  Well, for LCROSS, we’ve discovered that not all of the thermostats are activating their heaters before the thrusters fall to very low temperatures.  Thrusters are very important to our mission, and so keeping them warm is a big priority. 

Here’s where it gets interesting.  LCROSS is “three-axis stabilized” – it controls its orientation in all three of its axes to keep it in a fixed orientation in space.  As is typical for spacecraft, LCROSS is allowed to rotate a bit in each direction.  Beyond those limits, thrusters will fire to keep the spacecraft from rotating any further.  Inside those limits, the spacecraft is free to move unhindered.  That region is known in control circles as the “deadband”.  A normal spacecraft will tend to bounce from one side of the deadband to the other.  But since LCROSS is under the influence of a Centaur gas leak, it hugs one side of the deadband all the time.  Our deadband keeps our solar array pointed at the sun, and is supposed to keep our thermal conditions “happy”.  Hugging the side of our deadband means that one set of our thrusters tends to be in shadow, and other set tends to be sunlit.  The shadowed thrusters get cold, and the sunlit ones stay warm.  Our normal Cruise State uses an attitude control mode with a 10 degree deadband, so we can drift up to 8 degrees or so on a single axis before a thruster will push us back. 

So, what did our team do?  Our team tried using a smaller deadband so that our leak would only push us over by 1 degree instead of 8.  This caused our thrusters to fire, which warmed them up temporarily, but once we achieved the tighter pointing thruster firings didn’t happen frequently enough to keep the offending thrusters warm, and the sun orientation didn’t seem to help either.  Then we tried a new approach.  We rotated LCROSS about its long axis to expose the cold thrusters to the sun.  This had a nice stabilizing effect on the cold thrusters, but then the thrusters on the opposite side (the ones now pointed away from the sun slightly) began to get too cold, and the mirrored thrusters also did not activate their heaters.  Finally, at the end of tonight’s shift, our team rotated LCROSS in the other direction – a rotation that pointed the top of LCROSS slightly toward the sun (10 degrees), and the Centaur away from the sun.  When I left tonight, this was looking very promising.  The offending thrusters were exposed to sunlight, and they looked like they were keeping warm.  The shadowed thrusters had been demonstrating their automatic heating, and this trend continued in this new orientation.  So, the upper thrusters seemed happy in sunlight, the shadowed thrusters seemed happy under heater control.  And since we have such strong power margin, we can sustain this from a power standpoint.   We still need to experiment more, to make sure other things don’t get too hot or cold, but things are looking up here!

We have a few other issues, but this is all I can afford tonight!  I’ll hold here, and will keep you all posted on how things evolve.  Our team spirits remain high, and we’re looking forward to the problem-solving in the days ahead.  Finally, our team is recognizing the difference between how mission operations feel in simulations, and how they feel for real.  I am confident that we’ll make it through these problems.  We have a lot of smart, determined people, and these problems are not insurmountable.  More tomorrow!

15 thoughts on “Real-Life Operations: Day 3”

  1. Wow thank you for all the detail. Can you please make sure we can watch the LCROSS slingshot video online somewhere, most of us missed it!

  2. Thanks for the write-up. Nicely done.

    Was there any thought pre-flight to thermostats you could command new setting for? Unfortunately command interfaces drive costs up, and it’s a slippery slope as to which controls you want to maintain.

    Any communication impacts from the new attitude?

  3. Thanks for keeping us up-to-date on the mission. I find this to be fascinating reading as an armchair rocket scientist. Keep it up and good luck with the mission!

  4. Excellent!!!
    I cannot thank you enough for keeping us informed on the “behind the scenes” details of the mission.
    Best of luck with the troubleshooting sessions!
    All the world is watching (and cheering you guys!)

    Now go get some sleep 😉 LOL

    Best regards from Portugal to all in the LRO/LCROSS Team.
    Godspeed!

  5. Wow! This is great information.

    Thanks for sharing as much of it as you can.

    Space exploration needs to be exciting, and in order for it to be exciting, people have to be able to get information about it.

  6. Thank you very much for taking the time to write this cool blog, with all the juicy technical details… Keep them coming! I’ll be here every day…

    Best wishes,

    Chris

  7. Great job guys!!!! Nice to see it working pretty much as planned. And you are doing a great job of flying it! I was really worried about the Centaur handoff to LCROSS as well as the Centaur venting. The handoff was much smoother than I expected, particularly given the early discrete. And enough prep work went into the burping so that when the valve leakage aspects arose you folks hand a pretty good handle on how to mitigate any serious effects. Nice to see the worse case scenario did not arise.

    The thruster thermal aspects are unfortunate, but the axial bias seems pretty simple if it holds. I assume sun point survival now includes the bias as well?

    I am writing this the evening after fly-by. The comments out here have been incredibly positive. One comment I heard a couple of hours ago was: “Those LCROSS guys are flying the XXXX out of it!” Keep up the good work!

  8. I hope the folks at NASA know how much the public appreciates the multi-media information and coverage of the space missions. I hope it helps NASAs missions and funding. The effort is appreciated.
    Thanks from Atlanta

  9. Hats off to Paul! I’m on the ops team, and during the first really cold thruster event we were getting dangerously cold – Hydrazine starts to freeze at 1 degree C and we were pushing it. Paul as flight director started the procedure recommended by the engineering team…calmly, quietly, not skipping any steps, briefing the team on prerequisites…just what an edgy engineering team needs in flight directory. One of those momemts when people just doing their job made me think “wow”. Hats of to ya Paul!! It’s been a fun ride so far 🙂

    Emory

  10. Great recovery. I agree about the catch-22. A long boring flight would be preferred but it is nice as an engineer to have at least something to solve.

    I know I speak for a lot of us out here watching the mission and reading your blog, we are infinite sponges when it comes to details. Keep ’em coming!

    One question: What observations are planned while the craft is far from the moon or near earth?

  11. Greeting LCROSS team. I was the project leader for my wife’s 5th grade class in the NASA Quest Challenge, and I’ve been following your blog for weeks now. I just wanted to post letting you know that people actually are reading this, and we appreciate the level of detail you are providing. It makes me feel like I’m there with you.

    I have a question, too. How hard would it be to actually rotate the entire craft slowly so that the thrusters all get equal time facing the sun and don’t get too cold? If you could program your attitude controls to maintain a constant angular velocity with a plus-or-minus “deadband”, it shouldn’t take up any more fuel since the angular velocity wouldn’t be lost because of friction.

  12. MAY THE FORCE BE WITH U….. hoping that ET will not take shooting at its’ moon seriously or whatever….remember the ruskies on mars????

    good luck NASA…

    joe

    old nasa qa late 80’s and AF QA ccafs and vafb….

    rock on in retirement

  13. I have followed the mission with great interest and it was great to see how you overcame the difficulties. One thought that keeps on coming up on this and other missions to the Moon and Mars is this, who is going to clean up the mess we make there. The are bits of spacecraft all over the moon and lots on Mars, not to mention the possible biological contamination.

Comments are closed.