Brace for Impact! A Schedule of Events for the Final Day

Our last day in flight promises to be the most challenging and the most rewarding for the project.  Our 112 days in orbit are focused entirely on the last four minutes, after the Centaur impacts our target crater and raises a plume of lunar material for the LCROSS Shepherding Spacecraft to observe for signs of water, but before the Shepherd also impacts the moon. 

From a Flight Team perspective, the LCROSS impact sequence is a dream occasion, and yet provides some cause for trepidation.  Many things can go wrong, and with so little time, there is only so much that can be done. 

During you day tomorrow, I thought it might be fun for you to know what the Flight Team will be doing in lead-up to the event.  To put it plainly, we won’t be idle!  Enjoy!

A Recent Development: TCM 9 put LCROSS On-Target

The latest data from our Navigation team indicates that TCM 9 has already put LCROSS on target to hit the designated impact area, without the need for executing TCM 10 on Thursday evening. Our predicted impact point is already within our target 3.5 km diameter circle, and our team will only make very small adjustments to improve our impact accuracy.  This alters our original plan for Thursday.

Instead of performing TCM 10, the team will plan and execute a very slow rotation by feeding the spacecraft new target attitudes each minute (see “Once More Around the Earth” for a description of our “quaternion creep” attitude change) to minimally disturb the current orbit while turning to an orientation that is optimal for Separation.  At Separation, LCROSS will use the velocity imparted by the springs between the LCROSS Shepherding Spacecraft and the Centaur (adding an estimated 15 cm/s to the Centaur) as a final means of nudging the Centaur toward the center of our target.  Analysis of the Centaur separation springs, along with actual tests of the system conducted to simulate very harsh conditions of space (far harsher than LCROSS has actually experienced) indicate the separation will impart a fairly precise change in velocity to the Centaur. 

This plan represents less risk (the slow attitude change will be simpler to plan, test and execute than a TCM), and introduces less uncertainty into the prediction of our impact point (firing thrusters for very short durations adds a lot of uncertainty, while the separation springs in the LCROSS-Centaur interface mechanism are very repeatable).  We’re fortunate to find ourselves in this situation, and we’ll take full advantage of it to ensure we impact on-target.

DOY 281 (October 8): TCM 10 (or not), Separation, Centaur Observation and Braking Burn

The last 24 hours of the mission, bridging DOY 281 and 281 (October 8 and 9), will be a flurry of activity.  Here is the sequence of events.  I’ve provided both UTC and Pacific Daylight Time references:

  • 08:00 UTC/01:00 PDT: Final orbit determination delivery for Separation.  The Navigation team delivers its final orbit determination to the Maneuver Design Team.  This final trajectory estimate will be the basis for planning our slow rotation, Separation and Braking Burn.
  • Maneuver planning and communications link analysis for slow rotation through Braking Burn. For eight hours, the Maneuver Design team will determine the optimal attitude for Separation (when we let go of the Centaur), then plan and double-check plans for the slow rotation, Separation and Braking Burn.  Braking Burn happens after Separation, so will have no influence on the path of the Centaur.  It accomplishes two things: first, it slows our Shepherding Spacecraft with respect to the Centaur, such that at the time of Centaur impact, the spacecraft will be 4 minutes behind it.  This allows LCROSS to observe the Centaur impact while not being too close (risking damage from debris) and not being too far away.  Second, the Braking Burn independently targets the Shepherding Spacecraft impact point, which will be a few kilometers away from the Centaur impact point.  During the same time period, the Communications Link Analyst will refine his estimate of our communications link margin through all phases of the slow rotation, Separation and Braking Burn.
  • 13:00 UTC/06:00 PDT: Separation Activity Selection Review (ASR): Our team knows every last detail of what activities we’ll be running, but this meeting is our last chance to change any part of the command sequence, based on late-breaking data (e.g. changes on the spacecraft, etc).  The Maneuver Design Team and Communications Analysis Teams will present their results here, and will form the basis of command generation.
  • 14:00 UTC/07:00 PDT: Command generation and checking for Separation through Braking Burn. Our Activity Planning and Sequencing Engineer will generate all of the command sequences for TCM 10, Separation, Braking Burn, a preliminary version of Impact, as well as for several contingency cases.  He will hand his products over to both Engineering Analysis, and to the Simulation Engineer, who provide different aspects of quality assurance checks.  Engineering Analysis performs a number of computer-based checking against LCROSS flight rules to make everything in the sequence is legal.  Both the Simulation Engineer and the Engineering Analyst will run the commands on our spacecraft simulator to confirm that they do what we want.  During this time, Shift B (Flight Directory Rusty Hunt’s shift) will hand off to Shift A (my shift).  Shift B will get some sleep, and return before Impact.  Shift A will oversee all of the events through Braking Burn.
  • 19:00 UTC/12:00 PDT: Separation Command Approval Meeting (CAM): This is our final, team-level review of all plans, command products and quality assurance data before executing the slow rotation through Braking Burn.  We’ll make sure everything is correct, over a 90 minute review.  Then we’ll move to our seats in the Mission Operations Control Room (MOCR) to begin execution.
  • 20:30 UTC/13:30 PDT: Command loads for Separation, Centaur Observation and Braking Burn, and slow rotation to Separation attitude. Once “on console”, Shift A’s first priority will be to load the commands for Separation, Centaur Observation and Braking Burn.  We then turn our attention to the slow rotation to the Separation attitude, by loading the burn commands to the spacecraft in an alternate memory bank.  The slow rotation command sequence will re-orient the spacecraft from our Cruise attitude to the Separation attitude.  We’ll confirm the loaded parameters, and then wait for the reorientation to start.
  • ~00:00 UTC/17:00 PDT: Slow rotation to Separation attitude starts.  The maneuver is small, only 6 degrees or so, but will happen in chunks of less than 0.5 degrees each minute.  The onboard command sequence automatically switches over to our Separation, Centaur Observation and Braking Burn command sequence, just in case we lose communications with LCROSS.  In that off-nominal scenario, Separation would still happen without ground-based commanding by our team.
  • 01:40 UTC/18:40 PDT: Separation onboard command sequence starts.  The pre-Separation command sequence starts running.  Ten minutes to Separation.
  • 01:50 UTC/18:50 PDT: Separation.  Commands temporarily disable our ACS, then fire the relays that unlock the Centaur from our spacecraft.  Heavy springs push the Centaur and spacecraft apart at roughly 0.7 m/s, a firm but gentle shove.  The Centaur will accelerate approximately 15 cm/s, but with our optimal orientation, only 3.5 cm/s will be used for Centaur targeting.  After Separation, the ACS is re-activated with an entirely new set of parameters to handle the vastly different mass properties. With the Centaur separated, LCROSS will just have lost 2000 kg of mass.  The spacecraft motion (dynamics) will now behave very differently.  The Flight Team has only 10-15 seconds to confirm that Separation has occurred, and if not, only 50 seconds more to terminate the command sequence to progress any further.  We have practiced this critical timing many, many times.
  • 01:51 UTC/18:51 PDT: Flip to point LCROSS instruments at Centaur.  Just 1 minute 6 seconds after Separation, the onboard command sequence initiates a 180 degree pitch flip to point spacecraft cameras at the departing Centaur.  This takes less than 3 minutes to perform.  The command sequence also powers up the Data Handling Unit (DHU), which powers the science instruments, in preparation for Centaur Observation.  Following the pitch flip, commands roll the spacecraft to optimize the pointing of our omni-directional antenna toward Earth for best downlink rate.  At the end of the pitch maneuver, the Flight Team will re-configure the LCROSS downlink data rate for 256 kbps, and will command the DHU to go to a high-rate camera sampling sequence.  Imagery of the departing Centaur, with the moon in the background, will begin flowing to Earth.
  • 02:01 UTC/19:01 PDT: End of Centaur Observation.  Nineteen minutes after Separation, with the Centaur nearly 800 meters away, the Centaur Observation will terminate.  The Flight Team will reconfigure the communications downlink rate for Braking Burn (64 kbps).  The onboard command sequence automatically re-orients the spacecraft to the final burn attitude, and then squeezes down our attitude control deadband from 3.0 degrees to 0.1 degrees, in preparation for Braking Burn.
  • 02:30 UTC/19:30 PDT: Braking Burn starts.  This burn is longer our last few TCM’s, just over four minutes.  This is because there’s not much time remaining in the mission to build up a 4-minute delay between the Centaur and the Sheperding Spacecraft.  At the end of the burn, the onboard command sequence will re-orient the spacecraft to our Cruise attitude. 
  • ~03:00 UTC/20:00 PDT: Preliminary Impact command load.  As a precaution, the Flight Team will load a preliminary command sequence for Impact to the spacecraft.  If we lost communications with LCROSS sometime after this point, up until the final few minutes, this command sequence should be sufficient to point the LCROSS cameras at the Centaur impact point, run the instruments, and meet all mission objectives. However, before Impact, the team will re-estimate the orbit of the Centaur and Shepherding Spacecraft, and re-plan Impact with the best possible information.
  • 03:30 UTC/20:30 PDT: Shift Handover.  Shift A (my shift) hands control over to Shift B.  Shift B will oversee the Impact event.  We’ll review the status of the spacecraft, in particular the dynamic behavior following Separation, and any last-minute items.
  • 04:30 UTC/21:30 PDT: Final Orbit Determination Delivery. The Navigation team delivers its final estimate of the spacecraft and Centaur orbit. The spacecraft’s orbit can be measured directly, while, without a communications transponder aboard the Centaur, we have no direct measure of the Centaur’s orbit after Separation.  This final orbit determination will become the basis for Impact command sequences, in particular the spacecraft attitude sequence to maintain pointing on the Impact site, and the Impact timing.
  • Final Impact Planning and Command Generation: The Mission Design team will re-plan Impact using the latest orbit data from the Navigation team.  The changes between preliminary and final Impact plan will be very subtle.  The plan involves literally hundreds of Shepherding Spacecraft orientation changes to keep the onboard science instruments pointing at the expected Centaur impact point as we approach the moon.  The new orbit estimate will change all of these orientations very slightly.  The Sequencing Engineer will re-implement the command sequences, then pass his results to the Engineering Analyst and Simulation Engineer for final checking. 
  • 6:30 UTC/23:30 PDT: Disabling LCROSS Fault Management. Shift B will begin configuring LCROSS for the Impact.  One of the first steps is to nearly completely disable the LCROSS onboard fault management system.  Fault Management responds automatically to correct problems it detects onboard.  Sometimes these are benign responses, like switching from a primary sensor to a backup sensor. Other times, the responses can be all-encompassing.  It might seem strange to disable this function right before our most important phase of the mission.  However, the last thing the Flight Team wants is for a problem onboard the spacecraft to interrupt our Impact observations.  Some fault management responses are designed to throw LCROSS into a Survival State, turning off all power to the science payload, and disabling any onboard command sequences.  This could mean disaster for the Science Team, since there would not necessarily be sufficient time to recover and return to the pre-Impact configuration.  So, only minor fault management is enabled, but the more severe responses are disabled.  In preparation for Impact, aside from disabling fault management, Shift B will also coordinate with the Deep Space Network to transfer our downlink path from a 34 meter diameter antenna (DSS-24) to the Goldstone complex’s 70 meter dish (DSS-14).  The 70 meter antenna enables LCROSS to return science data at 1 megabit per second (1 Mbps).
  • 8:30 UTC/01:30 PDT: Impact Command Approval Meeting (CAM).  Shift B will review the final Impact plan and the associated onboard command sequences and ground commanding products.  This is our last chance to get things right.  Since the team is focused on a very specific set of checks, and for lack of time, this CAM lasts only 30 minutes.  Then Shift B goes back to the MOCR to perform Impact.
  • 9:00 UTC/02:00 PDT: Loading Impact command sequence to LCROSS. Shift B loads the final command products to the Shepherding Spacecraft, including a set of contingency command sequences to cover off-nominal scenarios.  In the event of a building fire or an earthquake, our team even has a command sequence that would allow Shift B to leave the building and have the entire Impact sequence and observation be automated.  The Deep Space Network has dedicated four antennas to this period of time, three from the Goldstone complex in California, and a fourth located at Madrid in Spain.  Shift B, with the help of DSN operators at JPL, will coordinate those antennas as LCROSS changes its communications configuration.  Hours earlie
  • 10:00 UTC/03:00 PDT: Start of Impact onboard command sequence.  Its first commands will perform a reorientation of the Shepherding Spacecraft to point the science instruments towards the expected Centaur impact point on the moon.  The cameras and other instruments will not yet be on.  This reorientation will also point the –Z Medium Gain Antenna (MGA) towards the Earth, enabling the team to switch the LCROSS downlink path from the omni-directional antenna to this MGA, in preparation for high-rate science data transmission. 
  • 10:10 UTC/03:10 PDT: Switch to –Z MGA. Shift B will command the switch from omnidirectional to the –Z MGA antenna.  This is a potentially critical step in achieving full-rate science data transmission after the Centaur impact.  However, since we did our combined Cold-Side Bakeout #3/MGA Test on September 24, we’re pretty confident this will work again.
  • 10:15 UTC/3:15 PDT: Transitioning to Science Rate. The Flight Team will now command a transition from a standard downlink data rate of 64 kbps to our full science rate, 1 Mbps.  This is another very important step to achieving full science return.  However, we do have backup procedures that would allow us to transmit science data at a lower rate, 256 kbps, if the DSN 70-meter dish were to fail, or if the MGA was non-functional.
  • 10:36 UTC/3:36 PDT: Payload powers on. The onboard Impact command sequence powers on and enables the DHU and science instruments.  At 10:41 UTC, the command sequence also starts DHU NVM sequence 1, a sequence of instrument commands that tests each instrument in the LCROSS  payload, save the Total Luminescence Photometer (TLP).  The MOCR at NASA Ames begins to receive data from the science instruments, and the Payload Team and Science Team begins analyzing the preliminary data to make sure everything is working.  This is still nearly one hour from Impact, but it’s the team’s last chance to find a problem in our suite of payload instruments that might otherwise foil our Impact observation.  The team continues checking the instruments, and via the Flight Controller and Flight Director, commanding small adjustments to exposure settings, for 35-40 minutes.
  • 11:10 UTC/4:10 PDT: TLP Instrument powers on. The Total Luminescence Photometer (TLP) instrument powers on for the first time since before launch.  This instrument is very sensitive, and can only be powered on a limited number of times.  The Science Team has been very careful not to overuse the instrument in tests.  However, if the instrument powers on as expected, this is a major success on the road to the Impact event.  The TLP, which gathers light measurements at 1000 times per second, will “catch” the Impact flash as the Centaur hits the moon, and is hence a very important instrument for water detection.
  • 11:30:20 UTC/4:30:20 PDT: Flash Mode begins. One minute prior to Centaur impact, the DHU will command NVM command sequence 2, which begins Flash Mode.  For the next 1 minute 3 seconds, Flash Mode will run the TLP and other instruments to capture the flash of light coming from the impact event. 
  • 11:31:20 UTC/4:31:20 PDT: Centaur Impact. Centaur impacts the moon at Cabeus.  The energy of impact emits a brief, intense flash of light.  A plume of lunar debris will rise in a pattern similar in shape to an inverted conical lampshade.
  • 11:31:23 UTC/4:31:23 PDT: Curtain Mode begins. The DHU will switch from Flash Mode to Curtain Mode, which is a sampling sequence optimized to observe the evolution of the debris plume as it rises from the lunar surface.  With this debris rising above the altitude of the Shepherding Spacecraft, our side-looking spectrometer will look towards the sun to measure light as it is transmitted through the debris.  The remainder of the payload will be pointed down towards the impact point. This mode lasts for 3 minutes.
  • 11:34:23 UTC/4:34:23 PDT: Crater Mode begins. At this late stage, the DHU will now switch from Curtain Mode to Crater Mode, which is designed to capture data about the properties of the new crater generated by the Centaur impact.  The Shepherding Spacecraft now has less than one minute of time to capture and transmit data before it also hits the moon.  With the Centaur impact point now off to the side, LCROSS will continue to try and track that point until its own contact with the moon.
  • 11:35:39 UTC/4:35:39 PDT: Shepherding Spacecraft impact. The Shepherding Spacecraft will also hit the moon at roughly this time.  The Flight Team will abruptly stop receiving telemetry a few seconds later, as the photons from LCROSS’s last transmission travel back to Earth to be received by the DSN 70 meter antenna.  The LCROSS flight mission will be over.

This will be my last post until after Impact.  I hope you enjoy the show tomorrow – it should be very exciting.  Though we won’t have immediate feedback for water detection, I hope to report good news to you on Friday regarding the accuracy of our impact, and the collection of the science data.  Then, over the coming weeks after Impact, the Science Team will review their data and interpret the observations. I’m sure you’ll be hearing news one way or the other.

Thanks for reading!

Farewell LCROSS

Saying Goodbye to a Really Amazing Spacecraft (and Team)

Well, we all knew it was going to happen.  It was inevitable.  It was the whole design of the mission.  LCROSS was destined to end its wonderfully fantastic journey by intentionally crashing into a permanently shadowed crater at the south pole of the Moon.  We are the ones who devised this fate for LCROSS.  So why should we be surprised (and just a little bit sad) now that the time has finally come?

As a proud member of the LCROSS Science Team and as the Observation Campaign Coordinator, I would have to say that working on this mission has been one of the highlights of my career thus far.  The mission itself is truly amazing (We’re impacting the Moon!  We’re looking to see if there’s water ice at the poles!  We’re going to this utterly unexplored place in our Solar System, so close to home, and are so excited about what we’re yet to learn!).  LCROSS is so important to both science and exploration.  This mission is blazing a new path in how to build small, robust spacecraft both on schedule and on budget.  LCROSS uses eight (yes eight!) commercial off-the-shelf instruments for its payload – also a very novel way for NASA to get more bang for the buck as well as good science to boot.  The technical aspects of the LCROSS mission are astounding, but none of this would be possible without the dedication of the *people* working on this project.

The LCROSS Team is made up of an amazing cadre of individuals.  LCROSS has a relatively lean and nimble team.  There’s still a lot of work to be done to send a spacecraft to the Moon, and so that means that everyone has to pull together to make things happen.  If somebody is extra busy and needs help, you help them.  If there’s something that needs to be done and you’ve never done it before, you figure out how to do it.  If you are stuck and need some assistance, just ask your teammates and without hesitation people are willing to help.  We all naturally come together to get the job done.  There is a high level of trust and commitment on this team, starting with the top Project management and all the way through the people working the nitty-gritty technical aspects.  It is truly a glorious experience to work with a team such as this.  The best part is that everyone is working towards a common goal, and everyone is willing and able to contribute in whatever way is needed in order to achieve the objective.  It is amazing what a group of people can do when presented with a fascinating project and an exciting challenge.

And it’s not just the Project folks who have helped make this happen, but it’s all of the students and members of the general public who have so substantially contributed to the successes of LCROSS.  Student interns at NASA Ames have had the opportunity to work with real honest-to-goodness flight hardware.  Not everyone has the opportunity in college to hold an instrument that will be on the Moon within the next year!  Such opportunities are tremendously powerful for encouraging the students of today to continue the pursuit of careers in math, science, and engineering.  Amateur astronomers from around the world have been imaging LCROSS in the night sky during its trip to the Moon and are planning to collect observations of the impacts as well.  This is a great way to actively participate in a NASA mission.  We’ve also been having a great time keeping folks updated regarding LCROSS activities through our NASA website as well as the LCROSS Facebook and Twitter accounts.  Thousands of people are following LCROSS on these sites and we’re thrilled to be able to have a two-way dialog to discuss all things lunar!

So, although this mission was destined to end in a spectacular grande finale culminating with two lunar impacts, it is a bit sad to see this phase of the project come to a close.  However, next up is the exciting analysis of the data to try and learn all we can about these enigmatic regions on our very own Moon.  And here’s to hoping there are lots more missions coming up in the future, because we are fired up and ready to go!

Our Centaur: Launch Vehicle Upper Stage Turned Lunar Impactor

The LCROSS Mission Design Team included the Centaur as a critical component of their mission from the very beginning.  It’s essentially a bonus impactor that stays with LCROSS after performing it’s normal function of delivering LRO and LCROSS on their way to the moon.  For the LRO-LCROSS launch and particularly for its contribution to the LCROSS mission, the Centaur was the focus of extra attention from both the LCROSS team and from the KSC/ULA-Atlas launch team.

Think of it as energy recycling.  The kinetic energy  [1/2 MV2] provided by the launch vehicle to get LCROSS and Centaur to lunar swing-by are later used to create the ejecta plume at the moon’s south pole.

Impacting a 2-ton Centaur upper stage enables the LCROSS mission to significantly increase the amount of lunar ejecta expelled from the target crater over what could have been achieved using only the spacecraft.

Centaur is a cryogenic (liquid hydrogen and liquid oxygen) powered rocket  upper stage. Its engine can be shut-down and restarted after one or more guided coast phases.  Centaur has a long history of successful  missions.  For LRO-LCROSS, instead of being parked in a safe orbit after sending LRO and LCROSS toward the moon, this is Centaur’s first role as a guided impactor for the science gathering  portion of a mission.

The Centaur upper stage used for the LCROSS mission is the same as Centaurs used to propel the Mars Reconnaissance Orbiter and Pluto-New Horizons on their way to Mars and Pluto, respectively.  The only significant modification to this Centaur is the addition of white thermal paint to help balance the temperature of the empty stage. 


Upper Photo: The LCROSS logo, painted on the Centaur.  Lower Photo: The Centaur upper stage is lifted for mating to the Atlas-V first stage (Tail Number 020) at the Vertical Integration Facility in Florida (April 30, 2009). The Centaur was painted white to help manage the thermal environment during the 100-plus day LCROSS mission.  Yellow lifting hardware is ground handling equipment and is removed after stacking on the first (booster) stage.  Courtesy of NASA Kennedy Space Center.

What is particularly unique about this Centaur was the way that the United Launch Alliance (ULA) Atlas Team designed the flight profile and maneuvers after LRO separation to minimize the amount of residual hydrogen, oxygen, hydrazine, and helium left on Centaur.  This special care to ensure that the Centaur is as empty is possible, makes the Centaur very clean. It reduces the hydrogen (H2), oxygen (O2), and hydroxyl (OH) species that could be confused with in-situ water ice ejecta.

Since Centaur would be attached to LCROSS until just before impact, all of the Centaur hardware and functional system were analyzed and scrutinized and if necessary tested to ensure that they would not adversely affect LCROSS after LCROSS took over control from Centaur.

The final function to liberate Centaur as a guided impactor will be the release of the clamp band holding the LCROSS/Centaur stack together. This separation command will be sent by LCROSS approximately 9 ½ hours before impact.  The clamp-band and the separation system that will push LCROSS and Centaur apart, underwent long-duration testing to give the Mission team confidence that it will function smoothly after 100-plus days in orbit.

Additional Centaur info and history, courtesy of NASA’s Glenn Research Center, can be found at: https://www.nasa.gov/centers/glenn/about/history/centaur.html

Once More Around the Earth: September 4 – October 5

The anomaly robbed the LCROSS Flight Team of precious time to prepare for Impact.  But with a healthy spacecraft, and enough propellant to do the job, our team was all too happy to prepare for the future.   Ahead of us were five more Trajectory Correction Maneuvers (TCM 6 – 10) to precisely refine our crater targeting, then Separation, Centaur Observation, Braking Burn, and finally, Impact.  In the midst of our TCM series, the Science Team continued refining their selection our target – the specific crater, and the point within the crater – based on the latest data from other missions.  As a final confirmation of the payload instruments, the Science Team also wanted to look at Earth one last time before Impact.  On top of all that, we needed to practice those final two critical days as much as we could.

DOY 247-251 (September 4 – 8): Housekeeping

Out of Emergency Status, we resumed operations gradually, monitoring spacecraft health and performing typical housekeeping duties.  Of special note, as a result of the anomaly and our very small propellant margin, we decided to substitute Earth Look Cal 2, originally scheduled for DOY 250 (September 7), with a new, more propellant-efficient calibration maneuver that we termed “Earth Gaze”, on DOY 261 (September 18; see below).   


DOY 252 (September 9): TCM 6 Waived

We received more good news about our trajectory.  Despite the long train of thruster pulses that resulted from our anomaly, LCROSS’s orbit was still right on target.  The Mission Design Team and Navigation Team agreed that TCM 6 would be too small to be worthwhile.  This meant one less event to attend to in our busy schedule, and one less reason to take risks at this late, sensitive stage in the mission.

This decision worked in our favor for saving propellant too.  Each TCM typically requires the spacecraft to reorient to a specific direction so that LCROSS can add velocity in a particular direction via the 22 N Delta-V thrusters, and then back again to the Cruise attitude.  All of our remaining “burns” were expected to be small; so much so that the propellant devoted to the reorientation maneuvers was expected to be several times more than for the burn itself.  Waiving TCM 6 meant we could save more propellant, and reduce risk.

DOY 253 (September 10): Omni Pitch “Creep”

Throughout the mission, Omni Pitch maneuvers have been necessary to keep our primary antenna pointed towards Earth.  In all previous Omni Pitch renditions, the “slew” was performed in SIM DB2 (Stellar Inertial Mode, deadband 2), an Attitude Control System mode that enables relatively efficient attitude changes.  However, SIM DB2 significantly accelerates the spacecraft rotation to induce the change in orientation, and this costs propellant. 

In our bid to save propellant, we re-examined all of our previous maneuver approaches for opportunities to be more efficient, including for Omni Pitch maneuvers.  Someone on the Flight Team came up with the idea of providing LCROSS a long sequence of small attitude changes rather than one big one, with the goal of avoiding the accelerations of SIM DB2 (small attitude changes don’t justify large rotational accelerations, and therefore are cheaper propellant-wise).  Simulations of this new approach, what we informally called “Quaternion Creep”, indicated this would lead to a significant propellant savings, so we baselined this approach for our Omni Pitch maneuver on DOY 253. 

Normal Omni Pitch maneuvers used to take 40+ minutes to execute.  This new approach took 4 hours 20 minutes, but saved a lot of propellant.  Well worth the bargain.

DOY 257 (September 14): Guarding Against Single-Event Upsets

Cosmic rays are high-energy photons that are known to interfere with spacecraft computers by in effect reprogramming single bits of memory as they pass through.  When a bit in computer memory is reprogrammed by radiation, but not permanently damaged, it is called a Single-Event Upset (SEU).  Most spacecraft computers and onboard memory chips are “radiation-hardened” to prevent or otherwise sidestep the effects of cosmic rays. 

The spacecraft Data Handling Unit (DHU) is the independent computer that controls the operation of the science instrument payload.  Command sequences onboard the DHU control all of the sampling sequences and instrument settings to make sure LCROSS collects the best possible set of data at impact, and for other calibration events.  They are clearly very important to the success of LCROSS.

The DHU, and those instrument command sequences specifically, are somewhat vulnerable to SEU, and the Payload Team had not checked the contents of the DHU in a long time.  On this day, our team dumped (downloaded) the full contents of DHU non-volatile memory and confirmed that the sequences were still intact. 

DOY 257 – 259 (September 14 – 16): Full Rehearsal of TCM 10 through Impact

It was hard to find time in our busy operational schedule to practice all of our pre-Impact procedures.  In the months before launch, we spent all of our time readying for Transfer Phase and the first parts of Cruise Phase.  Impact seemed so far away.  Then Cruise Phase, with the anomaly, was far busier than expected.  With only weeks remaining, our perspective was entirely different.   

We took full advantage of two free days in our schedule to hold the “Last Two Days Rehearsal”, a full-team, high-fidelity rehearsal of the last two days of the mission, similar in style to our First Week Rehearsal (see the post on “First Week Rehearsal”).  For 42 hours, following the exact schedule we planned to follow in flight, and synchronized with the actual times of day when events would occur, the Flight Team practiced nearly every aspect of those operations.  We ran 24 hours a day, with three overlapping execution shifts (Shift B, then A, then B again), and two planning sessions.  In the rehearsal, the team successfully hit the target crater and collected all of the science data.  However, the there were some procedural and process shortcomings that made things just a bit rocky at times.  We learned a lot, and began working improvements in the days after the test. 

DOY 260 – 261 (September 17 – 18): Our Last Gaze at Earth

In substitution for Earth Look Cal 2, we conceived of a more propellant-efficient calibration event we coined, for lack of something more official, “Earth Gaze Cal”.  We loaded the commands for this event on DOY 260 (September 17), and executed the calibration the following day.  Rather than sweep back and forth over the Earth (see the full description of our first Earth Look in “During Our Second Trip Around Earth”), Earth Gaze looked straight at the Earth for an extended period, during which time we collected camera and spectrometer data.  This was the Science Team’s last chance to evaluate instrument performance and settings before Impact. 

We captured some more images.  Here are a few, along with a simulation to show the orientation of Earth during the event.


DOY 267 (September 24): Cold Side Bakeout #3 and a Test of the –Z MGA

Despite previous efforts to rid the Centaur outer skin of water on Cold Side Bakeout #1 and #2, our Navigation team continued to observe the accelerating effects of escaping water at the end of the second of those events.  With the amounts of water remaining, the Science Team was no longer concerned that this water could interfere with water measurements at Impact – there was just too little left.  However, Navigation was still concerned that remaining water might push our Centaur off course in the hours before Impact, after Separation when we no longer had any control over its orbit. 

Recall that we had planned to execute Cold Side Bakeout #3 on DOY 234 (August 22), but our plans were thwarted by the discovery of the anomaly.  Cold Side Bakeout #3 was unfinished business that had to be completed. 

In an unrelated thread, we also wanted to test the antenna we’d be using for Impact.  LCROSS has two Medium Gain Antennas (MGA’s), one on the +Z axis, the other on the –Z axis, used to downlink high-speed science data to Earth.  We had used the +Z MGA during Lunar Swingby, but had never tested the –Z MGA in flight.  We didn’t want to discover a problem with this antenna in the hours before impact, so we devised a test that would expose any issues immediately, and that would couple very nicely with Cold Side Bakeout #3. 

The combined Cold Side Bakeout #3 and –Z MGA Test took advantage of the fact that LCROSS was passing right through the ecliptic plane, the plane of the Earth’s orbit around the sun.  The LCROSS +X axis was perpendicular to the ecliptic plane at the time, and so by rotating about the +X (roll) axis, we could simultaneously face the “cold side” of the Centaur towards the sun, and the –Z MGA towards the Earth (required to test communications via this antenna).  It was a perfectly-timed opportunity.

On DOY 267 (September 24), we performed the maneuver.  Unlike previous versions of Cold Side Bakeout, we stopped the spacecraft twice, once at 135 degrees rotation, and again at 225 degrees.  The first position pointed the –Z MGA (which is canted by 45 degrees) straight at the Earth, and warmed one side of the cold skin of the Centaur.  After 20 minutes, LCROSS rotated another 90 degrees, moving the MGA off the Earth, and moving another part of the cold face of the Centaur into full sunlight.  We characterized one “slice” of the –Z MGA antenna gain pattern, confirmed that it was operational and mounted according to specification, and removed more water from our impactor.

One of the risks of Cold Side Bakeouts is that we might induce a thermal instability in our thrusters, as we had in Cold Side Bakeout #1, prompting our improvised fault management to fire the thrusters to keep them warm (see the post entitled “Our First Orbit Around the Earth” for details).  Happily, our thrusters remained thermally stable, and we avoided any additional propellant cost.

DOY 268 (September 25): TCM 7

LCROSS had not performed a Trajectory Correction Maneuver since TCM 5a, way back on July 21.  We’d been literally coasting along our orbit since then.  By this time, the Science Team had selected a satellite of the Cabeus crater, Cabeus A1, as our impact target.  Trajectory Correction Maneuver 7 (TCM 7) was performed on September 25 to target that location on Impact Day.  According to predictions on burn pointing errors, TCM 7 would guarantee a hit within 38 km of our intended target.  Future burns would reduce the error a lot further.  Still 14 days from Impact, TCM 7 nudged LCROSS by 32 cm/s (just over 1 foot/sec), but enough to make a big difference in impact position.

DOY 273 (September 30): TCM 8

Following TCM 7, the Science Team continued to receive data from other lunar missions.  Lunar Reconnaissance Orbiter (LRO), our partner at liftoff, devoted much of its operational time to scouring our top target regions, and data from Chandrayaan-1 and other missions continued to improve our knowledge of the mapping of possible water concentrations at the lunar south pole.  New information gathered after TCM 7 prompted the Science Team to redirect LCROSS to the main Cabeus crater.  Relatively close to the previous target, changing represented a small amount of extra propellant, but a potentially significant improvement in science data. 

TCM 8 was performed on September 30, and changed LCROSS’s velocity by 35 cm/s.  Not only did TCM 8 target the new impact position, but it also refined our impact timing.  Our narrow target time window will allow the Hubble Space Telescope to take images of our impact plume.

DOY 274 (October 1): Impact Contingency Rehearsal

Our Last Two Days Rehearsal on DOY 258-259 did not emphasize off-nominal events, but rather the full integration of the team in a single push.  On October 1, Shift B, who will be overseeing Impact, practiced two complete run-throughs of the final two hours of the mission.  I helped our Test Conductor in setting up the tests, and planning a series of anomalies that would really challenge Shift B.  During the simulations, we failed banks of temperature sensors and heater circuits, caused various instruments to malfunction and the DHU to crash.  We faulted the IRU, forcing a recovery in the midst of early pre-Impact science operations.  We failed the –Z MGA switch (even though this is far less likely having conducted our MGA test), and forced the team to perform the Impact sequence at a contingency low data rate.  Their team did very well, and succeeded in meeting mission objectives on both runs.  The members of Shift B were very happy they had that final opportunity to practice.

DOY 275 (October 2): Planning Rehearsal

The flight planning team has one of the biggest challenges in the final 24 hours before Impact.  Based on a final assessment of the LCROSS orbit, they have to plan the final series of maneuvers that culminate in the Impact event: TCM 10, Separation, Centaur Observation, Braking Burn, and Impact itself.  From the maneuver plan, the team must generate command sequences for each of these events, and then run them on our simulator to prove that they are flight-worthy.  Furthermore, they also need to generate and test command products to handle specific contingency events, like a failed Separation, or a late Braking Burn.  They even generate a preliminary Impact command sequence, just in case we lose contact with LCROSS for hours before the event.  Some of these products can be produced well in advance, but a great number of them depend on the final orbit assessment, and the specific orientations that are required for TCM 10, Separation and Braking Burn to ensure we accurately hit our target.

The Last Two Days Rehearsal proved that the planning team could generate all of its products, but could not quite perform its full list of quality assurance tasks in time.  Following the rehearsal, the team took some significant steps to streamline the planning, command generation and quality assurance process.  We found opportunities for working in parallel, for skipping unnecessary steps, for nailing down some variables in advance to reduce the amount of variance on Impact Day. 

To test these changes, we conducted a Planning Rehearsal for DOY 275 (October 2) to span the full 12 hours of the process, exactly as it will happen on the Impact Day.  I was really happy with the results.  The team actually finished most of its tasks early, including simulations of all command products, and their delivered products contained no errors.  The Command Approval Meeting, our last visual evaluation of commands, finished early.  I was very encouraged by these results, and the planning team is now fully confident that they could repeat this level of performance on Impact Day.

DOY 278: Final DHU NVM Sequence Loads and TCM 9

The remainder of this post has to do with current and future events.  I’m finally caught up!

This morning, as I write this post, Shift B is loading its final set of science payload command sequences to the spacecraft.  The Science and Payload Teams have pored over the data we collected from all previous science calibrations to derive a set of camera exposure/gain settings for Impact.  Over-exposure means washed-out images, which under-exposure means less contrast and detail.  Six of nine sequences will be changed to reflect the Science Team’s best knowledge of camera performance, and expected conditions on our final descent to the lunar surface.

Later this afternoon, Shift A will be performing TCM 9, our second-to-last, and possibly our last orbit adjustment.  After TCM 8, based on the latest lunar mapping data, the Science Team selected a different spot within the Cabeus crater.  This is our final target.  With only 7.6 seconds of firing, TCM 9 moves our impact point from our old point in Cabeus to this new location roughly 9 km away.  TCM 9 should put us within 1.75 km of our target.  If we can “nail” this maneuver by staying within that range, we’ll be able to skip TCM 10. 

TCM 9 is our last spacecraft activity, other than monitoring and housekeeping, until our final rapid-paced series of maneuvers in the final 24 hours.

As you can imagine, things are starting to get very busy!  I will submit one more post with a detailed description of the final 24 hours.  I also have a few guest authors who would like to share some things with all of you before Impact.  Stay tuned…this is going to be an incredible week.

A Test of the Flight Team: The Near-Loss and Full Recovery of LCROSS

Early in the second half of our second Earth orbit, while out of DSN contact, LCROSS experienced an anomaly.  An error detected with our Inertial Reference Unit (IRU) resulted in an automatic response that consumed a large amount of propellant in a short amount of time.  We spent the remainder of our second orbit recovering from that anomaly, and protecting against any future excessive propellant usage.  We emerged safely, but with so little propellant remaining that, since that event, we’ve had to step very carefully to avoid wasting any more of this precious resource.  This posting describes our discovery of the problem, and our recovery steps to protect LCROSS against a reoccurrence.

My intent with this blog entry is to relate the facts as we know them, to demonstrate what an anomaly is, through a real example, and to show how our Flight Team responded.  My hope is that you come away with a better understanding of the challenges of space flight.

August 21/22 (DOY 234): Anomaly Detection, Spacecraft Safing, and Entry to Emergency Status

Shift A came into the Mission Operations Control Room (MOCR) on Saturday with a big agenda on LCROSS – to execute a third “Cold Side Bakeout” to rid the Centaur of more ice trapped in its skin, and to perform an Omni Pitch maneuver to flip the spacecraft to re-point the primary omni-directional antenna at the Earth, all under a tight schedule with little margin for error.  The shift started at 2:25 AM Pacific time, but despite that early hour, we were excited to have such a challenging pass ahead of us, and eager to get started.

We acquired spacecraft telemetry at 3:25 AM.  In the MOCR, each Flight Team operator sits behind a set of computer monitors that display fields of telemetry data – numbers, status indicators, etc. that indicate the health of LCROSS.  Many of the telemetry data fields show up in stoplight colors – green, yellow, or red, to indicate whether the value is “nominal” (green), or approaching an emergency state (yellow), or is in an emergency state (red).  Typically, telemetry comes up entirely green. 

That day, LCROSS came up with lots of yellow and red alarms.  We had never seen anything like this in flight.  The summary of our observations:


  • The Inertial Reference Unit (IRU), our onboard gyro, and primary means of measuring rotation rates around each axis for attitude control, had faulted;
  • The Star Tracker (STA) had replaced the IRU in providing rotation rate estimates to the attitude controller;
  • Spacecraft rotation rates were periodically exceeding yellow high alarm limits (rotating too fast);
  • LCROSS was firing thrusters almost continuously,
  • The propellant tank pressure sensor (our primary means of determining how much fuel we have left), indicated we had consumed a LOT of propellant.

In short, we were in serious trouble, and needed to make corrections immediately.

First, we needed to save our remaining propellant by reducing our thruster firings.  LCROSS has an Attitude Control System (ACS) to control its orientation automatically, by firing thrusters (LCROSS has no reaction wheels).  The ACS controls each of the axes (roll, pitch and yaw; see the diagram in “First Orbit  Around the Earth”) to within an acceptable error bound, defined by a control “deadband” from a fixed orientation in space.  The ACS has several modes, all with different characteristics.  The mode we were in, “Stellar Inertial Mode, deadband 2” (or “SIM DB2” for short), controlled each axis to a +/- 0.5 degree deadband.  Through the anomaly, LCROSS was keeping to within its assigned deadband.  Under attitude control with a deadband, thrusters typically fire most often when the spacecraft rotates to the very edge of the deadband, causing the spacecraft to stop its rotation and preventing it from exceeding the deadband limit.  So, in an effort to slow down the firings (and our propellant consumption), we switched to a wider deadband, SIM DB3, which controls to a 10 degree range.  Unfortunately, this didn’t reduce our thruster firing rate very much.  We needed to do something else.

The anomaly had caused LCROSS to automatically switch from using the IRU to the Star Tracker (STA) for measuring spacecraft rotation rates. However, when we studied the output of the IRU, it was producing good rotation rate data (or just “rate data” for short).  So, we ran through the IRU recovery procedure to re-engage the IRU with attitude control, and sure enough, that finally returned the thruster firing profile to normal and bought ourselves time to think.

One thing was immediately clear – we had suffered a significant “anomaly”, or bad problem and our plans for a Cold Side Bakeout and Omni Pitch maneuver were dashed.  We’d be spending the rest of the shift, at least, and possibly a lot longer, figuring out what had happened and how to prevent it from ever happening again.

With the spacecraft stable, but still in jeopardy, the next order of business was to extend our DSN coverage.  The Mission Operations Manager, with full agreement from the rest of the Flight Team, declared a “Spacecraft Emergency” with the Deep Space Network.  Under DSN guidelines, we could only call this if we thought the spacecraft was in immediate jeopardy of partial or total mission failure.  Under an emergency declaration, all missions using the Deep Space Network volunteer their normally-scheduled antenna time in a community effort to help the ailing mission.  Their help provided LCROSS with enough antenna time to work out its problems.

As a step to protecting LCROSS, the Flight Team had to figure out what caused the anomaly.  Since the spacecraft was healthy on our last contact, the anomaly had occurred while LCROSS was in a normally-scheduled 66 hour “out-of-view” period with the Deep Space Network (DSN).  The Flight Team could not collect telemetry during that time.  However, just for this purpose, LCROSS constantly records a part of its telemetry onboard, to enable our team to diagnose problems that happen while out of contact.  We set immediately to “downlinking” and analyzing our virtual recorder telemetry data to gather clues.  

Using this data, and over an hour or two of analysis, our team gradually pieced together the story.  As with most anomalies, this one stemmed from multiple vulnerabilities whose combined harmful effects had not been anticipated.

In summary, a spurious, short-lived error on the IRU was interpreted as a more serious fault by the spacecraft fault management system, resulting in a switch to a backup rate sensor (STA). That rate signal was noisy (including random variations over the “true” rate signal), but was misinterpreted as real, “clean” rate data, causing over-control by the attitude control system and resulting in a great deal of propellant consumption.  LCROSS detected the associated tank pressure drop, but with no fault management option available for disabling thruster control (thrusters are required to keep the LCROSS solar array pointed to the sun), and no ability to determine the specific nature of the pressure loss, the spacecraft fault management system performed steps to stop a leaking thruster (another potential reason for a pressure drop), and to power-up its transmitter to “phone home” to warn the operations team that there was a problem.  However, this call could not be detected over the Southern hemisphere, since there were no DSN assets that could “see” LCROSS in that location, so the call was missed.

The LCROSS team knew that it had a serious problem on its hands.  Did we have enough propellant to complete our mission?  Was the IRU truly healthy, or would it fault again and trigger another loss of fuel?  Thanks to the DSN Emergency Status agreement, LCROSS was able to get continuous ground antenna coverage, whenever the spacecraft was in view of one of the three DSN antennas.  However, given our position in our Cruise Phase orbit, only the Canberra antenna complex could actually “see” LCROSS at that time, and only for 15 hours per day. 

Given our limited DSN visibility, our third (now primary) order of business was to protect our spacecraft from events that might trigger another propellant loss, and hence a loss of the mission.  We didn’t have much time before we’d go below the horizon with Canberra, so we had to resort to simple fixes first – ones that would protect us, but not add even greater risk.  Our actions were:

  • To increase the “persistency” of the IRU fault check by flight software.  By requiring an IRU fault to persist for 5 consecutive seconds rather than 1 second before tripping to STA, we hoped to avoid switching to the STA for an inconsequential IRU fault, yet remain protected against a serious IRU fault.
  • To augment the IRU fault response by automatically recovering the IRU (as we had done manually) first, before switching to STA for rate information.  Even if an IRU fault lasted for 5 seconds, this second-tier change would try to re-instate the IRU first, and fail over to the STA only if this had failed.

We designed, implemented, tested and loaded these changes to the spacecraft in our remaining time.  By 1:50 PM Pacific Daylight Time, with these temporary fixes in place, we began configuring LCROSS for our forced DSN outage.  Just 10 ½ hours after discovering our problem, we had to release our hold on LCROSS again and anxiously look to the next time when we would reacquire communications with the spacecraft.


August 23 – September 3 (DOY 235 – 246): Our Recovery

The extended LCROSS team mobilized to get the mission back on track.  Our goals evolved over the recovery period, but here’s what we had to do:

  • Improve our fault protection against an IRU failure for the long term.  We had short-term fixes in place from the first day, but we ultimately needed to implement a more complete solution.  These would take longer to design and test.
  • Improve fault protection against further excessive propellant loss.  This was a core issue.  The loss of the IRU was one potential vulnerability, but there might be others we didn’t yet know about.  LCROSS could not tolerate another similar loss of propellant.
  • Determine whether the IRU was really showing signs of failure, or whether it was actually fine.
  • Assess our propellant margin.  Could we complete all of our mission objectives? Would we have to give up some of our planned activities to save propellant?
  • Develop a plan for LCROSS health monitoring for the remainder of the mission.  With so much at stake, we felt we needed a way to regularly monitor LCROSS after we emerged from Emergency Mode.  However, with a team as small as ours, watching the spacecraft 24 hours a day for the rest of the mission would exhaust us.  Besides, we needed to devote a lot of time in the coming weeks to prepare for impact.
  • Continue performing the nominal events that could not be put off until our anomaly resolution was complete.
We implemented a schedule of 16 hour days, covered by two overlapping execution shifts in the Mission Operations Control Room, and a back-room design, implementation and test team split between the Mission Support Room at Ames and the Remote Operations Centers at Northrop Grumman. 

Propellant Usage Monitor and Free-Drift Mode

Our first priority was to develop fault protection against excessive propellant consumption.  The sure way to save propellant is to not fire thrusters.  But firing thrusters is the only way to maintain attitude control, and that is critical for generating solar energy and to keep the spacecraft thermally stable.  It seemed we were in a bind.  But then we recalled a strategy the team had originally designed to fight large Centaur gas leak torques (see the post entitled “Real-Life Operations: Day 3” for a description of the Centaur leak issue). 

The attitude control system has a special safe mode called “Sun Point Mode” (SPM) that points the solar array at the sun using special Coarse Sun Sensors, and without the use of the Star Tracker.  SPM spins LCROSS very slowly, end over end, but keeps the solar array pointed at the sun at all times.  SPM still requires thruster control to maintain stability. However, through simulations, our engineering team discovered that by modifying SPM to spin faster, LCROSS would have enough angular momentum to keep it stable for a very long time.

We still needed a means to detect excessive propellant usage.  We deemed propellant tank pressure measurements a little too coarse for such an important job, and yet there was no other single, direct indicator of propellant usage in the system.  However, one of our team members came up with a great idea. LCROSS generates and stores special telemetry “packets” each time a thruster is fired.  These packets accumulate over time in spacecraft memory, and are downloaded to Earth for analysis.  His idea was to monitor the accumulation of thruster telemetry packets as an indirect indicator of propellant usage.  It was unconventional, but the idea worked.

So, we designed an excessive propellant usage monitor that watched telemetry accumulation, and if too many packets accumulated over a period of time, would cause LCROSS to transition to our modified SPM, “free drift” mode to stop thruster firings, yet keep the spacecraft safe.  This was onboard and operating by DOY 241 (August 29).  There were other “free drift” ideas, but the beauty of using a modified SPM is that SPM was already fully integrated into our fault management approach for serious spacecraft anomalies.  So, much of the software design and testing was already in place.  This was far more appealing that developing a new, complicated control mode that might have introduced more risk than it retired.

Efficient Control under Star Tracker Rates

With a general propellant savings approach, we also set to improving the ACS performance under Star Tracker rate information.  Northrop Grumman developed an entirely new controller with filters to remove the STA noise.  In simulations, it promised to drastically reduce propellant consumption under STA rate information if our IRU failed.  We loaded the final version to the spacecraft on DOY 246 (September 3), replacing our old SIM DB3 mode, and it performed even better than expected, reducing propellant consumption by a factor of approximately 50 as compared to our anomaly.  An IRU failure would now be far less severe in propellant cost.  Even better, it was more efficient than the original SIM DB3, so now we’d be saving propellant, relative to the initial plan, for the rest of the mission.

IRU Healthy

We never determined the root cause of the momentary IRU glitch, but it never happened again.  Between discussions with the IRU vendor and analysis of many days of telemetry, we ruled that there was no reason to think the IRU was in jeopardy.

Enough Propellant for Full Mission Success

There is no way to directly measure the amount of propellant left in the LCROSS tank.  One can estimate the remaining propellant load by measuring pressure, temperature and tank volume (and compute the result using gas law equations), and alternatively one can estimate the propellant consumed through the mission by measuring the accumulated on-times for all the thrusters, along with predictions of how much propellant each firing consumes. Neither method is perfect. 

Analysis shows that LCROSS expended about 150 kg out of the 200 kg it had remaining in its tank before the anomaly, roughly half of its launch propellant load.  However, even under worst-case assumptions, our engineering team determined that LCROSS would have enough propellant to meet the criteria for full mission success.  However, we wanted to be very protective of our remaining propellant.  In agreement with the Science Team, following great successes with Lunar Swingby, Earth Look Cal 1 and the Special Earth/Moon Look Cal, we canceled our final Earth Look Cal and our Moon Look Cal, but added an Earth Gaze Cal that would efficiently allow the scientists to check the instruments following the anomaly.  We’d also continue to look for other opportunities to save what little propellant remained.

The Mission Goes On

During our anomaly recovery, we had to support a number of activities that couldn’t wait until later.  The day after our discovery of the anomaly (DOY 235, August 23), orbit geometry dictated we needed to rotate our spacecraft to re-point our omnidirectional antenna toward the Earth again, or suffer very poor communications for many days to come.  We performed another of these 10 days later on DOY 246 (September 3).

In a bit of good news, TCM 5c, originally scheduled for DOY 236 (August 24), was determined to be unnecessary.  As with so many TCM’s in our original plan, the Mission & Maneuver Design and Navigation teams planned TCM 5a so accurately that neither of the follow-up burns (TCM 5b and TCM 5c) needed to be performed.

However, not all of our news that day was good.  Also on DOY 236, we thought we had also experienced signs of a failing Star Tracker.  Out of concern for its survival, we shut down the STA for two days, and spent time away from our main anomaly recovery developing protections for the STA.  As it turns out, the unusual STA signature was related to an onboard clock calibration we had just performed.  The STA was just fine, but you can imagine, in the midst of the tension of the anomaly recovery, this extra scare didn’t help matters!  As in so many cases with space flight, even seemingly benign changes can lead to unexpected results.

Finally, the anomaly recovery partially interfered with two planned DSN rehearsals for the separation and impact events.  Both events require a lot of coordination, since we’re juggling three antennas, each with multiple receiver configurations.  We were able to follow through with a rehearsal of TCM 10, Separation and Braking Burn, but our very busy schedule forced us to abandon our plans for the Impact DSN test.  Another victim of unexpected events.

Negotiations with Deep Space Network and other Missions

To address the need to monitor LCROSS more often after our exit from Emergency Status, the LCROSS project negotiated a schedule with the DSN mission community that shortened all out-of-contact periods to 9 hours duration or less.  This came at great impact to other missions, but was invaluable to LCROSS in achieving its Emergency Status exit criteria. 

To address the concern of Flight Team fatigue that would result from having to staff all of these extra hours, we designed a way for the DSN to perform simple, regular health checks on the spacecraft.  Our team standardized our communications downlink rate to 16 kbps to indicate a healthy spacecraft.  In the event of an emergency, LCROSS automatically switches to a lower downlink rate (2 kbps).  The DSN can easily distinguish between these two rates when it acquires the LCROSS signal, and so our team worked out a protocol for monitoring the signal during each view period, and a call-tree of LCROSS operations personnel should they discover LCROSS is transmitting at 2 kbps.  Under the modified Concept of Operations, the Flight Team must now check spacecraft telemetry daily to ensure LCROSS remains healthy.

DOY 247 (September 4): Emergency Over

Twelve days after the anomaly discovery, having augmented LCROSS fault protection and our contact plan with the DSN and other DSN-supported missions, we requested that DSN remove LCROSS from Emergency Status. 

One of the gratifying aspects of this effort was how the LCROSS team responded to make sure LCROSS stayed safe.  Members of the Flight Team, LCROSS project and NASA Ames worked tirelessly.  Northrop Grumman mobilized to provide a number of valuable improvements to LCROSS.  The DSN mission community generously volunteered their antenna time to support our bid to restore LCROSS to nominal status.  NASA management, from Headquarters to NASA Ames, provided constant support and valuable assistance. 

Having survived this test and emerged with the moon and full mission success still in our sights was quite an accomplishment.  I want to thank the dedicated members that make up this team, and to everyone else that helped with our recovery!