LCROSS Flight Director's Blog

The LCROSS mission was ultimately focused on the final four minutes of flight, starting at the time of the Centaur impact, and ending with the impact of the Shepherding Spacecraft.  During that time, the Payload Engineer and the Science Team took operational center stage.  Once the science payload was powered on, the team’s job was to confirm the full functionality of the instruments, and then to adjust instrument settings to make sure the data we received was the best it could possibly be.  For Impact, there were no second chances – the Shepherding Spacecraft was to be destroyed as a forgone outcome of its observation of the Centaur lunar collision. 

 

In this post, I’ve invited Payload Engineer Mark Shirley to provide his perspective of impact night.  Mark was the Payload Software Lead, in charge of the design and implementation of onboard instrument command sequences and other onboard software components, as well as a large fraction of the science-related data processing software used on the ground.  During flight, Mark sat in the Mission Operations Control Room (MOCR) with the rest of the Flight Operations team.  On the final day, his job was to assess and maintain the engineering functionality of the science payload in those minutes before impact. 

 

In the impact video sequence, from the public’s perspective, there were a lot of things that happened operationally that were probably difficult to understand.  Mark will explain the plan for gathering data from the impact and also describe what actually happened in flight.  On a lighter note, Mark was one half of our now famous (or notorious) “high-five malfunction” that created such a buzz on the social media circuit after impact.  He’ll explain that as well. I’ll let Mark take it from here.  Enjoy!

 

I’d like to describe our plan for collecting data about the Centaur impact from the shepherding spacecraft (S-S/C), how actual events differed from the plan, and what that says about the process of developing and flying spacecraft.  In particular, I’ll cover why some of the pictures were fuzzy and some were white and why we were sending commands during the last minutes.  I won’t touch on the scientific interpretation of the data, only the process of gathering it.  This story contains some hard work, a few mistakes, a little nail biting tension, and finally, success.

 

Central to the story is the type of mission LCROSS was: a cost-capped, fixed-schedule mission.  That meant if LCROSS had been late, LRO would have flown with a dead or inactive LCROSS.  If the project had run out of money, whatever hadn’t been done wouldn’t have gotten done.  Within the LCROSS project, the instruments were in a similar position.  The original idea was to observe the Centaur impact from Earth only.  Onboard instruments were soon added to the design but with a total instrument budget of approximately $2 million.  That’s much less than single instruments on many other missions.

 

Two things made success possible.  First, project managers kept a tight focus on using the barest minimum hardware and testing required to perform the science, and went beyond that, only as the budget allowed, to increase the likelihood of success.  Second, everyone stayed on budget. The payload team had no choice, but if any other part of the project had overrun by a lot, the payload might have been eliminated or flown only partially ready.

 

The LCROSS Instruments

 

LCROSS carried nine instruments.  Five were cameras to take pictures over a large range of wavelengths, that is, colors. One was for visible light that our eyes can see.  Two near-infrared cameras captured mineralogy and water signatures, and two mid-infrared cameras captured thermal signatures from -60C to +500C).  LCROSS carried three spectrometers to measure color very precisely.  One covered ultra-violet and visible wavelengths and two covered near-infrared wavelengths.  The latter spectrometers were the best at searching for water, and one looked down toward the Centaur impact the vapor cloud it kicked up and the other looked to the side as LCROSS passed through that cloud.  Finally, LCROSS carried a high-speed photometer to measure the brightness of the impact flash.

 

The instruments are described in detail here.  Data from all nine instruments had to share LCROSS’ one Mbps (one megabit or million bits per second) radio link to the ground.  At that rate, it takes 2 seconds to transmit a typical cell phone picture.  This was the maximum data rate available for the LCROSS mission and used only twice: lunar swingby on June 22^nd and impact on October 9^th. All other instrument activities that took place during the 112 day mission used speeds less than 256 kbps (kilobits or thousand bits per second), which was sufficient for collecting data to calibrate the instruments.

 

The Observation Plan

 

The two components of the LCROSS mission, the Centaur and the Shepherding Spacecraft (S-S/C), separated about 10 hours before they reached the moon.  At the moment the Centaur impacted, the S-S/C was still 600 kilometers above the surface.  Falling at 2.5 kilometers per second, the S-S/C reached the surface 4 minutes later.  Observations of the Centaur impact event made during those 4 minutes were the purpose of the mission.  Unlike orbital missions that can usually try multiple times to collect data, we had just one shot.

 

The diagram below shows the plan for observing from the S-S/C, starting one minute before Centaur impact, at the beginning of what we called “Sequence 2” in the NASA TV video.  The diagram plots our intended schedule of instrument observations against time: each row represents one of the instruments (instrument abbreviations appear below each row of data), and each tick mark along a row represents one observation, either an image or a spectrum.  Over some intervals, the observations are spaced so closely that the plot looks like a solid bar.

 

Figure 1 The LCROSS impact observation plan: These timelines indicate when each image and spectra was planned to occur during the final four minutes of the mission.  The horizontal axis represents time.  Each row represents an instrument, and each tick mark represents the timing of a sample (an image or spectrum) from that instrument.

 

The last four minutes were divided into three periods, called FLASH, CURTAIN and CRATER.  Each period focused on a different aspect of the expected impact event and emphasized data collection from different instruments.

 

FLASH started one minute before the Centaur impact and focused on the very short burst of light generated by the Centaur impact itself.  Starting from the top of the diagram, the plan was to stop both the Visible Light Camera (VIS) and the Near-Infrared Camera #2 (NIR2).  This would allow us to focus on NIR1 images which we felt had the best chance of catching the location of the impact flash which we expected to be visible for less than one second.  These three cameras shared a common input to our payload computer, called the Data Handling Unit (DHU), and could not be used simultaneously. By stopping VIS and NIR2, we could run NIR1 at a faster rate (see the segment labeled ‘A’), increasing the odds it would image the flash.  The planned sequence also increased the NIR1 exposure time to capture the flash signature even if it was very faint.  We knew this would produce a badly overexposed image of the illuminated lunar surface, but our goal was to locate the impact.  We’d have plenty of other pictures of the surface.  This sort of shifting attention between cameras accounts for the periods where one camera image would stop updating for a while. 

 

We also designed the FLASH strategy for the spectrometers around our expectation of a dim, short duration flash event.  Near Infrared Spectrometer #1 (NSP1), the main water-detection instrument, was put into a high-speed, low resolution mode (represented by the yellow bar).  The Visible and Ultraviolet light Spectrometer (VSP) was commanded to take long exposures, and Total Luminescence Photometer (TLP) was powered early enough to reach equilibrium and be at its most sensitive for the flash event.

 

The second phase, CURTAIN, started just after the Centaur impact and ran for three minutes.  Its purpose was to take spectra and images of the expanding vapor and dust clouds thrown up by the impact.  CURTAIN was the most important period and also the simplest.  All instruments ran in their default modes, as follows.  The DHU shifted between the three analog cameras in a stuttering pattern – VIS, VIS, NIR1, NIR2 – repeating.  Both thermal cameras monitored the plume shape and temperature. The two downward-facing spectrometers (NSP1 and VSP) looked

for water and other chemicals.  The side-looking spectrometer (NSP2) also looked for water and other compounds, but from sunlight scattered or absorbed by the dust and vapor cloud.  The TLP continued to take data during this period, but it’s primarily function was during FLASH.

 

The goal of CRATER, the final period, was to image the crater made by the Centaur impact to get its precise location and, more importantly, its size.  From its size and their detailed models of crater formation, the LCROSS Science Team can potentially tell us how the crater evolved over the few seconds of its formation and how much material was excavated.  The primary instruments in this period were the two thermal cameras, MIR1 and MIR2.  Their sample rates were increased relative to those for CURTAIN.  To image the crater in a second frequency band, NIR2, the more sensitive near infrared camera, was commanded to its most sensitive setting.  NIR1 and VIS would not be used during this period because neither was sensitive enough to see anything in the permanently shadowed area.  All spectrometers would continue running to look for light reflected off of any plume or vapor cloud.  At the end of this phase, the S-S/C would fall below the rim of Cabeus Crater, cutting off radio transmission to Earth, and then impact the surface a couple of seconds later.

There were three keys to making this plan work:

• Downlink Bandwidth: the data collected had to fit within the 1 megabit radio downlink. We did a lot of testing before launch to work out a data collection plan that was further confirmed and refined based on on-orbit performance. We gave priority to data from the most important instrument, the near-infrared spectrometers, to provide robustness to the design. The best scheme was pre-programmed and ready to go in case we were unable to command the spacecraft in the final hour.
• Camera Exposure: We had to change camera exposure settings during the descent to reflect the changing    brightness of the impact event and the surrounding scene.  Defaults were pre-programmed based on the latest lighting models for impact morning from NASA Goddard Space Flight Center.
• Command Timing: In the instrument command sequences governing FLASH, CURTAIN and CRATER periods, we had to orchestrate changes in instrument configuration as they were needed to focus on    different aspects of the impact event.  Sometimes these changes had to be interleaved with instrument data    collection in a way that was vulnerable to small timing changes.

 

So What Actually Happened?

 

Well, as reflected in the recent LCROSS press briefing, we collected a very rich and interesting data set that met the needs of our science objectives.  However, we had challenges in all three areas – bandwidth, exposure settings and timing – although all ultimately proved minor. However, in some ways, it was a close call.  This diagram shows what data was actually collected during the final four minutes of the mission.

 

Figure 2 Actual performance of the LCROSS payload: These timelines indicate when the images and spectra were taken on the morning of the impact.  The pattern gives clues about the performance of the hardware  and software systems that collected the data.

 

First, the rows representing the spectrometers, NSP1, NSP2 and VSP, look almost exactly as they should.  Except for one problem with the Visible and Ultraviolet light spectrometer (VSP), which I describe later, our plan for collecting spectra worked perfectly.  This is very good, because the spectra carried most of the information we were trying to collect.

 

As for the cameras, several differences from the plan jump out.  The most obvious is that the timing of observations along some timelines is irregular with many observations missing, e.g., the visible camera pointed to by note E. This occurred with all five cameras (the first five timelines) but not with the spectrometers (the next three timelines). 

 

Scene Complexity and Bandwidth Limitations

Image compression is the process of finding and reducing redundancy in an image in order to transmit it more efficiently.  On LCROSS, to fit within the 1 Mbps data rate limitation for our downlink, we used a lossy compression algorithm that typically reduced each image to 1/20th its original size.  Lossy methods achieve greater compression than lossless methods by actually removing parts of each image.  The algorithm tries to find subtle details whose removal the human eye won’t notice.  Being able to combine many different kinds of data into a single digital data stream is so useful that this approach has been standard practice for many years.

In flight, the irregularity of observations occurred because we underestimated the complexity of the lunar scene during ground testing.  We had done much of our testing with a large reproduction of the moon’s pole in front of the cameras, but it turned out this didn’t mimic the high contrast and detail of the real scene.  Scene complexity mattered because the images were highly compressed and changes in the moon scene changed the sizes of the compressed images by a factor of 4. We first observed this behavior during the lunar swingby LCROSS performed during the first week of its mission.  Turning on the instruments during the swingby was intended as a learning experience, and it proved critically important.  It provided the best operational practice we got for the impact as well as data to calibrate the instruments.

 

After the lunar swingby in June, I changed the thermal camera sampling rates in the instrument command sequences for the final hour.  Unfortunately, the compression problem turned out to be about 20% worse during the final hour of the mission than during the lunar swingby.  This forced us to change the thermal camera rates again in real-time, but we had practiced changing them during rehearsals, just in case.  In the NASA TV impact video sequence, you can hear the Science Team requesting a change of MIR1 rates to 1 Hz, and MIR2 to 0.1 Hz.  See note F in the figure.  The rate for thermal camera #1 (MIR1) changes just before this note and changes for MIR2 just after it.  Even though we’d practiced, this was still a very tense time as we were losing some data while the changes were being made. Changing the MIR rates felt like it took forever.

 

The bandwidth problem could have been avoided if in addition to changing camera sampling rates in the command sequences, we had also changed the stuttering pattern for the analog cameras mentioned above to eliminate one VIS image during each repetition.  However, our instrument simulator didn’t have the full set of instruments like the spacecraft, which made it impossible to adequately test this change on the ground.  At one point, we discussed testing this change onboard before the impact, but lost the opportunity due to the fuel loss Paul described in his blog on October 4^th (see “A Test of the Flight Team”).

 

One other problem caused by the complex lunar scene was damaged images.  After compression, some of the visible camera images were still too large to fit within a single data packet for transmission to Earth.  Here’s an example of the kind of damaged image that resulted.  The shadowed area should be completely dark, but instead contains wispy bright areas. These compression-artifacts are intimately linked to the scene and need to be taken out with image post-processing.

Figure 3 Example of damage to downlinked images due to clipping in the telemetry packet formatting software.

 

What caused these compression artifacts?  I didn’t know it at the time, but the software for compressing these images had been written some years before to clip the compressed form of images to ensure they always fit within a single data packet (maximum size 65536 bytes).  We used a wavelet-based compression algorithm, and clipping the compressed images removed some information needed to recreate the image accurately.  The alternative would have been to split the images across multiple packets and reassemble them on the ground.  This certainly could have been done in principle, but doing so would have introduced significant changes right at the heart of software that we had planned to reuse without change after its successful use on previous projects.  With what we know now, changing this would have been justified, but it would also have been risky given how central image compression was to the overall design.  With what we knew at the time, I believe we would have left it alone because of our short development schedule and all the other things that had to work.

 

November, November!

 

Most of the commanding we did from the ground was to adjust the exposure times of the near infrared cameras as the scene changed.  The other cameras either controlled themselves (VIS) or had only one appropriate setting (the thermal cameras, MIR1 and MIR2).  We controlled the exposure setting for the near infrared cameras explicitly because we were trying to image a relatively dim flash and ejecta curtain close to bright mountain peaks.

 

Near the beginning of the FLASH period, we discovered we didn’t get this balance right. To image the dim centaur impact flash, we deliberately overexposed the sunlit peaks.  This setting combined with the Cabeus scene overdid it. The sunlit areas electronically bled into nearby parts of the image.  That occurs when electrons in overexposed pixels move across the image detector to other pixels.  In this case, the shadowed area of Cabeus crater was completely covered, obscuring our view of the impact.  That was why the only image that was updating just before the Centaur impact was white.  We hadn’t seen this level of bleeding earlier in the mission, or in almost any of our testing.  However, after searching through our data archive, I realize now we did see it occur once, two years ago, in one flashlight test in a darkened room but did not fully comprehend the implications.

 

The FLASH period was designed to start 1 minute before the Centaur impact, so we had a little time to recover once we saw the problem.  During this minute, our first priority was to confirm that the spectrometers (NSP1 and VSP) and photometer (TLP) were working properly.  Once that was done, we focused on the NIR1. Since we still had commanding during this period, we tried to change the exposure setting (Payload Scientist Kim Ennico called out “Flight, this is Science, please change NIR1 to OPR 9, over.”). We had less than 30 seconds to get this command sent up to the spacecraft.  The command was actually sent but arrived a few seconds too late to capture the impact. In hindsight, this was a challenging stretch for the camera’s range, due to the scene and the potential for bleeding. Our strategy – to aim for the most sensitive exposure setting followed by one attempt to back-off depending on the data-might have worked had we been looking at another region of the moon, that is, had, we launched (and impacted) on a different date, where the terrain and lighting would have been different. While all this was going on, the impact flash was captured by NSP1, so the key science measurement was made.

 

We intentionally caused the same issue later, during the CRATER period, but we had better success (see above figure at the segment labeled ‘B’).  Initially, the NIR2 camera images were badly overexposed for the same reason as during FLASH (hence the white images that appear in the NASA TV video just after entry to DV Mode).  Kim Ennico, the Payload Scientist, made the call to reduce the exposure time slightly, from what we called OPR 15 to OPR 10.  (Again, you can hear this request over the voice loop in the NASA TV video.)  She was using live-information from the NSP1 spectrometer and checking those values in real time against a spreadsheet near her seat.  You can see her checking and rechecking on the video before making the choice.  We only had only one chance to choose the right one.  The command was sent and received 30 seconds before the S-S/C’s impact.  Kim’s call initially left the images overexposed, but as the lit peaks slid out of the field of view, her choice produced excellent images of the very dark crater floor, including the image that gave us our best estimate of the Centaur crater size.  These images go all the way down to 2 seconds before S-S/C impact where the craft was 5 kilometers above the surface. The crater floor of Cabeus was indeed brighter than any of the predictions, at least in the infrared. That’s another reminder of science and exploration. Sometimes you are surprised as you collect new data, especially data from areas never looked at before.

 

Figure 4 This image sequence was captured just before the end of the mission and shows the NIR2 camera going from badly overexposed to acceptably exposed as the lit peaks surrounding Cabeus leave the field of view.

 

How do we know that’s the Centaur crater?  Because it was also seen by the two thermal cameras (MIR1 and MIR2) while it was still warm, and we can overlay the images.  The left figure below shows aligned images from NIR2 and MIR1, taken before the Centaur impact.  The figure on the right shows aligned images from these cameras taken just before the S-S/C impacted and showing the Centaur impact crater (see inset).  These images don’t align perfectly because they were taken about a second and 2.5 kilometers apart.  To obtain images of the Centaur crater using three different cameras was our goal, and we succeeded.

 

Figure 5 The right image shows the Centaur impact crater in both near-infrared and mid-infrared images.  The left images overlays images taken before the impact by the same two cameras.

 

Kim’s call of “NIR2 to OPR 10” yielded a great image of the Centaur crater, but it also caused some confusion.  The name for Near-Infrared Camera #2 (NIR2) was too similar to the name of Mid-Infrared Camera #2 (MIR2).  We had practiced this interaction over our voice communication loops, but we hadn’t practiced it enough to do it quickly and perfectly under time pressure.  Kim had to repeat her call using the phonetic term ‘November’ for the ‘N’ at the beginning of NIR2 (during CURTAIN phase in the NASA TV video, you can hear the Flight Controller, Jim Strong, ask “is that ‘November’ or ‘Mike’?”, referring to NIR2 or MIR2, ).  We didn’t realize when we picked the obvious names for these cameras three years ago that the names could cause confusion when spoken over the voice loops connecting our mission control rooms.  Back then, our plan avoided real-time commanding completely, but as we learned through our practice sessions before launch and actual experience after launch, we realized we needed the flexibility.  We retrofitted a process for proposing and confirming real-time commands into our mission operations architecture as best we could given the facility and time constraints.  Note that we didn’t consider doing this for maneuvering the spacecraft or for the most critical science instruments, only the secondary instruments.

 

Command Timing

 

The commanding side of the automatic sequence ran almost perfectly.  We did have one problem with the Visible and Ultraviolet Spectrometer during the CURTAIN period, though.  Because the instrument data handling unit (DHU) was at its maximum data throughput capacity during the first part of CURTAIN, one command to change exposure time was delayed and sent during a period when the instrument wasn’t listening.  That command was ignored.  This resulted in capturing fewer spectra with longer-than-planned exposure times.  Luckily, the longer exposure times turned out to be a blessing, since the ejecta curtain was much fainter than some models predicted. The loss of more frequent sampling due to the longer exposures did not affect the science measurement.

 

High Five Fail

 

Yes, I should have.  After the end of the mission, I missed a high five that was captured on camera.  I was teased about it by my colleagues that morning and by my kids that night. The other operator involved, our Telemetry Data Manager, known as “Data” over the voice loop, is both a good colleague and a friend.  I didn’t intend to embarrass him and have since apologized.  We had been told to avoid high fives to prevent exactly the sort of mistake I made, but once the hand went up, I should have responded.  So, here it is, for the record:

 

Why didn’t I respond?  I honestly don’t remember the moment clearly, but I did have two things on my mind.  First, my job at that point was to move to the Science Operations Center (SOC) to prepare for the post-impact press conference.  We had two hours to make sure we’d gotten the data we expected, to prepare presentation charts and to look for anything obvious.  I was concerned about that because I had missed seeing the plume we had hoped for like most everyone else.

 

More importantly, I was really stressed out.  The DHU, the computer that processed all instrument data, was struggling with the very large packet sizes of visible camera images, and the DHU almost crashed a number of times during that final hour.  We had developed and practiced a procedure to recover from such a crash to prevent a substantial loss of science data.  Flying a spacecraft is a group effort with lots of cross-checking, and as the Payload Software Lead, I felt especially responsible.

 

During payload development testing, we found and fixed several problems that would have been problematic for the payload.  This problem which led to potential crashes of the DHU was known and was the most difficult software problem we saw.  The root of the problem was a small chip that controlled the data bus connecting the video capture and compression chips to each other and to the main processor within the DHU.  Under certain circumstances this bus controller chip would stop responding, and the DHU software would crash.  Since we didn’t have access to the chip’s design to understand why it would stop, and we didn’t have time to replace it, our approach was to create a method for quickly recovering on orbit.  This method had two parts.  The first part was a software patch we developed that reset the bus controller when the DHU’s main processor noticed it had stopped responding.  The second part was a procedure for quickly rebooting the whole DHU from the ground if the software patch didn’t catch the problem.

 

We developed and tested the software patch just a few weeks before the payload was shipped to Southern California for integration with the rest of the spacecraft. From that point on, through the rest of our testing on the ground and in orbit, we didn’t see this problem again. That is, we didn’t see it until the morning of the impact.  That morning, the patch needed to reset the bus controller two dozen times.  The vertical green lines in this figure show when.

 

Figure 6 During the final hour of the mission, Data Handling Unit (DHU) software detected and corrected an anomalous condition on a bus controller chip multiple times.  The green lines show when these events occurred.  The right end of this figure, starting at the label ‘I’ (the time of Centaur impact) corresponds to the time spans of the planned (Figure 1) and actual (Figure 2) performance plots. Earlier events happened while monitoring payload performance in the 56 minutes prior to Centaur impact.

 

Once these events started, I was prepared, on a hair trigger, to start the process of rebooting the DHU if the patch didn’t work.  I was constantly checking and rechecking the fault response procedure I had developed for our payload.  The details of this procedure varied over time.  As the on-board sequence progressed and we got closer and closer to the Centaur impact, we had different decisions to make to recover if something went wrong.  This strategizing was being done over another voice loop with Kim and Tony Colaprete, the LCROSS Principal Investigator, in the Science Operations Center (SOC) which was not audible to the audience watching on NASA TV.  We had to keep track of a lot of independent data threads and contingencies simultaneously.  Our actual trigger for starting this recovery, a gap in numeric sequence of the data coming from the instruments, even occurred once, but it was unrelated and didn’t need a response.  In the end, our defenses worked.  The software patch performed exactly as intended and no crash occurred.  After the S-S/C impact, I breathed a sigh of relief and moved to the next room to start preparing for the press conference.

 

Final Thoughts

 

As I said above, we had challenges in all three areas critical to making our plan work: downlink bandwidth, camera exposure, and command timing.  Ultimately, all of the problems we had proved minor, and we collected the data we needed to draw conclusions about the presence or absence of water and other substances in Cabeus crater.

 

We at NASA all too often strive to give the impression that complex, difficult missions are routine.  They’re not.  They’re complex and difficult.  What makes them possible is long planning, teamwork, and careful review by people both inside and outside the project.  One name for this process is “Systems Management”, which recognizes that

people need backup just like the parts of a complex machine.  I personally made some mistakes and caught some mistakes.  Together, we caught enough of them that we were successful.  For me, it was a huge privilege and a wonderful experience.  I’m very grateful to have been a part of this mission.

LCROSS, the flight mission, is over, but we’re still analyzing the data that was collected on impact night.  Our team is very excited about the recent announcement – that LCROSS confirmed the presence of water at the lunar south pole.  But, there’s a lot more data to analyze, both for science and engineering purposes, and I’m guessing there will be more interesting announcements in the months ahead. 

With the completion of mission operations, before concluding my account of the LCROSS journey, I wanted to add a few more posts relating our team’s experiences in the last day of the mission.  In this post, I’ll describe my last shift in the Mission Operations Control Room (MOCR), covering the Separation event and the final delivery of the Centaur impactor.  In a near-future post (I promise!), I’ve invited our Payload Engineer, Mark Shirley, to post with a detailed look at payload operations during the Impact event.  For all of you who watched the impact video and are wondering what happened that evening, Mark’s post should be very enlightening.

Now, a note about terminology – the two major pieces of hardware on LCROSS are the Centaur (the impactor) and the Shepherding Spacecraft (abbreviated as “S-S/C”), which carries the science instruments and that has acted as the “living” part of LCROSS since the Centaur’s batteries ran out on Day 1 of the mission.  Up until now, I’ve referred to the combined Centaur and S-S/C as “LCROSS”.  But in describing what happened after Separation, when the two pieces decoupled, I’m forced to refer to each vehicle independently.  Finally, as a reference for timeline details, check out my post entitled “Brace for Impact: A Schedule of Events for the Final Day”.

Shift A: Release of the Centaur and Slowing Down the Shepherding Spacecraft

On October 8, I was the Flight Director in charge of the second-to-last shift (Shift A) of the LCROSS mission.  Shift A’s main task was to perform final targeting of the Centaur, via the separation of the Centaur from the S-S/C.  The Separation event was the final influence our team would have on the Centaur impact location and time.  Soon after Separation, we were to command the S-S/C to image the Centaur as it receded from view.  Shift A was also in charge of the “braking burn”, a delta-v maneuver performed after Separation that would decelerate the S-S/C relative to the Centaur to delay its own impact on the moon, thereby improving our view of the impact event during the collection of science data.  

Figure 1: Diagram of Separation, Centaur Observation and Braking Burn

In all, we could not have had a better shift.  Everything went incredibly smoothly.  Here are some highlights:

Command Loads

Once “on console”, Shift A’s first responsibility was to load all of the command sequences to the S-S/C that would govern the initial attitude change to Separation attitude, the Separation itself, the Centaur Observation and the Braking Burn events. 

With so much at stake, the Flight Team was very focused, and we completed those command loads very early, with no problems.  The onboard commands were designed to execute the whole sequence through Braking Burn (minus the Centaur Observation) without human intervention.  Barring unforeseen problems, the Flight Team could have walked out of the control room at that point.  But in reality, there’s no way we’d leave the success of the mission to chance.  As I’ll describe later, Separation had a lot of potential risk.  If anything went wrong, our team needed to be there to take corrective actions and to set up for Impact.    Furthermore, the Flight Team had an integral role in initiating the Centaur Observation. 

Reorienting to Separation Attitude

To precisely target the Centaur into Cabeus crater, our plan was to use a fraction of the velocity change caused by Separation to push the Centaur into the right impact trajectory.  The interface ring between the Centaur and S-S/C contains springs that push the two vehicles apart as soon as they’re released.  They produce up to an estimated 500 lb of force, and were predicted to induce approximately 0.7 meters/sec of relative velocity between the two vehicles.  However, since the Centaur far outweighed the S-S/C, we expected only 0.15 m/s would be imparted to the Centaur, the rest to the S-S/C. 

But we didn’t need all of that extra velocity for a precise impact.  To use only a fraction of this added speed, we’d have to perform Separation in a specific direction, so that only the needed component of the velocity change would affect the impact position, while the other components would have no effect on impact position and little effect on timing.  Our Maneuver Design team determined, coincidentally, that this direction was only 7 degrees away from our starting Cruise attitude. 

So, what happened?  A portion of the commands we had just loaded automatically performed the change in orientation.  We needed to take every precaution not to disturb our orbit prior to Separation.  Rather than commanding the entire 7 degree change in one step, the command sequence performed the maneuver in a long series of small attitude updates.  This minimized the number of thruster firings required, though it took over 20 minutes to complete.   The whole sequence of commands executed perfectly and with one task completed, I prepared mentally for the next.  With the rotation completed, we were less than one hour from Separation.

Separation

Other than Impact, Separation was perhaps the most critical event of the mission.  Since the Centaur was not independently controllable after Day 1 of the mission, the de-coupling of the S-S/C from the Centaur was LCROSS’s last influence on the Centaur impact trajectory.  As Separation approached, my adrenaline started to flow.  Here are the reasons Separation made me a little nervous:

• Centaurs typically separate from their payloads within just hours of launch.  LCROSS’s Separation was 112 days after launch, far longer than ever done with a Centaur before.  The space thermal environment often wreaks havoc on mechanical elements, with extreme temperatures and many swings between hot and cold.  Analysis showed that the mechanism could operate over a very wide range of temperatures and after many warming/cooling cycles.  Still, we didn’t know if something had been overlooked.  The good news was that temperatures from the separation mechanism had remained well within allowable limits for the entire mission, increasing our probability of success.  However, if Separation were to fail, LCROSS would impact in one piece, the science payload would be unable to watch impact, and our mission success would be entirely reliant on ground-based and other orbiting observatories to collect impact data.  

• The Centaur separation mechanism was designed to very reliably push the S-S/C away smoothly, without causing the spacecraft to tumble.  To prevent attitude control from interfering with Separation, the command sequence would disable ACS thruster firings briefly through the transition.  But what if something snagged or re-contacted?  Disabling the ACS would allow rotations induced by separation to go unnoticed (and uncorrected) for a few seconds.  Again, analysis predicted success, but there were a lot of unknowns here.

• In the instant of Separation, LCROSS would transform from a 2894 kg spacecraft to a 617 kg spacecraft (about 1/5 of the mass).  The moments of inertia (the spacecraft’s tendency to resist changes in rates of rotation) would also decrease dramatically.  To remain in control after this enormous change, the ACS would use new sets of control gains optimized for post-separation conditions.  But none of these controllers had been tested on the real spacecraft (only in good simulations – they wouldn’t have worked well with the Centaur attached).  Dynamically, it would be like getting a whole new spacecraft, but with only 9 hours to figure out how it really behaved before Impact.  To make matters more exciting, our command sequence utilized 5 out of 6 attitude control modes in the first 40 minutes after Separation.  Each mode employed an independent set of control gains.  One mode might work, and another might be flawed and cause an instability.

• Though the Centaur and S-S/C would recede from each other at very slow speed (0.7 m/s, or about 1.6 mph) after Separation, it wouldn’t take very long for the Centaur to be quite far from the spacecraft (42 meters or half of a football field farther each minute).  The Science Team really wanted to film the departing Centaur to determine whether the separation had induced any tumble into the Centaur, and to use that knowledge to better understand the Impact behavior later.  To “see” any tumble, which might be very slow, we would need to be able to distinguish the Centaur long axis from its short axes.  To do that, the Centaur would have to be close enough to span at least a few pixels in the LCROSS cameras for several minutes.  But at Separation, the LCROSS cameras would be facing away from the Centaur.  The mission plan involved flipping the S-S/C 180 degrees to point them toward the Centaur as soon as possible after Separation.  But this ran counter to the desire of our engineers, who would have preferred to watch the ACS control behavior for a while.  As a compromise, we planned on rotating just one minute after Separation.  Good for Science, but a little uncomfortable for the Flight Team, which would have to confirm a good separation, then oversee a 180 degree flip on a new set of control gains!

• Neither the S-S/C nor the Centaur could automatically confirm that separation had actually succeeded, so the Flight Team needed to confirm success from the ground via telemetry.  A set of three small wires would break at Separation, and enable us to determine a physical separation.  Then indirectly, we expected to observe a dynamic response from the S-S/C from the push and any resulting torques.  When designing this event, we had a choice: either have the Flight Team initiate the 180 degree flip from the ground on successful Separation (but risk being late on the Centaur observation if we lost communications), or execute the flip using onboard commands and have the Flight Team terminate the sequence if Separation failed (but, if we lost communications and the ability to command, risk performing the flip with the Centaur still attached).  Mission-wide, we had adopted a policy that assumed success, so we opted for executing these commands onboard.  This meant that within 60 seconds, the Flight Team had to identify whether Separation had been successful, and terminate the running command sequence if it had failed.  Sixty seconds may sound like plenty of time, but with multiple operators on console evaluating multiple telemetry indicators, allowing time for a possible termination of the command sequence, sixty seconds was barely enough. 

So, what actually happened? At 10 minutes until the designated Separation time, the onboard command sequence started to run.  Game time.    Months of design, weeks of training would all come together now.  I counted down the last 5 seconds until the commands to fire the Clamp Band Ordnance Device (CBOD) would have issued from the onboard sequence.  Then we focused on telemetry.   Silence on the voice loop.

Ten seconds after the separation time, the Flight Team broke into chatter over the voice loop.  I polled the System Engineer.  He returned with “three wires, Flight”, meaning that all three wires that indicate a clean break between Centaur and S-S/C all showed positive.  I polled ACS (Attitude Control System Engineer).  He announced “GO Flight”, indicating he had observed a significant attitude disturbance that could only have happened with a successful separation.  I polled the Flight Controller, and he confirmed “three wires”.  In the meantime, the control system had re-enabled, and quickly brought the attitude back under control.  Separation was a “GO”!  Relief.  No need to terminate the nominal sequence.  I instructed the Flight Controller to waive that option.  Then, at the expected time, the S-S/C began to perform its 180 degree flip. 

With no time to celebrate, the Flight Team re-focused on the next task – Centaur Observation.

Centaur Observation

To ready for Centaur Observation, the Flight Team’s next job was to reconfigure the communications subsystem and payload to enable the spacecraft to collect imagery of the Centaur as it receded from view.  To enable the downlink of high-rate imagery, we first had to switch from our nominal downlink data rate of 64 kilobits per second (kbps) to 256 kbps.  Once the data rate had been increased, we’d then increase the allowable data output rate of the science payload, and switch from the low-rate to the high-rate imagery sequence designed for this event. 

Data rate changes and antenna changes were two operations that we typically didn’t perform via onboard commands.  Each requires the DSN antenna providing the link to re-acquire the spacecraft signal.  The time needed to do that varied significantly, but fixed-time commanding doesn’t allow for that flexibility.  For this event, we had three Madrid DSN antennas watching simultaneously (two 34-meter dishes and a 70-meter dish), and each needed to re-acquire the LCROSS signal after the downlink rate change.  To complicate things, we knew we couldn’t achieve an adequate link margin at 256 kbps until the spacecraft had completed its 180 degree flip (to point the center of the secondary omni-directional antenna pattern towards Earth), the duration of which wasn’t known with high certainty given our lack of in-flight testing with the new gains.  By design, we planned to utilize ground-based commanding. 

In actual flight, as expected, the science payload powered-on just as the S-S/C began to flip.  Payload Engineering reported good status.  Soon, the cameras were transmitting low-rate imagery, but we hadn’t rotated far enough to see the Centaur yet.  But without the Centaur on its aft end, LCROSS behaved less like a school bus and more like a sports car.  It was making short work of the 180 degree flip.  At three minutes, with the flip complete, we commanded the downlink rate change.  At about the same time, Science reported the receiving the first image of the Centaur, under the low sampling rate.  The DSN stations re-acquired the signal, and as soon as possible, we commanded the payload to begin the high-rate image sequence.  We has successfully configured the S-S/C for the Centaur Observation, and we now had a little time to breathe, and to watch the Centaur images flowing into the control room.    

This link points to a movie made by the Science Team using the images collected during the Centaur Observation (with time sped up significantly). 

https://www.nasa.gov/mov/392770main_mir1-zoomed-uncompressed-21fps.mov

In the movie, you’ll notice it looks like the Centaur is moving all over the place, but really it’s the S-S/C rotating back and forth.  The attitude control system keeps the spacecraft pointed to within a small “deadband”, a small imaginary box around a target pointing direction.  The ACS only fires thrusters if the S-S/C is getting near the edge of the deadband.  What you’re seeing is the ACS “bouncing” on the edge of the deadband.  If you look carefully, you’ll notice that the ACS seems to change to a smaller deadband later in the movie.  This not an illusion.  The spacecraft switched from a wider deadband (1.0 deg) to a narrower deadband (0.5 deg) partway through the observation sequence, as part of a test of the new control gains.  You’ll also observe that the Centaur is in a very slow tumble.

Braking Burn

At the time of the Centaur impact, to meet the observation requirements, the S-S/C needed to be roughly 600 km above the lunar surface.  This was a trade-off between being too close (and possibly being hit by debris from the Impact), and being too far away (the spatial resolution of our imagery would decrease with distance from the surface).  The velocity induced by Separation between the S-S/C and Centaur (0.7 m/s) would be too little to achieve that distance in only 9 hours 40 minutes, so the Braking Burn was inserted into the mission design to slow down the S-S/C down by an additional 9.0 m/s (about 13 times the velocity change of Separation).   The Braking Burn was also designed to independently target the S-S/C impact point, also selected by the Science Team.

In flight, nineteen minutes after Separation, the onboard sequence terminated the Centaur Observation and began configuring the spacecraft for the burn.  The Flight Team quickly commanded the return to 64 kbps downlink rate (necessary before changing our attitude again), and the DSN re-acquired the signal.  The onboard sequence performed another attitude change, this time to the optimal burn attitude.  Ten minutes later, after passing through two more new attitude control modes, the burn started on time.  Four minutes of firing the 22 N thrusters was predicted to be sufficient to slow ourselves down to meet the impact distance requirement.  With the burn over, we had crossed Shift A’s last major hurdle. 

Shift A Cleanup and Farewell to LCROSS

We had a few final things to do before our handoff to Shift B.  We biased our attitude to keep our thrusters warm, re-enabled payload heaters, and loaded a preliminary Impact command sequence to the spacecraft.  Planned before Separation using predictions of the post-Separation and Braking Burn orbit, those commands were designed to perform the full Impact sequence without the Flight Team.  They were an insurance policy to protect the team in the off-chance that we lost communications with LCROSS until the last few minutes.  However, without accurate knowledge of the S-S/C post-burn orbit, they wouldn’t point the cameras as accurately as the re-planned versions.  That planning was up to the next shift, using the actual, measured orbit after Braking Burn. 

With that, Shift A was done.  In a rush all day, we sat on console for a few more minutes, soaked up our last experiences as members of the LCROSS Flight Team.  It was a bittersweet moment – so many experiences in those chairs in such a short time, and the culmination of three years of preparation.  Our final shift had gone perfectly – the result of months of thought and practice, and a great spacecraft design.  The moment we’d all been waiting for was only 8 hours away, and then the LCROSS flight would be over.  Officially, our time was over now.  Shift B folks began milling into the control room, with a combination of big smiles and game faces on, ready to get started on their last shift.  With just a little reluctance, we stood up from our chairs, and got on with shift handover.

Figure 3: MOCR Shift A: The Ames-local part of Shift A that ran Separation through Braking Burn.  Pictured from R to L: Matt D’Ortenzio (Flight Controller, NASA Ames), Matt Reed (Attitude Control Engineer, Northrop Grumman), Tony Lindsey (Data Management Engineer, NASA Ames), Tony Colaprete (LCROSS Principal Investigator, NASA Ames), Darin Foreman (Systems Engineer, NASA Ames), and myself, Paul Tompkins (Flight Director, Stinger Ghaffarian Technologies at NASA Ames).  A lot of other people also worked this shift and contributed to its success!

Remember, the next post will feature Mark Shirley, our Payload Engineer.  Stay tuned, and thanks for reading!

Our last day in flight promises to be the most challenging and the most rewarding for the project.  Our 112 days in orbit are focused entirely on the last four minutes, after the Centaur impacts our target crater and raises a plume of lunar material for the LCROSS Shepherding Spacecraft to observe for signs of water, but before the Shepherd also impacts the moon. 

From a Flight Team perspective, the LCROSS impact sequence is a dream occasion, and yet provides some cause for trepidation.  Many things can go wrong, and with so little time, there is only so much that can be done. 

During you day tomorrow, I thought it might be fun for you to know what the Flight Team will be doing in lead-up to the event.  To put it plainly, we won’t be idle!  Enjoy!

A Recent Development: TCM 9 put LCROSS On-Target

The latest data from our Navigation team indicates that TCM 9 has already put LCROSS on target to hit the designated impact area, without the need for executing TCM 10 on Thursday evening. Our predicted impact point is already within our target 3.5 km diameter circle, and our team will only make very small adjustments to improve our impact accuracy.  This alters our original plan for Thursday.

Instead of performing TCM 10, the team will plan and execute a very slow rotation by feeding the spacecraft new target attitudes each minute (see “Once More Around the Earth” for a description of our “quaternion creep” attitude change) to minimally disturb the current orbit while turning to an orientation that is optimal for Separation.  At Separation, LCROSS will use the velocity imparted by the springs between the LCROSS Shepherding Spacecraft and the Centaur (adding an estimated 15 cm/s to the Centaur) as a final means of nudging the Centaur toward the center of our target.  Analysis of the Centaur separation springs, along with actual tests of the system conducted to simulate very harsh conditions of space (far harsher than LCROSS has actually experienced) indicate the separation will impart a fairly precise change in velocity to the Centaur. 

This plan represents less risk (the slow attitude change will be simpler to plan, test and execute than a TCM), and introduces less uncertainty into the prediction of our impact point (firing thrusters for very short durations adds a lot of uncertainty, while the separation springs in the LCROSS-Centaur interface mechanism are very repeatable).  We’re fortunate to find ourselves in this situation, and we’ll take full advantage of it to ensure we impact on-target.

DOY 281 (October 8): TCM 10 (or not), Separation, Centaur Observation and Braking Burn

The last 24 hours of the mission, bridging DOY 281 and 281 (October 8 and 9), will be a flurry of activity.  Here is the sequence of events.  I’ve provided both UTC and Pacific Daylight Time references:

• 08:00 UTC/01:00 PDT: Final orbit determination delivery for Separation.  The Navigation team delivers its final orbit determination to the Maneuver Design Team.  This final trajectory estimate will be the basis for planning our slow rotation, Separation and Braking Burn.
• Maneuver planning and communications link analysis for slow rotation through Braking Burn. For eight hours, the Maneuver Design team will determine the optimal attitude for Separation (when we let go of the Centaur), then plan and double-check plans for the slow rotation, Separation and Braking Burn.  Braking Burn happens after Separation, so will have no influence on the path of the Centaur.  It accomplishes two things: first, it slows our Shepherding Spacecraft with respect to the Centaur, such that at the time of Centaur impact, the spacecraft will be 4 minutes behind it.  This allows LCROSS to observe the Centaur impact while not being too close (risking damage from debris) and not being too far away.  Second, the Braking Burn independently targets the Shepherding Spacecraft impact point, which will be a few kilometers away from the Centaur impact point.  During the same time period, the Communications Link Analyst will refine his estimate of our communications link margin through all phases of the slow rotation, Separation and Braking Burn.
• 13:00 UTC/06:00 PDT: Separation Activity Selection Review (ASR): Our team knows every last detail of what activities we’ll be running, but this meeting is our last chance to change any part of the command sequence, based on late-breaking data (e.g. changes on the spacecraft, etc).  The Maneuver Design Team and Communications Analysis Teams will present their results here, and will form the basis of command generation.
• 14:00 UTC/07:00 PDT: Command generation and checking for Separation through Braking Burn. Our Activity Planning and Sequencing Engineer will generate all of the command sequences for TCM 10, Separation, Braking Burn, a preliminary version of Impact, as well as for several contingency cases.  He will hand his products over to both Engineering Analysis, and to the Simulation Engineer, who provide different aspects of quality assurance checks.  Engineering Analysis performs a number of computer-based checking against LCROSS flight rules to make everything in the sequence is legal.  Both the Simulation Engineer and the Engineering Analyst will run the commands on our spacecraft simulator to confirm that they do what we want.  During this time, Shift B (Flight Directory Rusty Hunt’s shift) will hand off to Shift A (my shift).  Shift B will get some sleep, and return before Impact.  Shift A will oversee all of the events through Braking Burn.
• 19:00 UTC/12:00 PDT: Separation Command Approval Meeting (CAM): This is our final, team-level review of all plans, command products and quality assurance data before executing the slow rotation through Braking Burn.  We’ll make sure everything is correct, over a 90 minute review.  Then we’ll move to our seats in the Mission Operations Control Room (MOCR) to begin execution.
• 20:30 UTC/13:30 PDT: Command loads for Separation, Centaur Observation and Braking Burn, and slow rotation to Separation attitude. Once “on console”, Shift A’s first priority will be to load the commands for Separation, Centaur Observation and Braking Burn.  We then turn our attention to the slow rotation to the Separation attitude, by loading the burn commands to the spacecraft in an alternate memory bank.  The slow rotation command sequence will re-orient the spacecraft from our Cruise attitude to the Separation attitude.  We’ll confirm the loaded parameters, and then wait for the reorientation to start.
• ~00:00 UTC/17:00 PDT: Slow rotation to Separation attitude starts.  The maneuver is small, only 6 degrees or so, but will happen in chunks of less than 0.5 degrees each minute.  The onboard command sequence automatically switches over to our Separation, Centaur Observation and Braking Burn command sequence, just in case we lose communications with LCROSS.  In that off-nominal scenario, Separation would still happen without ground-based commanding by our team.
• 01:40 UTC/18:40 PDT: Separation onboard command sequence starts.  The pre-Separation command sequence starts running.  Ten minutes to Separation.
• 01:50 UTC/18:50 PDT: Separation.  Commands temporarily disable our ACS, then fire the relays that unlock the Centaur from our spacecraft.  Heavy springs push the Centaur and spacecraft apart at roughly 0.7 m/s, a firm but gentle shove.  The Centaur will accelerate approximately 15 cm/s, but with our optimal orientation, only 3.5 cm/s will be used for Centaur targeting.  After Separation, the ACS is re-activated with an entirely new set of parameters to handle the vastly different mass properties. With the Centaur separated, LCROSS will just have lost 2000 kg of mass.  The spacecraft motion (dynamics) will now behave very differently.  The Flight Team has only 10-15 seconds to confirm that Separation has occurred, and if not, only 50 seconds more to terminate the command sequence to progress any further.  We have practiced this critical timing many, many times.
• 01:51 UTC/18:51 PDT: Flip to point LCROSS instruments at Centaur.  Just 1 minute 6 seconds after Separation, the onboard command sequence initiates a 180 degree pitch flip to point spacecraft cameras at the departing Centaur.  This takes less than 3 minutes to perform.  The command sequence also powers up the Data Handling Unit (DHU), which powers the science instruments, in preparation for Centaur Observation.  Following the pitch flip, commands roll the spacecraft to optimize the pointing of our omni-directional antenna toward Earth for best downlink rate.  At the end of the pitch maneuver, the Flight Team will re-configure the LCROSS downlink data rate for 256 kbps, and will command the DHU to go to a high-rate camera sampling sequence.  Imagery of the departing Centaur, with the moon in the background, will begin flowing to Earth.
• 02:01 UTC/19:01 PDT: End of Centaur Observation.  Nineteen minutes after Separation, with the Centaur nearly 800 meters away, the Centaur Observation will terminate.  The Flight Team will reconfigure the communications downlink rate for Braking Burn (64 kbps).  The onboard command sequence automatically re-orients the spacecraft to the final burn attitude, and then squeezes down our attitude control deadband from 3.0 degrees to 0.1 degrees, in preparation for Braking Burn.
• 02:30 UTC/19:30 PDT: Braking Burn starts.  This burn is longer our last few TCM’s, just over four minutes.  This is because there’s not much time remaining in the mission to build up a 4-minute delay between the Centaur and the Sheperding Spacecraft.  At the end of the burn, the onboard command sequence will re-orient the spacecraft to our Cruise attitude. 
• ~03:00 UTC/20:00 PDT: Preliminary Impact command load.  As a precaution, the Flight Team will load a preliminary command sequence for Impact to the spacecraft.  If we lost communications with LCROSS sometime after this point, up until the final few minutes, this command sequence should be sufficient to point the LCROSS cameras at the Centaur impact point, run the instruments, and meet all mission objectives. However, before Impact, the team will re-estimate the orbit of the Centaur and Shepherding Spacecraft, and re-plan Impact with the best possible information.
• 03:30 UTC/20:30 PDT: Shift Handover.  Shift A (my shift) hands control over to Shift B.  Shift B will oversee the Impact event.  We’ll review the status of the spacecraft, in particular the dynamic behavior following Separation, and any last-minute items.
• 04:30 UTC/21:30 PDT: Final Orbit Determination Delivery. The Navigation team delivers its final estimate of the spacecraft and Centaur orbit. The spacecraft’s orbit can be measured directly, while, without a communications transponder aboard the Centaur, we have no direct measure of the Centaur’s orbit after Separation.  This final orbit determination will become the basis for Impact command sequences, in particular the spacecraft attitude sequence to maintain pointing on the Impact site, and the Impact timing.
• Final Impact Planning and Command Generation: The Mission Design team will re-plan Impact using the latest orbit data from the Navigation team.  The changes between preliminary and final Impact plan will be very subtle.  The plan involves literally hundreds of Shepherding Spacecraft orientation changes to keep the onboard science instruments pointing at the expected Centaur impact point as we approach the moon.  The new orbit estimate will change all of these orientations very slightly.  The Sequencing Engineer will re-implement the command sequences, then pass his results to the Engineering Analyst and Simulation Engineer for final checking. 
• 6:30 UTC/23:30 PDT: Disabling LCROSS Fault Management. Shift B will begin configuring LCROSS for the Impact.  One of the first steps is to nearly completely disable the LCROSS onboard fault management system.  Fault Management responds automatically to correct problems it detects onboard.  Sometimes these are benign responses, like switching from a primary sensor to a backup sensor. Other times, the responses can be all-encompassing.  It might seem strange to disable this function right before our most important phase of the mission.  However, the last thing the Flight Team wants is for a problem onboard the spacecraft to interrupt our Impact observations.  Some fault management responses are designed to throw LCROSS into a Survival State, turning off all power to the science payload, and disabling any onboard command sequences.  This could mean disaster for the Science Team, since there would not necessarily be sufficient time to recover and return to the pre-Impact configuration.  So, only minor fault management is enabled, but the more severe responses are disabled.  In preparation for Impact, aside from disabling fault management, Shift B will also coordinate with the Deep Space Network to transfer our downlink path from a 34 meter diameter antenna (DSS-24) to the Goldstone complex’s 70 meter dish (DSS-14).  The 70 meter antenna enables LCROSS to return science data at 1 megabit per second (1 Mbps).
• 8:30 UTC/01:30 PDT: Impact Command Approval Meeting (CAM).  Shift B will review the final Impact plan and the associated onboard command sequences and ground commanding products.  This is our last chance to get things right.  Since the team is focused on a very specific set of checks, and for lack of time, this CAM lasts only 30 minutes.  Then Shift B goes back to the MOCR to perform Impact.
• 9:00 UTC/02:00 PDT: Loading Impact command sequence to LCROSS. Shift B loads the final command products to the Shepherding Spacecraft, including a set of contingency command sequences to cover off-nominal scenarios.  In the event of a building fire or an earthquake, our team even has a command sequence that would allow Shift B to leave the building and have the entire Impact sequence and observation be automated.  The Deep Space Network has dedicated four antennas to this period of time, three from the Goldstone complex in California, and a fourth located at Madrid in Spain.  Shift B, with the help of DSN operators at JPL, will coordinate those antennas as LCROSS changes its communications configuration.  Hours earlie
• 10:00 UTC/03:00 PDT: Start of Impact onboard command sequence.  Its first commands will perform a reorientation of the Shepherding Spacecraft to point the science instruments towards the expected Centaur impact point on the moon.  The cameras and other instruments will not yet be on.  This reorientation will also point the –Z Medium Gain Antenna (MGA) towards the Earth, enabling the team to switch the LCROSS downlink path from the omni-directional antenna to this MGA, in preparation for high-rate science data transmission. 
• 10:10 UTC/03:10 PDT: Switch to –Z MGA. Shift B will command the switch from omnidirectional to the –Z MGA antenna.  This is a potentially critical step in achieving full-rate science data transmission after the Centaur impact.  However, since we did our combined Cold-Side Bakeout #3/MGA Test on September 24, we’re pretty confident this will work again.
• 10:15 UTC/3:15 PDT: Transitioning to Science Rate. The Flight Team will now command a transition from a standard downlink data rate of 64 kbps to our full science rate, 1 Mbps.  This is another very important step to achieving full science return.  However, we do have backup procedures that would allow us to transmit science data at a lower rate, 256 kbps, if the DSN 70-meter dish were to fail, or if the MGA was non-functional.
• 10:36 UTC/3:36 PDT: Payload powers on. The onboard Impact command sequence powers on and enables the DHU and science instruments.  At 10:41 UTC, the command sequence also starts DHU NVM sequence 1, a sequence of instrument commands that tests each instrument in the LCROSS  payload, save the Total Luminescence Photometer (TLP).  The MOCR at NASA Ames begins to receive data from the science instruments, and the Payload Team and Science Team begins analyzing the preliminary data to make sure everything is working.  This is still nearly one hour from Impact, but it’s the team’s last chance to find a problem in our suite of payload instruments that might otherwise foil our Impact observation.  The team continues checking the instruments, and via the Flight Controller and Flight Director, commanding small adjustments to exposure settings, for 35-40 minutes.
• 11:10 UTC/4:10 PDT: TLP Instrument powers on. The Total Luminescence Photometer (TLP) instrument powers on for the first time since before launch.  This instrument is very sensitive, and can only be powered on a limited number of times.  The Science Team has been very careful not to overuse the instrument in tests.  However, if the instrument powers on as expected, this is a major success on the road to the Impact event.  The TLP, which gathers light measurements at 1000 times per second, will “catch” the Impact flash as the Centaur hits the moon, and is hence a very important instrument for water detection.
• 11:30:20 UTC/4:30:20 PDT: Flash Mode begins. One minute prior to Centaur impact, the DHU will command NVM command sequence 2, which begins Flash Mode.  For the next 1 minute 3 seconds, Flash Mode will run the TLP and other instruments to capture the flash of light coming from the impact event. 
• 11:31:20 UTC/4:31:20 PDT: Centaur Impact. Centaur impacts the moon at Cabeus.  The energy of impact emits a brief, intense flash of light.  A plume of lunar debris will rise in a pattern similar in shape to an inverted conical lampshade.
• 11:31:23 UTC/4:31:23 PDT: Curtain Mode begins. The DHU will switch from Flash Mode to Curtain Mode, which is a sampling sequence optimized to observe the evolution of the debris plume as it rises from the lunar surface.  With this debris rising above the altitude of the Shepherding Spacecraft, our side-looking spectrometer will look towards the sun to measure light as it is transmitted through the debris.  The remainder of the payload will be pointed down towards the impact point. This mode lasts for 3 minutes.
• 11:34:23 UTC/4:34:23 PDT: Crater Mode begins. At this late stage, the DHU will now switch from Curtain Mode to Crater Mode, which is designed to capture data about the properties of the new crater generated by the Centaur impact.  The Shepherding Spacecraft now has less than one minute of time to capture and transmit data before it also hits the moon.  With the Centaur impact point now off to the side, LCROSS will continue to try and track that point until its own contact with the moon.
• 11:35:39 UTC/4:35:39 PDT: Shepherding Spacecraft impact. The Shepherding Spacecraft will also hit the moon at roughly this time.  The Flight Team will abruptly stop receiving telemetry a few seconds later, as the photons from LCROSS’s last transmission travel back to Earth to be received by the DSN 70 meter antenna.  The LCROSS flight mission will be over.

This will be my last post until after Impact.  I hope you enjoy the show tomorrow – it should be very exciting.  Though we won’t have immediate feedback for water detection, I hope to report good news to you on Friday regarding the accuracy of our impact, and the collection of the science data.  Then, over the coming weeks after Impact, the Science Team will review their data and interpret the observations. I’m sure you’ll be hearing news one way or the other.

Thanks for reading!

Saying Goodbye to a Really Amazing Spacecraft (and Team)

Well, we all knew it was going to happen.  It was inevitable.  It was the whole design of the mission.  LCROSS was destined to end its wonderfully fantastic journey by intentionally crashing into a permanently shadowed crater at the south pole of the Moon.  We are the ones who devised this fate for LCROSS.  So why should we be surprised (and just a little bit sad) now that the time has finally come?

As a proud member of the LCROSS Science Team and as the Observation Campaign Coordinator, I would have to say that working on this mission has been one of the highlights of my career thus far.  The mission itself is truly amazing (We’re impacting the Moon!  We’re looking to see if there’s water ice at the poles!  We’re going to this utterly unexplored place in our Solar System, so close to home, and are so excited about what we’re yet to learn!).  LCROSS is so important to both science and exploration.  This mission is blazing a new path in how to build small, robust spacecraft both on schedule and on budget.  LCROSS uses eight (yes eight!) commercial off-the-shelf instruments for its payload – also a very novel way for NASA to get more bang for the buck as well as good science to boot.  The technical aspects of the LCROSS mission are astounding, but none of this would be possible without the dedication of the *people* working on this project.

The LCROSS Team is made up of an amazing cadre of individuals.  LCROSS has a relatively lean and nimble team.  There’s still a lot of work to be done to send a spacecraft to the Moon, and so that means that everyone has to pull together to make things happen.  If somebody is extra busy and needs help, you help them.  If there’s something that needs to be done and you’ve never done it before, you figure out how to do it.  If you are stuck and need some assistance, just ask your teammates and without hesitation people are willing to help.  We all naturally come together to get the job done.  There is a high level of trust and commitment on this team, starting with the top Project management and all the way through the people working the nitty-gritty technical aspects.  It is truly a glorious experience to work with a team such as this.  The best part is that everyone is working towards a common goal, and everyone is willing and able to contribute in whatever way is needed in order to achieve the objective.  It is amazing what a group of people can do when presented with a fascinating project and an exciting challenge.

And it’s not just the Project folks who have helped make this happen, but it’s all of the students and members of the general public who have so substantially contributed to the successes of LCROSS.  Student interns at NASA Ames have had the opportunity to work with real honest-to-goodness flight hardware.  Not everyone has the opportunity in college to hold an instrument that will be on the Moon within the next year!  Such opportunities are tremendously powerful for encouraging the students of today to continue the pursuit of careers in math, science, and engineering.  Amateur astronomers from around the world have been imaging LCROSS in the night sky during its trip to the Moon and are planning to collect observations of the impacts as well.  This is a great way to actively participate in a NASA mission.  We’ve also been having a great time keeping folks updated regarding LCROSS activities through our NASA website as well as the LCROSS Facebook and Twitter accounts.  Thousands of people are following LCROSS on these sites and we’re thrilled to be able to have a two-way dialog to discuss all things lunar!

So, although this mission was destined to end in a spectacular grande finale culminating with two lunar impacts, it is a bit sad to see this phase of the project come to a close.  However, next up is the exciting analysis of the data to try and learn all we can about these enigmatic regions on our very own Moon.  And here’s to hoping there are lots more missions coming up in the future, because we are fired up and ready to go!

The LCROSS Mission Design Team included the Centaur as a critical component of their mission from the very beginning.  It’s essentially a bonus impactor that stays with LCROSS after performing it’s normal function of delivering LRO and LCROSS on their way to the moon.  For the LRO-LCROSS launch and particularly for its contribution to the LCROSS mission, the Centaur was the focus of extra attention from both the LCROSS team and from the KSC/ULA-Atlas launch team.

Think of it as energy recycling.  The kinetic energy  [1/2 MV²] provided by the launch vehicle to get LCROSS and Centaur to lunar swing-by are later used to create the ejecta plume at the moon’s south pole.

Impacting a 2-ton Centaur upper stage enables the LCROSS mission to significantly increase the amount of lunar ejecta expelled from the target crater over what could have been achieved using only the spacecraft.

Centaur is a cryogenic (liquid hydrogen and liquid oxygen) powered rocket  upper stage. Its engine can be shut-down and restarted after one or more guided coast phases.  Centaur has a long history of successful  missions.  For LRO-LCROSS, instead of being parked in a safe orbit after sending LRO and LCROSS toward the moon, this is Centaur’s first role as a guided impactor for the science gathering  portion of a mission.

The Centaur upper stage used for the LCROSS mission is the same as Centaurs used to propel the Mars Reconnaissance Orbiter and Pluto-New Horizons on their way to Mars and Pluto, respectively.  The only significant modification to this Centaur is the addition of white thermal paint to help balance the temperature of the empty stage. 

Upper Photo: The LCROSS logo, painted on the Centaur.  Lower Photo: The Centaur upper stage is lifted for mating to the Atlas-V first stage (Tail Number 020) at the Vertical Integration Facility in Florida (April 30, 2009). The Centaur was painted white to help manage the thermal environment during the 100-plus day LCROSS mission.  Yellow lifting hardware is ground handling equipment and is removed after stacking on the first (booster) stage.  Courtesy of NASA Kennedy Space Center.

What is particularly unique about this Centaur was the way that the United Launch Alliance (ULA) Atlas Team designed the flight profile and maneuvers after LRO separation to minimize the amount of residual hydrogen, oxygen, hydrazine, and helium left on Centaur.  This special care to ensure that the Centaur is as empty is possible, makes the Centaur very clean. It reduces the hydrogen (H2), oxygen (O2), and hydroxyl (OH) species that could be confused with in-situ water ice ejecta.

Since Centaur would be attached to LCROSS until just before impact, all of the Centaur hardware and functional system were analyzed and scrutinized and if necessary tested to ensure that they would not adversely affect LCROSS after LCROSS took over control from Centaur.

The final function to liberate Centaur as a guided impactor will be the release of the clamp band holding the LCROSS/Centaur stack together. This separation command will be sent by LCROSS approximately 9 ½ hours before impact.  The clamp-band and the separation system that will push LCROSS and Centaur apart, underwent long-duration testing to give the Mission team confidence that it will function smoothly after 100-plus days in orbit.

Additional Centaur info and history, courtesy of NASA’s Glenn Research Center, can be found at: https://www.nasa.gov/centers/glenn/about/history/centaur.html

The anomaly robbed the LCROSS Flight Team of precious time to prepare for Impact.  But with a healthy spacecraft, and enough propellant to do the job, our team was all too happy to prepare for the future.   Ahead of us were five more Trajectory Correction Maneuvers (TCM 6 – 10) to precisely refine our crater targeting, then Separation, Centaur Observation, Braking Burn, and finally, Impact.  In the midst of our TCM series, the Science Team continued refining their selection our target – the specific crater, and the point within the crater – based on the latest data from other missions.  As a final confirmation of the payload instruments, the Science Team also wanted to look at Earth one last time before Impact.  On top of all that, we needed to practice those final two critical days as much as we could.

DOY 247-251 (September 4 – 8): Housekeeping

Out of Emergency Status, we resumed operations gradually, monitoring spacecraft health and performing typical housekeeping duties.  Of special note, as a result of the anomaly and our very small propellant margin, we decided to substitute Earth Look Cal 2, originally scheduled for DOY 250 (September 7), with a new, more propellant-efficient calibration maneuver that we termed “Earth Gaze”, on DOY 261 (September 18; see below).   

DOY 252 (September 9): TCM 6 Waived

We received more good news about our trajectory.  Despite the long train of thruster pulses that resulted from our anomaly, LCROSS’s orbit was still right on target.  The Mission Design Team and Navigation Team agreed that TCM 6 would be too small to be worthwhile.  This meant one less event to attend to in our busy schedule, and one less reason to take risks at this late, sensitive stage in the mission.

This decision worked in our favor for saving propellant too.  Each TCM typically requires the spacecraft to reorient to a specific direction so that LCROSS can add velocity in a particular direction via the 22 N Delta-V thrusters, and then back again to the Cruise attitude.  All of our remaining “burns” were expected to be small; so much so that the propellant devoted to the reorientation maneuvers was expected to be several times more than for the burn itself.  Waiving TCM 6 meant we could save more propellant, and reduce risk.

DOY 253 (September 10): Omni Pitch “Creep”

Throughout the mission, Omni Pitch maneuvers have been necessary to keep our primary antenna pointed towards Earth.  In all previous Omni Pitch renditions, the “slew” was performed in SIM DB2 (Stellar Inertial Mode, deadband 2), an Attitude Control System mode that enables relatively efficient attitude changes.  However, SIM DB2 significantly accelerates the spacecraft rotation to induce the change in orientation, and this costs propellant. 

In our bid to save propellant, we re-examined all of our previous maneuver approaches for opportunities to be more efficient, including for Omni Pitch maneuvers.  Someone on the Flight Team came up with the idea of providing LCROSS a long sequence of small attitude changes rather than one big one, with the goal of avoiding the accelerations of SIM DB2 (small attitude changes don’t justify large rotational accelerations, and therefore are cheaper propellant-wise).  Simulations of this new approach, what we informally called “Quaternion Creep”, indicated this would lead to a significant propellant savings, so we baselined this approach for our Omni Pitch maneuver on DOY 253. 

Normal Omni Pitch maneuvers used to take 40+ minutes to execute.  This new approach took 4 hours 20 minutes, but saved a lot of propellant.  Well worth the bargain.

DOY 257 (September 14): Guarding Against Single-Event Upsets

Cosmic rays are high-energy photons that are known to interfere with spacecraft computers by in effect reprogramming single bits of memory as they pass through.  When a bit in computer memory is reprogrammed by radiation, but not permanently damaged, it is called a Single-Event Upset (SEU).  Most spacecraft computers and onboard memory chips are “radiation-hardened” to prevent or otherwise sidestep the effects of cosmic rays. 

The spacecraft Data Handling Unit (DHU) is the independent computer that controls the operation of the science instrument payload.  Command sequences onboard the DHU control all of the sampling sequences and instrument settings to make sure LCROSS collects the best possible set of data at impact, and for other calibration events.  They are clearly very important to the success of LCROSS.

The DHU, and those instrument command sequences specifically, are somewhat vulnerable to SEU, and the Payload Team had not checked the contents of the DHU in a long time.  On this day, our team dumped (downloaded) the full contents of DHU non-volatile memory and confirmed that the sequences were still intact. 

DOY 257 – 259 (September 14 – 16): Full Rehearsal of TCM 10 through Impact

It was hard to find time in our busy operational schedule to practice all of our pre-Impact procedures.  In the months before launch, we spent all of our time readying for Transfer Phase and the first parts of Cruise Phase.  Impact seemed so far away.  Then Cruise Phase, with the anomaly, was far busier than expected.  With only weeks remaining, our perspective was entirely different.   

We took full advantage of two free days in our schedule to hold the “Last Two Days Rehearsal”, a full-team, high-fidelity rehearsal of the last two days of the mission, similar in style to our First Week Rehearsal (see the post on “First Week Rehearsal”).  For 42 hours, following the exact schedule we planned to follow in flight, and synchronized with the actual times of day when events would occur, the Flight Team practiced nearly every aspect of those operations.  We ran 24 hours a day, with three overlapping execution shifts (Shift B, then A, then B again), and two planning sessions.  In the rehearsal, the team successfully hit the target crater and collected all of the science data.  However, the there were some procedural and process shortcomings that made things just a bit rocky at times.  We learned a lot, and began working improvements in the days after the test. 

DOY 260 – 261 (September 17 – 18): Our Last Gaze at Earth

In substitution for Earth Look Cal 2, we conceived of a more propellant-efficient calibration event we coined, for lack of something more official, “Earth Gaze Cal”.  We loaded the commands for this event on DOY 260 (September 17), and executed the calibration the following day.  Rather than sweep back and forth over the Earth (see the full description of our first Earth Look in “During Our Second Trip Around Earth”), Earth Gaze looked straight at the Earth for an extended period, during which time we collected camera and spectrometer data.  This was the Science Team’s last chance to evaluate instrument performance and settings before Impact. 

We captured some more images.  Here are a few, along with a simulation to show the orientation of Earth during the event.

DOY 267 (September 24): Cold Side Bakeout #3 and a Test of the –Z MGA

Despite previous efforts to rid the Centaur outer skin of water on Cold Side Bakeout #1 and #2, our Navigation team continued to observe the accelerating effects of escaping water at the end of the second of those events.  With the amounts of water remaining, the Science Team was no longer concerned that this water could interfere with water measurements at Impact – there was just too little left.  However, Navigation was still concerned that remaining water might push our Centaur off course in the hours before Impact, after Separation when we no longer had any control over its orbit. 

Recall that we had planned to execute Cold Side Bakeout #3 on DOY 234 (August 22), but our plans were thwarted by the discovery of the anomaly.  Cold Side Bakeout #3 was unfinished business that had to be completed. 

In an unrelated thread, we also wanted to test the antenna we’d be using for Impact.  LCROSS has two Medium Gain Antennas (MGA’s), one on the +Z axis, the other on the –Z axis, used to downlink high-speed science data to Earth.  We had used the +Z MGA during Lunar Swingby, but had never tested the –Z MGA in flight.  We didn’t want to discover a problem with this antenna in the hours before impact, so we devised a test that would expose any issues immediately, and that would couple very nicely with Cold Side Bakeout #3. 

The combined Cold Side Bakeout #3 and –Z MGA Test took advantage of the fact that LCROSS was passing right through the ecliptic plane, the plane of the Earth’s orbit around the sun.  The LCROSS +X axis was perpendicular to the ecliptic plane at the time, and so by rotating about the +X (roll) axis, we could simultaneously face the “cold side” of the Centaur towards the sun, and the –Z MGA towards the Earth (required to test communications via this antenna).  It was a perfectly-timed opportunity.

On DOY 267 (September 24), we performed the maneuver.  Unlike previous versions of Cold Side Bakeout, we stopped the spacecraft twice, once at 135 degrees rotation, and again at 225 degrees.  The first position pointed the –Z MGA (which is canted by 45 degrees) straight at the Earth, and warmed one side of the cold skin of the Centaur.  After 20 minutes, LCROSS rotated another 90 degrees, moving the MGA off the Earth, and moving another part of the cold face of the Centaur into full sunlight.  We characterized one “slice” of the –Z MGA antenna gain pattern, confirmed that it was operational and mounted according to specification, and removed more water from our impactor.

One of the risks of Cold Side Bakeouts is that we might induce a thermal instability in our thrusters, as we had in Cold Side Bakeout #1, prompting our improvised fault management to fire the thrusters to keep them warm (see the post entitled “Our First Orbit Around the Earth” for details).  Happily, our thrusters remained thermally stable, and we avoided any additional propellant cost.

DOY 268 (September 25): TCM 7

LCROSS had not performed a Trajectory Correction Maneuver since TCM 5a, way back on July 21.  We’d been literally coasting along our orbit since then.  By this time, the Science Team had selected a satellite of the Cabeus crater, Cabeus A1, as our impact target.  Trajectory Correction Maneuver 7 (TCM 7) was performed on September 25 to target that location on Impact Day.  According to predictions on burn pointing errors, TCM 7 would guarantee a hit within 38 km of our intended target.  Future burns would reduce the error a lot further.  Still 14 days from Impact, TCM 7 nudged LCROSS by 32 cm/s (just over 1 foot/sec), but enough to make a big difference in impact position.

DOY 273 (September 30): TCM 8

Following TCM 7, the Science Team continued to receive data from other lunar missions.  Lunar Reconnaissance Orbiter (LRO), our partner at liftoff, devoted much of its operational time to scouring our top target regions, and data from Chandrayaan-1 and other missions continued to improve our knowledge of the mapping of possible water concentrations at the lunar south pole.  New information gathered after TCM 7 prompted the Science Team to redirect LCROSS to the main Cabeus crater.  Relatively close to the previous target, changing represented a small amount of extra propellant, but a potentially significant improvement in science data. 

TCM 8 was performed on September 30, and changed LCROSS’s velocity by 35 cm/s.  Not only did TCM 8 target the new impact position, but it also refined our impact timing.  Our narrow target time window will allow the Hubble Space Telescope to take images of our impact plume.

DOY 274 (October 1): Impact Contingency Rehearsal

Our Last Two Days Rehearsal on DOY 258-259 did not emphasize off-nominal events, but rather the full integration of the team in a single push.  On October 1, Shift B, who will be overseeing Impact, practiced two complete run-throughs of the final two hours of the mission.  I helped our Test Conductor in setting up the tests, and planning a series of anomalies that would really challenge Shift B.  During the simulations, we failed banks of temperature sensors and heater circuits, caused various instruments to malfunction and the DHU to crash.  We faulted the IRU, forcing a recovery in the midst of early pre-Impact science operations.  We failed the –Z MGA switch (even though this is far less likely having conducted our MGA test), and forced the team to perform the Impact sequence at a contingency low data rate.  Their team did very well, and succeeded in meeting mission objectives on both runs.  The members of Shift B were very happy they had that final opportunity to practice.

DOY 275 (October 2): Planning Rehearsal

The flight planning team has one of the biggest challenges in the final 24 hours before Impact.  Based on a final assessment of the LCROSS orbit, they have to plan the final series of maneuvers that culminate in the Impact event: TCM 10, Separation, Centaur Observation, Braking Burn, and Impact itself.  From the maneuver plan, the team must generate command sequences for each of these events, and then run them on our simulator to prove that they are flight-worthy.  Furthermore, they also need to generate and test command products to handle specific contingency events, like a failed Separation, or a late Braking Burn.  They even generate a preliminary Impact command sequence, just in case we lose contact with LCROSS for hours before the event.  Some of these products can be produced well in advance, but a great number of them depend on the final orbit assessment, and the specific orientations that are required for TCM 10, Separation and Braking Burn to ensure we accurately hit our target.

The Last Two Days Rehearsal proved that the planning team could generate all of its products, but could not quite perform its full list of quality assurance tasks in time.  Following the rehearsal, the team took some significant steps to streamline the planning, command generation and quality assurance process.  We found opportunities for working in parallel, for skipping unnecessary steps, for nailing down some variables in advance to reduce the amount of variance on Impact Day. 

To test these changes, we conducted a Planning Rehearsal for DOY 275 (October 2) to span the full 12 hours of the process, exactly as it will happen on the Impact Day.  I was really happy with the results.  The team actually finished most of its tasks early, including simulations of all command products, and their delivered products contained no errors.  The Command Approval Meeting, our last visual evaluation of commands, finished early.  I was very encouraged by these results, and the planning team is now fully confident that they could repeat this level of performance on Impact Day.

DOY 278: Final DHU NVM Sequence Loads and TCM 9

The remainder of this post has to do with current and future events.  I’m finally caught up!

This morning, as I write this post, Shift B is loading its final set of science payload command sequences to the spacecraft.  The Science and Payload Teams have pored over the data we collected from all previous science calibrations to derive a set of camera exposure/gain settings for Impact.  Over-exposure means washed-out images, which under-exposure means less contrast and detail.  Six of nine sequences will be changed to reflect the Science Team’s best knowledge of camera performance, and expected conditions on our final descent to the lunar surface.

Later this afternoon, Shift A will be performing TCM 9, our second-to-last, and possibly our last orbit adjustment.  After TCM 8, based on the latest lunar mapping data, the Science Team selected a different spot within the Cabeus crater.  This is our final target.  With only 7.6 seconds of firing, TCM 9 moves our impact point from our old point in Cabeus to this new location roughly 9 km away.  TCM 9 should put us within 1.75 km of our target.  If we can “nail” this maneuver by staying within that range, we’ll be able to skip TCM 10. 

TCM 9 is our last spacecraft activity, other than monitoring and housekeeping, until our final rapid-paced series of maneuvers in the final 24 hours.

As you can imagine, things are starting to get very busy!  I will submit one more post with a detailed description of the final 24 hours.  I also have a few guest authors who would like to share some things with all of you before Impact.  Stay tuned…this is going to be an incredible week.

Early in the second half of our second Earth orbit, while out of DSN contact, LCROSS experienced an anomaly.  An error detected with our Inertial Reference Unit (IRU) resulted in an automatic response that consumed a large amount of propellant in a short amount of time.  We spent the remainder of our second orbit recovering from that anomaly, and protecting against any future excessive propellant usage.  We emerged safely, but with so little propellant remaining that, since that event, we’ve had to step very carefully to avoid wasting any more of this precious resource.  This posting describes our discovery of the problem, and our recovery steps to protect LCROSS against a reoccurrence.

August 21/22 (DOY 234): Anomaly Detection, Spacecraft Safing, and Entry to Emergency Status

Shift A came into the Mission Operations Control Room (MOCR) on Saturday with a big agenda on LCROSS – to execute a third “Cold Side Bakeout” to rid the Centaur of more ice trapped in its skin, and to perform an Omni Pitch maneuver to flip the spacecraft to re-point the primary omni-directional antenna at the Earth, all under a tight schedule with little margin for error.  The shift started at 2:25 AM Pacific time, but despite that early hour, we were excited to have such a challenging pass ahead of us, and eager to get started.

We acquired spacecraft telemetry at 3:25 AM.  In the MOCR, each Flight Team operator sits behind a set of computer monitors that display fields of telemetry data – numbers, status indicators, etc. that indicate the health of LCROSS.  Many of the telemetry data fields show up in stoplight colors – green, yellow, or red, to indicate whether the value is “nominal” (green), or approaching an emergency state (yellow), or is in an emergency state (red).  Typically, telemetry comes up entirely green. 

• The Inertial Reference Unit (IRU), our onboard gyro, and primary means of measuring rotation rates around each axis for attitude control, had faulted;
• The Star Tracker (STA) had replaced the IRU in providing rotation rate estimates to the attitude controller;
• Spacecraft rotation rates were periodically exceeding yellow high alarm limits (rotating too fast);
• LCROSS was firing thrusters almost continuously,
• The propellant tank pressure sensor (our primary means of determining how much fuel we have left), indicated we had consumed a LOT of propellant.

In short, we were in serious trouble, and needed to make corrections immediately.

First, we needed to save our remaining propellant by reducing our thruster firings.  LCROSS has an Attitude Control System (ACS) to control its orientation automatically, by firing thrusters (LCROSS has no reaction wheels).  The ACS controls each of the axes (roll, pitch and yaw; see the diagram in “First Orbit  Around the Earth”) to within an acceptable error bound, defined by a control “deadband” from a fixed orientation in space.  The ACS has several modes, all with different characteristics.  The mode we were in, “Stellar Inertial Mode, deadband 2” (or “SIM DB2” for short), controlled each axis to a +/- 0.5 degree deadband.  Through the anomaly, LCROSS was keeping to within its assigned deadband.  Under attitude control with a deadband, thrusters typically fire most often when the spacecraft rotates to the very edge of the deadband, causing the spacecraft to stop its rotation and preventing it from exceeding the deadband limit.  So, in an effort to slow down the firings (and our propellant consumption), we switched to a wider deadband, SIM DB3, which controls to a 10 degree range.  Unfortunately, this didn’t reduce our thruster firing rate very much.  We needed to do something else.

The anomaly had caused LCROSS to automatically switch from using the IRU to the Star Tracker (STA) for measuring spacecraft rotation rates. However, when we studied the output of the IRU, it was producing good rotation rate data (or just “rate data” for short).  So, we ran through the IRU recovery procedure to re-engage the IRU with attitude control, and sure enough, that finally returned the thruster firing profile to normal and bought ourselves time to think.

One thing was immediately clear – we had suffered a significant “anomaly”, or bad problem and our plans for a Cold Side Bakeout and Omni Pitch maneuver were dashed.  We’d be spending the rest of the shift, at least, and possibly a lot longer, figuring out what had happened and how to prevent it from ever happening again.

With the spacecraft stable, but still in jeopardy, the next order of business was to extend our DSN coverage.  The Mission Operations Manager, with full agreement from the rest of the Flight Team, declared a “Spacecraft Emergency” with the Deep Space Network.  Under DSN guidelines, we could only call this if we thought the spacecraft was in immediate jeopardy of partial or total mission failure.  Under an emergency declaration, all missions using the Deep Space Network volunteer their normally-scheduled antenna time in a community effort to help the ailing mission.  Their help provided LCROSS with enough antenna time to work out its problems.

As a step to protecting LCROSS, the Flight Team had to figure out what caused the anomaly.  Since the spacecraft was healthy on our last contact, the anomaly had occurred while LCROSS was in a normally-scheduled 66 hour “out-of-view” period with the Deep Space Network (DSN).  The Flight Team could not collect telemetry during that time.  However, just for this purpose, LCROSS constantly records a part of its telemetry onboard, to enable our team to diagnose problems that happen while out of contact.  We set immediately to “downlinking” and analyzing our virtual recorder telemetry data to gather clues.  

Using this data, and over an hour or two of analysis, our team gradually pieced together the story.  As with most anomalies, this one stemmed from multiple vulnerabilities whose combined harmful effects had not been anticipated.

In summary, a spurious, short-lived error on the IRU was interpreted as a more serious fault by the spacecraft fault management system, resulting in a switch to a backup rate sensor (STA). That rate signal was noisy (including random variations over the “true” rate signal), but was misinterpreted as real, “clean” rate data, causing over-control by the attitude control system and resulting in a great deal of propellant consumption.  LCROSS detected the associated tank pressure drop, but with no fault management option available for disabling thruster control (thrusters are required to keep the LCROSS solar array pointed to the sun), and no ability to determine the specific nature of the pressure loss, the spacecraft fault management system performed steps to stop a leaking thruster (another potential reason for a pressure drop), and to power-up its transmitter to “phone home” to warn the operations team that there was a problem.  However, this call could not be detected over the Southern hemisphere, since there were no DSN assets that could “see” LCROSS in that location, so the call was missed.

The LCROSS team knew that it had a serious problem on its hands.  Did we have enough propellant to complete our mission?  Was the IRU truly healthy, or would it fault again and trigger another loss of fuel?  Thanks to the DSN Emergency Status agreement, LCROSS was able to get continuous ground antenna coverage, whenever the spacecraft was in view of one of the three DSN antennas.  However, given our position in our Cruise Phase orbit, only the Canberra antenna complex could actually “see” LCROSS at that time, and only for 15 hours per day. 

• To increase the “persistency” of the IRU fault check by flight software.  By requiring an IRU fault to persist for 5 consecutive seconds rather than 1 second before tripping to STA, we hoped to avoid switching to the STA for an inconsequential IRU fault, yet remain protected against a serious IRU fault.
• To augment the IRU fault response by automatically recovering the IRU (as we had done manually) first, before switching to STA for rate information.  Even if an IRU fault lasted for 5 seconds, this second-tier change would try to re-instate the IRU first, and fail over to the STA only if this had failed.

We designed, implemented, tested and loaded these changes to the spacecraft in our remaining time.  By 1:50 PM Pacific Daylight Time, with these temporary fixes in place, we began configuring LCROSS for our forced DSN outage.  Just 10 ½ hours after discovering our problem, we had to release our hold on LCROSS again and anxiously look to the next time when we would reacquire communications with the spacecraft.

August 23 – September 3 (DOY 235 – 246): Our Recovery

• Improve our fault protection against an IRU failure for the long term.  We had short-term fixes in place from the first day, but we ultimately needed to implement a more complete solution.  These would take longer to design and test.
• Improve fault protection against further excessive propellant loss.  This was a core issue.  The loss of the IRU was one potential vulnerability, but there might be others we didn’t yet know about.  LCROSS could not tolerate another similar loss of propellant.
• Determine whether the IRU was really showing signs of failure, or whether it was actually fine.
• Assess our propellant margin.  Could we complete all of our mission objectives? Would we have to give up some of our planned activities to save propellant?
• Develop a plan for LCROSS health monitoring for the remainder of the mission.  With so much at stake, we felt we needed a way to regularly monitor LCROSS after we emerged from Emergency Mode.  However, with a team as small as ours, watching the spacecraft 24 hours a day for the rest of the mission would exhaust us.  Besides, we needed to devote a lot of time in the coming weeks to prepare for impact.
• Continue performing the nominal events that could not be put off until our anomaly resolution was complete.

Propellant Usage Monitor and Free-Drift Mode

Our first priority was to develop fault protection against excessive propellant consumption.  The sure way to save propellant is to not fire thrusters.  But firing thrusters is the only way to maintain attitude control, and that is critical for generating solar energy and to keep the spacecraft thermally stable.  It seemed we were in a bind.  But then we recalled a strategy the team had originally designed to fight large Centaur gas leak torques (see the post entitled “Real-Life Operations: Day 3” for a description of the Centaur leak issue). 

The attitude control system has a special safe mode called “Sun Point Mode” (SPM) that points the solar array at the sun using special Coarse Sun Sensors, and without the use of the Star Tracker.  SPM spins LCROSS very slowly, end over end, but keeps the solar array pointed at the sun at all times.  SPM still requires thruster control to maintain stability. However, through simulations, our engineering team discovered that by modifying SPM to spin faster, LCROSS would have enough angular momentum to keep it stable for a very long time.

We still needed a means to detect excessive propellant usage.  We deemed propellant tank pressure measurements a little too coarse for such an important job, and yet there was no other single, direct indicator of propellant usage in the system.  However, one of our team members came up with a great idea. LCROSS generates and stores special telemetry “packets” each time a thruster is fired.  These packets accumulate over time in spacecraft memory, and are downloaded to Earth for analysis.  His idea was to monitor the accumulation of thruster telemetry packets as an indirect indicator of propellant usage.  It was unconventional, but the idea worked.

Efficient Control under Star Tracker Rates

IRU Healthy

Enough Propellant for Full Mission Success

There is no way to directly measure the amount of propellant left in the LCROSS tank.  One can estimate the remaining propellant load by measuring pressure, temperature and tank volume (and compute the result using gas law equations), and alternatively one can estimate the propellant consumed through the mission by measuring the accumulated on-times for all the thrusters, along with predictions of how much propellant each firing consumes. Neither method is perfect. 

The Mission Goes On

During our anomaly recovery, we had to support a number of activities that couldn’t wait until later.  The day after our discovery of the anomaly (DOY 235, August 23), orbit geometry dictated we needed to rotate our spacecraft to re-point our omnidirectional antenna toward the Earth again, or suffer very poor communications for many days to come.  We performed another of these 10 days later on DOY 246 (September 3).

In a bit of good news, TCM 5c, originally scheduled for DOY 236 (August 24), was determined to be unnecessary.  As with so many TCM’s in our original plan, the Mission & Maneuver Design and Navigation teams planned TCM 5a so accurately that neither of the follow-up burns (TCM 5b and TCM 5c) needed to be performed.

However, not all of our news that day was good.  Also on DOY 236, we thought we had also experienced signs of a failing Star Tracker.  Out of concern for its survival, we shut down the STA for two days, and spent time away from our main anomaly recovery developing protections for the STA.  As it turns out, the unusual STA signature was related to an onboard clock calibration we had just performed.  The STA was just fine, but you can imagine, in the midst of the tension of the anomaly recovery, this extra scare didn’t help matters!  As in so many cases with space flight, even seemingly benign changes can lead to unexpected results.

Negotiations with Deep Space Network and other Missions

To address the need to monitor LCROSS more often after our exit from Emergency Status, the LCROSS project negotiated a schedule with the DSN mission community that shortened all out-of-contact periods to 9 hours duration or less.  This came at great impact to other missions, but was invaluable to LCROSS in achieving its Emergency Status exit criteria. 

DOY 247 (September 4): Emergency Over

Twelve days after the anomaly discovery, having augmented LCROSS fault protection and our contact plan with the DSN and other DSN-supported missions, we requested that DSN remove LCROSS from Emergency Status. 

One of the gratifying aspects of this effort was how the LCROSS team responded to make sure LCROSS stayed safe.  Members of the Flight Team, LCROSS project and NASA Ames worked tirelessly.  Northrop Grumman mobilized to provide a number of valuable improvements to LCROSS.  The DSN mission community generously volunteered their antenna time to support our bid to restore LCROSS to nominal status.  NASA management, from Headquarters to NASA Ames, provided constant support and valuable assistance. 

With our first orbit around the Earth behind us, the Flight Team team looked optimistically ahead to a number of upcoming events, including two science instrument calibrations, and two Trajectory Correction Maneuvers, TCM 5b and 5c.  Thankfully for us, the schedule of planned events for the next 34 days to be a little more relaxed than for the previous 42 days.  The key distinction being the planned events.

August 1 (DOY 213): A Look at Planet Earth, and a “Discovery” of Water

Before the mission began, our team had identified key geometric opportunities for performing our Earth and moon-looking science instrument calibrations.  One of the important factors in determining the best time is spacecraft-to-target distance.  The closer the Earth and moon are to LCROSS, the better resolution we can attain, and the stronger the signals will be (hence why Lunar Swingby was so important).  The below plot shows the distance between LCROSS and the Earth (light blue) and LCROSS and the moon (dark blue), over the entire mission.  The labels indicate when specific LCROSS events happened (or will happen), including launch, Lunar Swingby, Earth Look Cal 1, Earth Look Cal 2, Moon Look Cal, and Impact.  As you can see, the Earth Look events coincide with low points on the curve, while Moon Look timing was influenced by other factors.

Another factor is “phase” with respect to sun illumination on the target.  Both the Earth and the moon go through cycles depending on the relative positions of the sun, the Earth or moon, and the observer, in this case LCROSS.  So at various times, the Earth will look like a crescent, at other times will be “full” (like a full moon) and still other times will be “new”, with its face completely un-illuminated.  For science measurements, our Science Team tended to prefer “fuller” views of the Earth and moon. 

As planned prior to launch, Earth Look started by rotating the payload boresight (pointed along the LCROSS –X axis) to point at the center of the Earth, then repeatedly “swept” the boresight in alternating East-West and North-South passes across the Earth’s disk.  The objective was to collect spectra of the known Earth signal (for spectral calibration) and also re-affirm the alignment of the instruments.  Similar to the limb crossings in Lunar Swingby (see “Lunar Swingby: Development of a Procedure”) by sweeping the cameras across the disk, the Science Team could compare the timing of the rising and falling signal with the well-known motion of the spacecraft relative to the position of Earth.  Consistent mismatches in timing would indicate a misalignment. 

We collected some cool photos of the event, included here.  Note that our instruments are optimized for lunar impact and finding water, so you can’t make out the landforms in the visible images, and the IR images detect thermal differences in the upper atmosphere.  It was really exciting to be looking back at our home planet from so far away.  In contrast to images taken from the Space Shuttle, International Space Station, or even Apollo, our images really show how insignificant the Earth is in comparison to the immensity of space.

LCROSS, via its spectrometers, also “discovered” water on the Earth – an event that didn’t exactly make headlines, but made the Science Team quite happy.  Detailed analysis of the spectra and imagery indicated that all of our instruments were all performing very well.

In an unrelated piece of good news, the Mission & Maneuver Design team determined that TCM 5a had been so accurate that TCM 5b would be unnecessary.  Originally, TCM 5b would have incorporated another Omni Pitch maneuver, but now with TCM 5b unnecessary, we’d perform a standalone Omni Pitch instead.

August 6 – 7 (DOY 218, 219): Re-Pointing our Antenna – Omni Pitch 3

Nearing the “top” of our orbit (see the flight path diagram), it was time again to perform an Omni Pitch maneuver to re-point the primary omni antenna towards the Earth.  These times are tricky because at the time of the maneuver, neither the initial nor the final orientation provide very good angles to the omni antenna.  You can plan to perform the maneuver early (favoring the initial attitude, but making the final attitude margins really bad), or perform the maneuver late (favoring the final attitude).  Omni Pitch 3 favored the final attitude.  We lost telemetry data sporadically before it began, indicating very poor link margins (downlink rate at our lowest rate of 2 kbits per second or kbps), but then had sufficient margin to boost our downlink data rate to 32 kbps for post-maneuver virtual recorder playbacks.  Throughout the mission, the Flight Team constantly has to evaluate LCROSS link margins to make sure we can command the spacecraft and monitor maneuvers via telemetry.

August 8 – 13 (DOY 220 – 225): Good and Boring

The mission operations philosophy states that “a boring spacecraft is a good spacecraft.”  This is certainly true because “exciting” usually means risky, dangerous and proximity to failure.  As a Flight Team member, you really don’t want to be responsible for a mission disaster!  Despite the boredom that sets in on those uneventful days, slow and steady is a good thing.  Thankfully for us, LCROSS was behaving very well, and occasionally we found ourselves in long stretches of time between planned events.  During those times, the Flight Team simply monitored spacecraft health, and hoped for boredom.

Cruise Phase health monitoring passes typically include the following steps:

August 14 (DOY 227): Opportunistic Science and Design on the Fly

During the mission, the Science team and Engineering team occasionally requested activities that were not on the original schedule.  The Science team wished to find an opportunity to image a point source with a strong infra-red (IR) signal in order to work on a solution to an Mid-IR 2 camera focusing issue. The considered Venus and Jupiter, and though both were very bright in the visible spectrum, neither provided a very good IR signature.  Working with the Mission & Maneuver Design team, the Science team discovered a golden observation opportunity – on August 16, just as LCROSS would be passing through orbital plane of the moon, the Earth and moon would appear only a few degrees apart from each other (see the plot of the angle between the Earth and the moon, as viewed by LCROSS).  The moon has known IR reflective properties, and because the moon would be on the opposite side of the Earth from LCROSS, it would approximate a point source (look very small).  About the only problem with the proposed timing is that LCROSS would be very far from Earth (see the “Special Earth/Moon Cal” in the above plot of Earth and moon distance.  However, our team considered the opportunity, and decided to go for it.

With a simple mission like LCROSS, a Flight Team is wise to pre-plan as much as possible before launch.  As the saying goes, “the devil is in the details”, and that certainly holds true when piecing together command sequences and operational procedures.  We worked very hard before launch, without the pressure of a flying spacecraft to care for, to pre-plan, construct and test commands and procedures for each event in the mission plan. 

This new activity, dubbed “Special Earth/Moon Look” didn’t match any of our pre-planned maneuvers, so we had to design it on the fly.  We designed the overall approach in a meeting between the Science team, Mission & Maneuver Design Lead, Sequencing Lead, Systems Engineers, Flight Directors and Flight Controllers.  To the credit of the team, especially the Activity Planning & Sequencing Lead (John Bresina), we were able to create command a sequence, adapted from our Star Field Calibration sequence, that could accomplish the goals of the activity.  One major difference was that we’d be looking at the Earth and moon (two targets) instead of a star field (one target), so it was like merging two Star Field Calibrations together.  There were a lot of other changes at the detailed level to make this work.  The Engineering Analyst had to extend his automated checking tools to work on this new command sequence.  The Simulation Engineer also simulated the commands on the LCROSS simulator to make sure things were going to work.  Within a few days we had a set of command products ready to go.

A day before the event, to save time on the scheduled observation DSN pass, we loaded the new command sequences to LCROSS and hoped we’d gotten everything right.

August 16 (DOY 229): Family Portrait: Observing the Earth and Moon

Our Special Earth/Moon Look Cal went off pretty much just as it should have.  We collected images of both the moon and the Earth on all of the cameras, and met the full objectives set out by the Science team.  There were some minor issues with payload heaters, and a small command sequence timing error, but we were satisfied with the results.

The photo here shows a family portrait taken by the LCROSS visible camera.  At the time of the calibration, the LCROSS was approximately 520,000 km from Earth and 881,000 km from the moon.

August 19 (DOY 231): Re-Scaling our Faulty Coarse Sun Sensor

Recall from my previous post (“Our First Orbit Around the Earth”), one of our Coarse Sun Sensors (CSS 1) had been producing low signal outputs since Day 1 of the mission.  Now “South” of the moon’s orbit plane, on the second half of our second orbit around the Earth, we took the opportunity of an open DSN contact period to finally load parameters to the spacecraft that would effectively re-scale the output of CSS 1 to have it match the output levels of the other CSS’s.  This closed out another of our original spacecraft anomalies, and left LCROSS even safer.

August 20 – 21 (DOY 232 – 233): Planning for Cold Side Bakeout 3, Omni Pitch 4 and TCM 5c

We had a challenging few days ahead of us.  The Navigation team had identified another good opportunity to perform a third Cold Side Bakeout (unfortunately on a Saturday).  Then on Monday and Tuesday, we’d have a back-to-back execution of TCM 5c and another Omni Pitch maneuver.  We generated, tested and reviewed commanding products for all three events on Thursday and Friday.  Excited to observe our next Cold Side Bakeout, I elected to take the weekend DSN pass so that I could oversee the maneuver.

August 22 (DOY 234): Innocence Lost

I came onto shift on Saturday morning, very excited for a difficult, busy shift in overseeing Cold Side Bakeout.  This third repetition was planned to look a lot like the first one, which ended up being pretty exciting and somewhat complicated.  After the standard coordination with our Canberra antenna, we prepared for Acquisition of Signal or AOS.

We acquired telemetry, and ran through our standard set of health checks, but it was immediately clear that LCROSS was in trouble.  Inertial Reference Unit (IRU) faulted, Kalman Filter faulted.  Attitude control system using the Star Tracker for body rate information.  Propellant tank low pressure fault.  Thrusters firing nearly continuously.    

It’s hard for me to believe, but LCROSS is now only 18 days from impact.  After a long time away from the LCROSS Flight Director’s Blog, I wanted to catch you all up in the amazing journey LCROSS and the Flight Team have been through in the time since I last wrote. 

On my last post (from July 22), LCROSS had not yet completed its first orbit about the Earth.  We are now on our third and final orbit.  Since then, we have observed the Earth and moon from various distant vantage points in the orbit to calibrate our science instruments.  We’ve baked a lot of ice off of the surface of the Centaur upper stage.  We orbited the Earth freely, with rarely a problem at all, and those only minor ones.  Then in late August, nearly tragically, LCROSS encountered a major problem, which resulted in the loss of most of its propellant reserves, and could very easily have resulted in the end of our mission.  You’ve read the headlines, but I’m happy to report that we’ve climbed our way out of that very precarious position, and are now poised to meet our full set of mission objectives, with less than three weeks remaining until impact.  The Science Team recently selected the impact target – Cabeus A1 – and the LCROSS team is now making final refinements in the orbit to make sure we hit that spot.  This is an exciting time.

In this post, I’d like to provide a summary of the events following Lunar Swingby on our first orbit around the Earth.  The next post will cover our second orbit and the lead-up to our “anomaly”.  Then in following posts I’ll detail the anomaly, and provide some running commentary in the lead-up to impact. 

By Deep Space Network convention, our mission uses a “day-of-year” reference for dates, rather than standard calendar months/days.  So, January 1 is DOY 1, and December 31 is either DOY 365 or 366 (depending on whether it’s a leap year or not). 

I’ll provide both for you here, so that you can interpret the orbit diagram I’ve included.  Each of the below events are shown along the flight path LCROSS took in its first trip around the Earth.  Each major “tick” mark is a 24-hour day.  Transfer Phase events are shown for reference.

June 23-25 (DOY 174 – 176): Taming our Thruster Thermal Control Problem

The Flight Team was working at full capacity during the first five days of the mission (Transfer Phase), with 24-hour operations, and major events happening daily.  It was an impossible pace to maintain indefinitely with such a small team.  But before we could ease off to our 4-out-of-72-hours contact schedule planned for Cruise Phase, we had to protect against some of the problems we encountered early in the mission, and build confidence that LCROSS would remain safe long-term. This took some extra work.

Just after Lunar Swingby, five days into the mission, we were still fighting our newly-discovered thruster thermal problem (see “Real-Life Operations: Day 3” post).  In a temporary move to keep them above the freezing point of our hydrazine propellant, our team watched thruster valve temperatures for T1 and T7 (both 5 Newton attitude control thrusters – see diagram), and fired specific sets of thrusters via ground commanding if we saw any of them get too cold.  This was not an effective long-term solution – we commanded the sequence from the ground many times per shift (very crew intensive), and this was also a waste of propellant.  But by the end of Day 6, we had loaded our first automated solution (no human intervention required) that performed the same sequence of thruster firings if either of the cold thruster valves fell below 7 degrees Celsius.  Though we refined the solution over the coming weeks, establishing this “safety net” was a major milestone in freeing up the Flight Team from having to staff 24 hours a day to keep LCROSS safe.

In a parallel effort, our team worked to find an orientation for LCROSS that would keep our cold thrusters warm using the sun, without firing thrusters at all – intended as a primary means of thermal control.  By June 25, we settled on a “yaw” bias of -20 degrees, tipping the “top” of LCROSS towards the sun slightly to warm the cold thrusters (T1 and T7 – see the figure).  We’re still using the same sun-relative orientation for LCROSS today, and plan to use it for the rest of the mission.

Thankfully, in the midst of this work, the Navigation and Mission & Maneuver Design teams informed us that we could skip our first post-Swingby Trajectory Correction Maneuver, TCM 4a.  It was always optional, but it was great to hear that Lunar Swingby was so accurate that performing this burn was unnecessary.

June 26 – July 14 (DOY 177 – 195): Gaining Confidence

Since our mission wasn’t certain how quickly we could transition to our Cruise Phase schedule, we grabbed as much leftover time on the Deep Space Network (DSN) as we could in the early days of the mission to allow a “gentle” transition to hands-off operations.  This wasn’t easy on the Flight Team – odd hours, numerous contacts, and lots of planning to do on the side.  We were anxious to scale back, but only when we could prove LCROSS would remain safe during long periods out of contact. 

In our new -20 degree yaw-biased sun attitude, LCROSS never triggered the automatic thruster firings to keep our two cold thrusters warm.  This was great news.  With automatic thruster thermal control showing promise, we moved to retire some of our other concerns with LCROSS.  Recall that the Centaur upper stage was leaking some of its residual propellant, and causing LCROSS to “fight” the resulting torque with thruster firings each day (again see “Real-Life Operations: Day 3”).  Well, our team observed a gradual but steady decline in the daily thruster firings to counteract the Centaur leak.  We were using only 0.24 kg/day at this point, easily sustainable for the remainder of the mission with our substantial propellant margin (warning: foreshadowing).  Good news for propellant usage, but the reduced thruster firing frequency also caused our cold thrusters to get colder!  Space operations are never simple.

We also got to skip Trajectory Correction Maneuver (TCM) 4b, scheduled for June 30 (DOY 181), our second skipped maneuver of the mission. Our orbit remained right on track stemming entirely from the accurate targeting of Lunar Swingby.

As we gained confidence, we eased off of our DSN contact schedule, de-staffing some contact periods, and releasing others.  This finally gave our team some much-needed down-time.  We also started figuring out our operational “rhythm” – what shift schedules worked and didn’t work, how to communicate information over the team without regular shift handovers, etc.  Simulations only go so far, and there’s nothing like flight experience to understand how things really work.  We were learning every day.

July 15 (DOY 196): Thawing the Centaur

As I describe in “Introduction to Cruise Phase”, we had anticipated that water ice might accumulate on the exterior of the Centaur prior to launch and remain there throughout the mission.  It was essential that we remove as much of that ice as possible prior to Impact, to avoid having it interfere with both our trajectory and water measurements at Impact.  Mid-July provided a great opportunity for our first attempt.

The plan for Cold Side Bakeout (CSB for short) was pretty simple.  From our sun-pointed (but yaw-biased) orientation, we would command the spacecraft to first remove the yaw bias, then to slowly rotate 180 degrees about the pitch axis (the long axis of the Centaur – see the figure) to point the solar array directly away from the sun, and the “cold side” of the Centaur directly towards the sun.  We would remain in that orientation for one full hour to warm the Centaur surface and to bake off as much water as possible.  At the end, we’d rotate back to our yaw-biased Cruise attitude and continue the mission. 

The risky part of CSB is that we’d be rotating the spacecraft in exactly the wrong direction for the Power and Thermal design, with the solar array dark and cold, and normally hot electronics equipment panels straight into sunlight.  Analysis indicated we could only tolerate this thermally for two hours.  We had to be on the alert to terminate the maneuver if anything heated or cooled more quickly than expected. 

The really interesting part is that our Navigation team selected July 15 for its particular geometry (with LCROSS passing through the ecliptic plane – the plane of Earth’s orbit about the sun) that would make it possible to actually measure the tiny change in LCROSS speed resulting from water escaping from the Centaur surface!  We were very curious to see what the measurements would tell us.

In practice, Cold Side Bakeout was a very interesting event.  Halfway around to the bakeout attitude, our Star Tracker (STA) suddenly dropped into Standby Mode just as it had before TCM 3 when we inadvertently pointed it at the Earth (read “Multi-Tasking: Day 4”).  We’re uncertain what caused the STA to trip to standby mode (it was not pointed at the Earth, moon or sun), but an enticing theory is that escaping water vapor from the Centaur scattered incoming sunlight and confused the sensor (though it may also have been sunlight reflected off the inside of the conical shroud protecting the STA).  Magically, as we pointed the Centaur cold side to the sun, the Navigation team noticed a Doppler shift in our ranging signal, indicating a change in our spacecraft velocity.  The Engineering team also noticed a change in the attitude control behavior, presumably due to a torque induced by escaping water vapor.   On its final assessment the following day, the Navigation had some startling news.  The impulse induced by the escaping water was actually 3.5 cm/second, nearly 3 times more than expected, and the effect did not dissipate over the hour-long bakeout.  This meant one thing – the Centaur was carrying lots of water, and we’d need to perform yet another Cold Side Bakeout in the future.

In a great act of forethought, one of our Flight Controllers, Matt D’Ortenzio, predicted that exposing all of our back-side thrusters to the sun during the CSB might actually prevent the thruster heaters from activating themselves later.  In our original implementation of the automated thruster thermal protection monitors, we only protected the valve temperatures for the two coldest of eight (T1 and T7) attitude control thrusters (all others were warming via heaters).  On the prediction that we might have trouble with others, we loaded similar monitors for the remaining six.  This proved to be a good call.

July 16-17 (DOY 197 – 198): Omni Pitch Flip and Thruster Thermal Flare-Up

On this day we made our second “Omni Pitch Flip” of the entire spacecraft (about the Pitch axis – see figure) to reorient the primary omnidirectional antenna toward the Earth again (see “Introduction to Cruise Phase”). 

Before we executed the Pitch Flip, however, we found that our automated thruster warming sequences had fired 70 times while we were out of contact with LCROSS.  The thruster 2 heater, which had up to this point properly activated to control the T2 valve temperature, had stopped activating.  The thruster now depended on our thruster firing “safety net” to avoid freezing.  Matt’s prediction of CSB had come true.  We were worried that perhaps we had induced a thermal instability – that the thruster would never settle back into its nice heater-based thermal control cycle.  It took another day before the T2 temperature equilibrated, allowing its heater to warm the valve without using additional propellant.  In the final assessment, we used on the order of 2.1 kg of propellant to fight the thermal issue after Cold Side Bakeout.  CSB had its cost.

July 18 – 20 (DOY 199 – 201): Steady as She Goes

LCROSS stayed healthy as we periodically monitored telemetry, downloaded and reviewed telemetry data stored onboard while out of contact, and performed a clock calibration.  Our thrusters continued behaving well, and we were burning less and less propellant in fighting the Centaur leak.  Things were looking great!

July 21 (DOY 202): Trajectory Correction Maneuver (TCM) 5a

As I described in “Welcome to Cruise Phase”, the LCROSS mission plan contained two major “delta-v” maneuvers, or changes in velocity, that had to be performed.  The remaining planned TCMs were “non-deterministic”, to be performed only as needed to clean up the errors stemming from previous maneuvers (e.g. minor mis-pointing, thruster performance modeling inaccuracies) and due to slight mis-prediction of effects on-orbit (e.g. solar radiation effects, effects of imbalance between attitude control thrusters).  We had been able to skip TCM 4a and 4b because of how well things had been planned and predicted by our Navigation and Maneuver Design teams. 

TCM 5a, however, was our second “deterministic” burn, meaning it had to be done.  TCM 5a, which used the larger 22 Newton thrusters, increased our velocity by 21.1 m/s, and was the biggest “delta-v” maneuver of the mission.  It went off just as planned, and put us on a collision course with the moon.

July 22 – 29 (DOY 203 – 210): Health Monitoring and Maintenance

Most of our time in this week was spent monitoring LCROSS health, reviewing both real-time stored telemetry data.  Because of onboard clock drift effects, we also re-calibrated the clock again to have it better match ground time.

Aside from monitoring, we loaded a new command sequence to the Data Handling Unit to control science instrument sampling for our upcoming Earth Look Calibration, our first payload calibration since Lunar Swingby.  We also downloaded the contents of all 10 of the DHU command sequences, to make sure none had been affected by radiation effects.  The Payload team determined everything was fine.

On Day 1 of the mission, we noticed that one of our Coarse Sun Sensors (CSS 1) was outputting abnormally low readings.  LCROSS uses the CSS’s to automatically point the spacecraft solar array toward the sun in Sun Point Mode (SPM), our “safe mode” for more severe fault management responses (and also the mode LCROSS “woke up” in on Day 1).  If you have a significant problem on the spacecraft, the first thing you want to do is switch to a safe power and thermal configuration.  The low output readings on CSS 1 meant degraded sun-pointing performance, potentially a risk if we ever had a serious problem.

Fortunately, LCROSS has two sets of CSS’s, called Primary and Redundant.  Upon entry to SPM, LCROSS defaults to the Primary set.  If the Primary set is malfunctioning, LCROSS will switch to the Redundant set automatically.  Unfortunately, CSS 1 was in the Primary set.  So, way back on Day 2, to avoid using CSS 1 by default, we designated the Redundant set as the default.  However, if the Redundant set ever malfunctioned, we wanted LCROSS to switch to the Primary set.  On July 23, we loaded a small flight software patch to allow LCROSS to switch automatically to the Primary set.  Meanwhile, we worked on an upload to actually re-scale the CSS 1 output to make it fall in line with the other CSS’s.  This is an example of the active maintenance that happened in various ways throughout the mission.

July 30 (DOY 211): Cold Side Bakeout 2 – Still More Ice on the Centaur

We executed a second Cold Side Bakeout maneuver at the end of our first orbit around the Earth, again near our passage through the ecliptic plane.  This time we rotated to the bakeout orientation more quickly (with high propellant expense), and faced the cold side of the Centaur to the sun for longer.  The good news is that Navigation saw a reduction of the delta-v effects due to escaping water – about 1/3 of what we saw for CSB 1.  However, there was still a substantial disturbance to our orbit, so we began looking for a third CSB opportunity in a future orbit.

All of the above happened in our first orbit around the Earth.  At this point, we were well on our way in the mission, having mastered our initial problems with LCROSS, having performed the largest TCM, and having removed a significant quantity of water from the Centaur.  However, we hadn’t operated the science payload instruments since Lunar Swingby.  This would be our first priority on the second revolution about the Earth.  In, my next post, I’ll pick up where this leaves off.

The most frequent questions I get about LCROSS go something like, “You’ve been flying for ‘x’ days…aren’t you at the moon already?”, “Why does it take so long to reach the moon?”, or “What are you doing with all the extra time between now and impact?” Admittedly, the answers are hard to communicate, because the LCROSS mission profile isn’t as straightforward as one might guess. 

Between Transfer Phase and Impact Phase is the longest phase of the LCROSS mission – Cruise Phase.  Transfer and Impact Phases are the most intense and busy, and therefore get the most attention.  Cruise Phase is when the Flight Team performs long-term preparations for Impact in every respect – we calibrate our science instruments, we better characterize our spacecraft, we finalize our lunar impact target and refine our impact trajectory.  Many of us on the Flight Team envisioned Cruise to be a lot less busy than Transfer, but our team has been steadily busy since Lunar Swingby.

Over the next two posts, I’ll introduce you to Cruise Phase, and get you all up to speed on what’s happened since Lunar Swingby so far.  Maybe in doing so, I’ll answer those questions that you’ve wanted to ask about Cruise Phase that seems, at first glance, to be nothing more than a long delay between the really cool parts of the LCROSS mission.

The Real Reason for Cruise Phase: Impact Energy

Cruise Phase is first and foremost about trajectory and impact energy, so I’ll start there.  So, why does it take so long to get to the moon, and why are we still orbiting the Earth?  Actually, it doesn’t take very long to get to the moon.  LCROSS passed by the moon on Day 5 of our mission, way back on the morning of June 22.  We could have impacted the moon back then, but our objective was to impart as much energy into our impact as possible, and it just wasn’t possible to do that on our first lunar encounter.  I’ll explain.

On our initial outbound trajectory from the Earth in the days after launch, we could have hit the moon’s equatorial region very directly, but the only way we could have impacted the lunar south pole is at a very low, grazing angle.  A low impact angle results in relatively little energy release.  Our goal is to hit our south pole target as steeply as possible, transferring all of our kinetic energy into the impact site, thereby raising the greatest amount of lunar material from the surface for analysis.  The only way to do that is to approach the moon roughly perpendicularly to its orbit plane, to smack the moon from “below”, at the south pole.  At Lunar Swingby, we used the moon like a slingshot to throw LCROSS from its Trans-Lunar Orbit, roughly in the same plane as the moon’s orbit about the Earth, into a steeply-inclined Cruise Phase orbit that would allow us to do just that.  So, we got to the moon in five days, but we never entered an orbit around the moon.  Instead, we kept on orbiting the Earth, but in a much different orbit than we started in, thanks to the help of lunar gravity as the moon flew by us.  That’s right…you read this right: when you look at the animations of the LCROSS trajectory, the moon had the larger component of velocity (as compared to LCROSS) relative to the Earth, and actually flew by LCROSS rather than the other way around.  LCROSS was at the right place at the right time to be thrown into its Cruise orbit!

 

This figure illustrates the Transfer Phase and Cruise Phase segments of the orbit, as seen for July 22, 2009.  The moon’s gravitational influence threw LCROSS into the Cruise Phase trajectory where it will remain, orbiting the Earth, until October 9.  The moon has already made a full revolution around the Earth, and LCROSS is (at the time of posting) far below the moon’s orbital plane, not quite 3/4 of the way around its first orbit around the Earth.  At the time of Impact, LCROSS will rise and intersect the moon’s orbit just at the time the moon crosses the same point.  Courtesy of NASA Ames Research Center.

Now that we’re in the Cruise Phase orbit, we’ll orbit the Earth three times before we impact.  As of today’s post, we haven’t even made it around the Earth once yet, though the moon has made one orbit around the Earth since Lunar Swingby.  Among other things, Cruise Phase is a waiting period for when the orbits of the moon and LCROSS cross paths again on Impact day on October 9.  Next time, the moon will be squarely in our sights!

Trajectory Correction Maneuvers: Fine-Tuning Impact

At special positions in the orbit during Cruise, LCROSS will perform additional TCM’s (Trajectory Correction Maneuvers) to converge on the impact target parameters – position, timing, and impact angle.  Originally we had 10 Cruise Phase TCM’s planned: TCM 4a, 4b, 5a, 5b, 5c, 6, 7, 8, 9, 10.  Some of these “maneuvers” are mandatory, others are optional.  The mandatory ones cannot be skipped if we want to hit the moon.  In the LCROSS mission, these are TCM 1 (executed during Transfer Phase) and TCM 5a.  The others are optional, and are only performed if errors remain from previous burns, or stemming from other influences (e.g. solar radiation pressure, disturbances caused by our own attitude control thruster firings, etc).  It’ll be rare that we skip a burn opportunity, but it has happened already.  We skipped the TCM 4a and 4b opportunities because we were so close to our target trajectory after Lunar Swingby that performing the burns wasn’t worth the extra risk and effort involved in doing such maneuvers. 

These burns vary dramatically in magnitude.  Earlier burns change the orbit more significantly.  By TCM 10, we’ll really be polishing things.  To give you a feel for the difference, TCM 5a was designed to alter the LCROSS velocity by 21.1 meters/second (47 miles per hour).  By comparison, our final burn will be changing the velocity by less than 10 cm/s (0.22 miles per hour), perhaps even less.  However, when you think about it, this kind of refinement is necessary.  Imagine the effect of 1 cm/s of velocity error over 10 hours of drift time prior to Impact.  The distance adds up fast: 0.01 m/s x 3600 s/hr x 10 hr = 360 meters of position error!

The fewer burns we have remaining in the mission, the more “locked in” we’ll be to a very specific impact target position and timing.  We started the mission with quite a bit of flexibility in both, but that will soon change.  In fact, the LCROSS science team has to make its final impact target selection by 60 days prior to Impact.  Any later, and the propellant cost to change to a different target becomes prohibitive. 

Science Instrument Calibrations

Cruise Phase provides some nice opportunities to continue calibrating our science instruments before Impact.  Our most valuable calibration opportunity was Lunar Swingby, executed on Day 5 as the close of Transfer Phase.  The close range to the moon meant that the moon filled a high percentage of the fields of view of our instruments, thereby improving our resolution, etc.  As I mentioned, I’ll have the Science/Payload team provide a guest posting sometime soon to discuss those results so far. 

Cruise Phase never gets another opportunity like Lunar Swingby.  We only get that close to the moon once more – on our final approach before Impact.  It’s far too late to be worrying about calibrations by then.  However, there are times on each orbit when LCROSS gets closest to the moon and Earth (making them look bigger in the instruments), and these are valuable opportunities for follow-on calibrations. 

Our mission has three nominal calibrations planned: two “Earth Look” calibrations, and one “Moon Look” calibration.  As the names indicate, we’ll train the instruments on the Earth and moon, and perform a series of sweeps back and forth across their distant shapes to improve our knowledge of instrument pointing accuracy, radiometric response and spectral response to known inputs.  The Science team’s lessons from Lunar Swingby will guide the specific measurements and focus of these activities.

Omni Pitch Maneuvers: Staying in Contact

The Cruise Phase orbit takes LCROSS high above and far below the moon’s orbital plane on each revolution.  Our standard spacecraft attitude for Cruise places our long axis (the one common to LCROSS and the Centaur) perpendicular to the ecliptic plane – the plane of the Earth’s orbit around the sun.  There’s nothing particularly special about this orientation, but it makes things a lot easier to think about operationally, and that’s worth a lot.  The catch is that our primary low-gain (omni-directional) antenna points in one direction off one side of the spacecraft, and it’s not always pointed toward the Earth.  It enables communications over about a full hemisphere of angle, but as we orbit the Earth without changing orientation, the Earth will move into and out of the antenna pattern.  When Earth is out of the pattern, we can’t communicate with LCROSS (note that we have a secondary omni antenna on the opposite side of the spacecraft, but it’s less effective, so we try to stay on the primary whenever possible). 

 

This figure depicts the LCROSS shepherding spacecraft and the Centaur, shortly after separation.  The spacecraft depiction shows the location of the primary omni-directional antenna and the solar array, relative to the standard body reference frame axes.  Omni Pitch Maneuvers rotate the spacecraft about the Pitch axis to re-orient the primary omni antenna.  The Cold Side Bakeout maneuvers rotate the spacecraft 180 degrees about the Roll axis to orient the cold side of the spacecraft (and Centaur) towards the sun.  Artwork courtesy of Northrop Grumman.

The remedy is to flip the spacecraft 180 degrees, keeping our solar array pointed at the sun, but turning the omni antenna pattern to the opposite direction.  At the time of the flip, neither the starting nor the ending orientation is very good for communications.  However, if left in the starting orientation, the Earth would move out of the antenna pattern.  By flipping the antenna (and the spacecraft) over, the Earth will tend to move deeper into the pattern over time.  These flips are rotations about the spacecraft “pitch” axis, and so are officially called “omni pitch maneuvers”.  During Cruise Phase, we have to perform these roughly every two weeks, though the timing depends on the specific orbit geometry.

Cold Side Bakeout: Getting the Water (and Other Stuff) Out of the Centaur

The Centaur upper stage (our impactor) is covered with material that is naturally absorbent and was expected to absorb some water from the atmosphere when sitting on the pad in the humid conditions in Florida.  In fact, if you watch the LCROSS launch video closely (the Centaur-mounted camera view), you can see ice accumulated in ridges on the vehicle surface, frozen by the extremely cold temperatures caused by the cryogenic contents (liquid oxygen and hydrogen) of the Centaur propellant tanks. 

In space, this accumulated water might melt or sublimate (a straight conversion from ice to vapor) if exposed to the extreme heat of the sun.  This is what we expected the water to do on the side of Centaur aligned with our spacecraft’s solar array, since that side is almost always pointed toward the sun.  After several days, we expected all of the water on the sun side of the Centaur to be removed naturally. 

Water trapped within the very cold foam on the opposite side of the Centaur, however, might remain for a lot longer, perhaps through the end of the mission.  In fact, Lunar Swingby spectrometer measurements show a significant water signal (along with other exotic hydrocarbons) – given the dry mid-latitude targets we were measuring, the water must have come from the spacecraft, and the strongest possibility is that it was from ice debris floating around the spacecraft.  There are two problems with water trapped on the Centaur.  The more obvious of these, but actually the less problematic, is that LCROSS is trying to find water on the moon.  Bringing water to the moon from the Earth will only confuse the issue!  Our Science team is actually not so worried about this, because we’re talking about only 10 kg of water or so, and they wouldn’t expect to “see” such a small amount relative to the hundreds of tons of material LCROSS is expected to loft and the percentages (albeit small) of that material expected to be water.  Still, it’s a concern.

Interestingly, the larger problem is that water escaping from the Centaur is predicted to cause a disturbance to its trajectory (via conservation of momentum).  The worry is that in the hours before Impact, as the Centaur is descending alone to the lunar surface, its shadowed side will be exposed to sunlight, causing a release of water and a notable trajectory disturbance.  A significant release of water could turn a perfectly-aimed impactor into a poorly-targeted dud.  We’re talking potentially 100’s of meters error or more!  It’s hard to believe that water escaping the surface could cause that much harm, but welcome to space physics.

To prevent this from happening, we’ve scheduled a Cold Side Bakeout maneuver that rotates the spacecraft about its long axis to point the solar array directly away from the sun, and bakes the water right out of the Centaur in full sunlight.  Over enough exposure time, the water should vent out of the Centaur.  The challenge is that LCROSS is designed to spend all but short periods of time pointing its solar array toward the sun.  Thermal and Power subsystem flight rules prevent long dwells in this opposite orientation.

Other Engineering Stuff

Aside from giving us time to conduct all of the above activities, Cruise Phase is a chance to get more familiar with our spacecraft so that we can totally predict how it will behave at Impact.  Recall that we came out of Transfer Phase with a few “anomalies”, or unexpected problems.  The Flight Team needs time to sort through all of the issues, starting with the most severe ones, and gradually moving down to the more benign ones.

Our goal is to figure out why each of the problems is occurring, and then to come up with ways to work around the problems, or to show that their behaviors are completely benign and not worth worrying about.  Each must be considered in terms of risks against achieving our mission objectives.  Our team can only bring an anomaly investigation to closure by proving to our mission stakeholders (NASA Ames, LPRP and NASA Headquarters management) that our mitigating strategies maintain our risk of failure to acceptable levels.

Flight Team Planning, Training and Rest

As Flight Team Lead, I’m responsible for creating and modifying the short and long-term plans for the mission, including Cruise Phase.  Before launch, most of our time was spent preparing for the first week of flight.  Once Lunar Swingby came and went, I had to drop into overdrive to get our Cruise Phase planning in order.  This meant (and continues to mean) nailing down times for TCM’s and other maneuvers (courtesy of our MMDS team – see the post on “LCROSS Flight Team Breakdown”), nailing down DSN contact times (via our CLASS team at Ames and JPL), putting together staffing schedules, etc. 

Then there’s planning for Impact.  Again, our team spent the 2-3 months prior to launch working out final details on Transfer Phase operations.  We last practiced Separation and Impact in February.  We have things worked out pretty well, but now we need to practice some more so that we nail our Impact in October.  As Flight Team Lead, I am responsible for setting down the rehearsal schedule, working with the CLASS team to schedule DSN readiness tests, and refining our procedures as needed.

When we’re not planning for our next activity, coordinating on anomaly resolution, working out schedules, or training for Impact, the Flight Team has a little extra time to rest than it did before launch, and during Transfer Phase.  We’re certainly not avoiding work – it’s what you as a taxpayer are paying us to do.  But the Flight Team has been working full steam ahead for six months or more, and now that the schedule is a little more relaxed, we can afford to take some hours and days off here and there.  It’s not much, but it sure helps!

This is it for now.  In the next post, I’ll present a brief diary to catch everyone up on all the activities we’ve performed since Lunar Swingby!