What Happened on Impact Night?

The LCROSS mission was ultimately focused on the final four minutes of flight, starting at the time of the Centaur impact, and ending with the impact of the Shepherding Spacecraft.  During that time, the Payload Engineer and the Science Team took operational center stage.  Once the science payload was powered on, the team’s job was to confirm the full functionality of the instruments, and then to adjust instrument settings to make sure the data we received was the best it could possibly be.  For Impact, there were no second chances – the Shepherding Spacecraft was to be destroyed as a forgone outcome of its observation of the Centaur lunar collision. 

 

In this post, I’ve invited Payload Engineer Mark Shirley to provide his perspective of impact night.  Mark was the Payload Software Lead, in charge of the design and implementation of onboard instrument command sequences and other onboard software components, as well as a large fraction of the science-related data processing software used on the ground.  During flight, Mark sat in the Mission Operations Control Room (MOCR) with the rest of the Flight Operations team.  On the final day, his job was to assess and maintain the engineering functionality of the science payload in those minutes before impact. 

 

In the impact video sequence, from the public’s perspective, there were a lot of things that happened operationally that were probably difficult to understand.  Mark will explain the plan for gathering data from the impact and also describe what actually happened in flight.  On a lighter note, Mark was one half of our now famous (or notorious) “high-five malfunction” that created such a buzz on the social media circuit after impact.  He’ll explain that as well. I’ll let Mark take it from here.  Enjoy!

 

I’d like to describe our plan for collecting data about the Centaur impact from the shepherding spacecraft (S-S/C), how actual events differed from the plan, and what that says about the process of developing and flying spacecraft.  In particular, I’ll cover why some of the pictures were fuzzy and some were white and why we were sending commands during the last minutes.  I won’t touch on the scientific interpretation of the data, only the process of gathering it.  This story contains some hard work, a few mistakes, a little nail biting tension, and finally, success.

 

Central to the story is the type of mission LCROSS was: a cost-capped, fixed-schedule mission.  That meant if LCROSS had been late, LRO would have flown with a dead or inactive LCROSS.  If the project had run out of money, whatever hadn’t been done wouldn’t have gotten done.  Within the LCROSS project, the instruments were in a similar position.  The original idea was to observe the Centaur impact from Earth only.  Onboard instruments were soon added to the design but with a total instrument budget of approximately $2 million.  That’s much less than single instruments on many other missions.

 

Two things made success possible.  First, project managers kept a tight focus on using the barest minimum hardware and testing required to perform the science, and went beyond that, only as the budget allowed, to increase the likelihood of success.  Second, everyone stayed on budget. The payload team had no choice, but if any other part of the project had overrun by a lot, the payload might have been eliminated or flown only partially ready.

 

The LCROSS Instruments

 

LCROSS carried nine instruments.  Five were cameras to take pictures over a large range of wavelengths, that is, colors. One was for visible light that our eyes can see.  Two near-infrared cameras captured mineralogy and water signatures, and two mid-infrared cameras captured thermal signatures from -60C to +500C).  LCROSS carried three spectrometers to measure color very precisely.  One covered ultra-violet and visible wavelengths and two covered near-infrared wavelengths.  The latter spectrometers were the best at searching for water, and one looked down toward the Centaur impact the vapor cloud it kicked up and the other looked to the side as LCROSS passed through that cloud.  Finally, LCROSS carried a high-speed photometer to measure the brightness of the impact flash.

 

The instruments are described in detail here.  Data from all nine instruments had to share LCROSS’ one Mbps (one megabit or million bits per second) radio link to the ground.  At that rate, it takes 2 seconds to transmit a typical cell phone picture.  This was the maximum data rate available for the LCROSS mission and used only twice: lunar swingby on June 22nd and impact on October 9th. All other instrument activities that took place during the 112 day mission used speeds less than 256 kbps (kilobits or thousand bits per second), which was sufficient for collecting data to calibrate the instruments.

 

The Observation Plan

 

The two components of the LCROSS mission, the Centaur and the Shepherding Spacecraft (S-S/C), separated about 10 hours before they reached the moon.  At the moment the Centaur impacted, the S-S/C was still 600 kilometers above the surface.  Falling at 2.5 kilometers per second, the S-S/C reached the surface 4 minutes later.  Observations of the Centaur impact event made during those 4 minutes were the purpose of the mission.  Unlike orbital missions that can usually try multiple times to collect data, we had just one shot.

 

The diagram below shows the plan for observing from the S-S/C, starting one minute before Centaur impact, at the beginning of what we called “Sequence 2” in the NASA TV video.  The diagram plots our intended schedule of instrument observations against time: each row represents one of the instruments (instrument abbreviations appear below each row of data), and each tick mark along a row represents one observation, either an image or a spectrum.  Over some intervals, the observations are spaced so closely that the plot looks like a solid bar.


 

Figure 1 The LCROSS impact observation plan: These timelines indicate when each image and spectra was planned to occur during the final four minutes of the mission.  The horizontal axis represents time.  Each row represents an instrument, and each tick mark represents the timing of a sample (an image or spectrum) from that instrument.

 

The last four minutes were divided into three periods, called FLASH, CURTAIN and CRATER.  Each period focused on a different aspect of the expected impact event and emphasized data collection from different instruments.

 

FLASH started one minute before the Centaur impact and focused on the very short burst of light generated by the Centaur impact itself.  Starting from the top of the diagram, the plan was to stop both the Visible Light Camera (VIS) and the Near-Infrared Camera #2 (NIR2).  This would allow us to focus on NIR1 images which we felt had the best chance of catching the location of the impact flash which we expected to be visible for less than one second.  These three cameras shared a common input to our payload computer, called the Data Handling Unit (DHU), and could not be used simultaneously. By stopping VIS and NIR2, we could run NIR1 at a faster rate (see the segment labeled ‘A’), increasing the odds it would image the flash.  The planned sequence also increased the NIR1 exposure time to capture the flash signature even if it was very faint.  We knew this would produce a badly overexposed image of the illuminated lunar surface, but our goal was to locate the impact.  We’d have plenty of other pictures of the surface.  This sort of shifting attention between cameras accounts for the periods where one camera image would stop updating for a while. 

 

We also designed the FLASH strategy for the spectrometers around our expectation of a dim, short duration flash event.  Near Infrared Spectrometer #1 (NSP1), the main water-detection instrument, was put into a high-speed, low resolution mode (represented by the yellow bar).  The Visible and Ultraviolet light Spectrometer (VSP) was commanded to take long exposures, and Total Luminescence Photometer (TLP) was powered early enough to reach equilibrium and be at its most sensitive for the flash event.

 

The second phase, CURTAIN, started just after the Centaur impact and ran for three minutes.  Its purpose was to take spectra and images of the expanding vapor and dust clouds thrown up by the impact.  CURTAIN was the most important period and also the simplest.  All instruments ran in their default modes, as follows.  The DHU shifted between the three analog cameras in a stuttering pattern – VIS, VIS, NIR1, NIR2 – repeating.  Both thermal cameras monitored the plume shape and temperature. The two downward-facing spectrometers (NSP1 and VSP) looked

for water and other chemicals.  The side-looking spectrometer (NSP2) also looked for water and other compounds, but from sunlight scattered or absorbed by the dust and vapor cloud.  The TLP continued to take data during this period, but it’s primarily function was during FLASH.

 

The goal of CRATER, the final period, was to image the crater made by the Centaur impact to get its precise location and, more importantly, its size.  From its size and their detailed models of crater formation, the LCROSS Science Team can potentially tell us how the crater evolved over the few seconds of its formation and how much material was excavated.  The primary instruments in this period were the two thermal cameras, MIR1 and MIR2.  Their sample rates were increased relative to those for CURTAIN.  To image the crater in a second frequency band, NIR2, the more sensitive near infrared camera, was commanded to its most sensitive setting.  NIR1 and VIS would not be used during this period because neither was sensitive enough to see anything in the permanently shadowed area.  All spectrometers would continue running to look for light reflected off of any plume or vapor cloud.  At the end of this phase, the S-S/C would fall below the rim of Cabeus Crater, cutting off radio transmission to Earth, and then impact the surface a couple of seconds later.

There were three keys to making this plan work:

  • Downlink Bandwidth: the data collected had to fit within the 1 megabit radio downlink. We did a lot of testing before launch to work out a data collection plan that was further confirmed and refined based on on-orbit performance. We gave priority to data from the most important instrument, the near-infrared spectrometers, to provide robustness to the design. The best scheme was pre-programmed and ready to go in case we were unable to command the spacecraft in the final hour.
  • Camera Exposure: We had to change camera exposure settings during the descent to reflect the changing    brightness of the impact event and the surrounding scene.  Defaults were pre-programmed based on the latest lighting models for impact morning from NASA Goddard Space Flight Center.
  • Command Timing: In the instrument command sequences governing FLASH, CURTAIN and CRATER periods, we had to orchestrate changes in instrument configuration as they were needed to focus on    different aspects of the impact event.  Sometimes these changes had to be interleaved with instrument data    collection in a way that was vulnerable to small timing changes.

     

    So What Actually Happened?

     

    Well, as reflected in the recent LCROSS press briefing, we collected a very rich and interesting data set that met the needs of our science objectives.  However, we had challenges in all three areas – bandwidth, exposure settings and timing – although all ultimately proved minor. However, in some ways, it was a close call.  This diagram shows what data was actually collected during the final four minutes of the mission.


     

    Figure 2 Actual performance of the LCROSS payload: These timelines indicate when the images and spectra were taken on the morning of the impact.  The pattern gives clues about the performance of the hardware  and software systems that collected the data.

     

    First, the rows representing the spectrometers, NSP1, NSP2 and VSP, look almost exactly as they should.  Except for one problem with the Visible and Ultraviolet light spectrometer (VSP), which I describe later, our plan for collecting spectra worked perfectly.  This is very good, because the spectra carried most of the information we were trying to collect.

     

    As for the cameras, several differences from the plan jump out.  The most obvious is that the timing of observations along some timelines is irregular with many observations missing, e.g., the visible camera pointed to by note E. This occurred with all five cameras (the first five timelines) but not with the spectrometers (the next three timelines). 

     

    Scene Complexity and Bandwidth Limitations


    Image compression is the process of finding and reducing redundancy in an image in order to transmit it more efficiently.  On LCROSS, to fit within the 1 Mbps data rate limitation for our downlink, we used a lossy compression algorithm that typically reduced each image to 1/20th its original size.  Lossy methods achieve greater compression than lossless methods by actually removing parts of each image.  The algorithm tries to find subtle details whose removal the human eye won’t notice.  Being able to combine many different kinds of data into a single digital data stream is so useful that this approach has been standard practice for many years.


    In flight, the irregularity of observations occurred because we underestimated the complexity of the lunar scene during ground testing.  We had done much of our testing with a large reproduction of the moon’s pole in front of the cameras, but it turned out this didn’t mimic the high contrast and detail of the real scene.  Scene complexity mattered because the images were highly compressed and changes in the moon scene changed the sizes of the compressed images by a factor of 4. We first observed this behavior during the lunar swingby LCROSS performed during the first week of its mission.  Turning on the instruments during the swingby was intended as a learning experience, and it proved critically important.  It provided the best operational practice we got for the impact as well as data to calibrate the instruments.

     

    After the lunar swingby in June, I changed the thermal camera sampling rates in the instrument command sequences for the final hour.  Unfortunately, the compression problem turned out to be about 20% worse during the final hour of the mission than during the lunar swingby.  This forced us to change the thermal camera rates again in real-time, but we had practiced changing them during rehearsals, just in case.  In the NASA TV impact video sequence, you can hear the Science Team requesting a change of MIR1 rates to 1 Hz, and MIR2 to 0.1 Hz.  See note F in the figure.  The rate for thermal camera #1 (MIR1) changes just before this note and changes for MIR2 just after it.  Even though we’d practiced, this was still a very tense time as we were losing some data while the changes were being made. Changing the MIR rates felt like it took forever.

     

    The bandwidth problem could have been avoided if in addition to changing camera sampling rates in the command sequences, we had also changed the stuttering pattern for the analog cameras mentioned above to eliminate one VIS image during each repetition.  However, our instrument simulator didn’t have the full set of instruments like the spacecraft, which made it impossible to adequately test this change on the ground.  At one point, we discussed testing this change onboard before the impact, but lost the opportunity due to the fuel loss Paul described in his blog on October 4th (see “A Test of the Flight Team”).

     

    One other problem caused by the complex lunar scene was damaged images.  After compression, some of the visible camera images were still too large to fit within a single data packet for transmission to Earth.  Here’s an example of the kind of damaged image that resulted.  The shadowed area should be completely dark, but instead contains wispy bright areas. These compression-artifacts are intimately linked to the scene and need to be taken out with image post-processing.

     

    Figure 3 Example of damage to downlinked images due to clipping in the telemetry packet formatting software.

     

    What caused these compression artifacts?  I didn’t know it at the time, but the software for compressing these images had been written some years before to clip the compressed form of images to ensure they always fit within a single data packet (maximum size 65536 bytes).  We used a wavelet-based compression algorithm, and clipping the compressed images removed some information needed to recreate the image accurately.  The alternative would have been to split the images across multiple packets and reassemble them on the ground.  This certainly could have been done in principle, but doing so would have introduced significant changes right at the heart of software that we had planned to reuse without change after its successful use on previous projects.  With what we know now, changing this would have been justified, but it would also have been risky given how central image compression was to the overall design.  With what we knew at the time, I believe we would have left it alone because of our short development schedule and all the other things that had to work.

     

    November, November!

     

    Most of the commanding we did from the ground was to adjust the exposure times of the near infrared cameras as the scene changed.  The other cameras either controlled themselves (VIS) or had only one appropriate setting (the thermal cameras, MIR1 and MIR2).  We controlled the exposure setting for the near infrared cameras explicitly because we were trying to image a relatively dim flash and ejecta curtain close to bright mountain peaks.

     

    Near the beginning of the FLASH period, we discovered we didn’t get this balance right. To image the dim centaur impact flash, we deliberately overexposed the sunlit peaks.  This setting combined with the Cabeus scene overdid it. The sunlit areas electronically bled into nearby parts of the image.  That occurs when electrons in overexposed pixels move across the image detector to other pixels.  In this case, the shadowed area of Cabeus crater was completely covered, obscuring our view of the impact.  That was why the only image that was updating just before the Centaur impact was white.  We hadn’t seen this level of bleeding earlier in the mission, or in almost any of our testing.  However, after searching through our data archive, I realize now we did see it occur once, two years ago, in one flashlight test in a darkened room but did not fully comprehend the implications.

     

    The FLASH period was designed to start 1 minute before the Centaur impact, so we had a little time to recover once we saw the problem.  During this minute, our first priority was to confirm that the spectrometers (NSP1 and VSP) and photometer (TLP) were working properly.  Once that was done, we focused on the NIR1. Since we still had commanding during this period, we tried to change the exposure setting (Payload Scientist Kim Ennico called out “Flight, this is Science, please change NIR1 to OPR 9, over.”). We had less than 30 seconds to get this command sent up to the spacecraft.  The command was actually sent but arrived a few seconds too late to capture the impact. In hindsight, this was a challenging stretch for the camera’s range, due to the scene and the potential for bleeding. Our strategy – to aim for the most sensitive exposure setting followed by one attempt to back-off depending on the data-might have worked had we been looking at another region of the moon, that is, had, we launched (and impacted) on a different date, where the terrain and lighting would have been different. While all this was going on, the impact flash was captured by NSP1, so the key science measurement was made.

     

    We intentionally caused the same issue later, during the CRATER period, but we had better success (see above figure at the segment labeled ‘B’).  Initially, the NIR2 camera images were badly overexposed for the same reason as during FLASH (hence the white images that appear in the NASA TV video just after entry to DV Mode).  Kim Ennico, the Payload Scientist, made the call to reduce the exposure time slightly, from what we called OPR 15 to OPR 10.  (Again, you can hear this request over the voice loop in the NASA TV video.)  She was using live-information from the NSP1 spectrometer and checking those values in real time against a spreadsheet near her seat.  You can see her checking and rechecking on the video before making the choice.  We only had only one chance to choose the right one.  The command was sent and received 30 seconds before the S-S/C’s impact.  Kim’s call initially left the images overexposed, but as the lit peaks slid out of the field of view, her choice produced excellent images of the very dark crater floor, including the image that gave us our best estimate of the Centaur crater size.  These images go all the way down to 2 seconds before S-S/C impact where the craft was 5 kilometers above the surface. The crater floor of Cabeus was indeed brighter than any of the predictions, at least in the infrared. That’s another reminder of science and exploration. Sometimes you are surprised as you collect new data, especially data from areas never looked at before.

     

    Figure 4 This image sequence was captured just before the end of the mission and shows the NIR2 camera going from badly overexposed to acceptably exposed as the lit peaks surrounding Cabeus leave the field of view.

     

    How do we know that’s the Centaur crater?  Because it was also seen by the two thermal cameras (MIR1 and MIR2) while it was still warm, and we can overlay the images.  The left figure below shows aligned images from NIR2 and MIR1, taken before the Centaur impact.  The figure on the right shows aligned images from these cameras taken just before the S-S/C impacted and showing the Centaur impact crater (see inset).  These images don’t align perfectly because they were taken about a second and 2.5 kilometers apart.  To obtain images of the Centaur crater using three different cameras was our goal, and we succeeded.

     

    Figure 5 The right image shows the Centaur impact crater in both near-infrared and mid-infrared images.  The left images overlays images taken before the impact by the same two cameras.

     

    Kim’s call of “NIR2 to OPR 10” yielded a great image of the Centaur crater, but it also caused some confusion.  The name for Near-Infrared Camera #2 (NIR2) was too similar to the name of Mid-Infrared Camera #2 (MIR2).  We had practiced this interaction over our voice communication loops, but we hadn’t practiced it enough to do it quickly and perfectly under time pressure.  Kim had to repeat her call using the phonetic term ‘November’ for the ‘N’ at the beginning of NIR2 (during CURTAIN phase in the NASA TV video, you can hear the Flight Controller, Jim Strong, ask “is that ‘November’ or ‘Mike’?”, referring to NIR2 or MIR2, ).  We didn’t realize when we picked the obvious names for these cameras three years ago that the names could cause confusion when spoken over the voice loops connecting our mission control rooms.  Back then, our plan avoided real-time commanding completely, but as we learned through our practice sessions before launch and actual experience after launch, we realized we needed the flexibility.  We retrofitted a process for proposing and confirming real-time commands into our mission operations architecture as best we could given the facility and time constraints.  Note that we didn’t consider doing this for maneuvering the spacecraft or for the most critical science instruments, only the secondary instruments.

     

    Command Timing

     

    The commanding side of the automatic sequence ran almost perfectly.  We did have one problem with the Visible and Ultraviolet Spectrometer during the CURTAIN period, though.  Because the instrument data handling unit (DHU) was at its maximum data throughput capacity during the first part of CURTAIN, one command to change exposure time was delayed and sent during a period when the instrument wasn’t listening.  That command was ignored.  This resulted in capturing fewer spectra with longer-than-planned exposure times.  Luckily, the longer exposure times turned out to be a blessing, since the ejecta curtain was much fainter than some models predicted. The loss of more frequent sampling due to the longer exposures did not affect the science measurement.

     

    High Five Fail

     

    Yes, I should have.  After the end of the mission, I missed a high five that was captured on camera.  I was teased about it by my colleagues that morning and by my kids that night. The other operator involved, our Telemetry Data Manager, known as “Data” over the voice loop, is both a good colleague and a friend.  I didn’t intend to embarrass him and have since apologized.  We had been told to avoid high fives to prevent exactly the sort of mistake I made, but once the hand went up, I should have responded.  So, here it is, for the record:

     

    Why didn’t I respond?  I honestly don’t remember the moment clearly, but I did have two things on my mind.  First, my job at that point was to move to the Science Operations Center (SOC) to prepare for the post-impact press conference.  We had two hours to make sure we’d gotten the data we expected, to prepare presentation charts and to look for anything obvious.  I was concerned about that because I had missed seeing the plume we had hoped for like most everyone else.

     

    More importantly, I was really stressed out.  The DHU, the computer that processed all instrument data, was struggling with the very large packet sizes of visible camera images, and the DHU almost crashed a number of times during that final hour.  We had developed and practiced a procedure to recover from such a crash to prevent a substantial loss of science data.  Flying a spacecraft is a group effort with lots of cross-checking, and as the Payload Software Lead, I felt especially responsible.

     

    During payload development testing, we found and fixed several problems that would have been problematic for the payload.  This problem which led to potential crashes of the DHU was known and was the most difficult software problem we saw.  The root of the problem was a small chip that controlled the data bus connecting the video capture and compression chips to each other and to the main processor within the DHU.  Under certain circumstances this bus controller chip would stop responding, and the DHU software would crash.  Since we didn’t have access to the chip’s design to understand why it would stop, and we didn’t have time to replace it, our approach was to create a method for quickly recovering on orbit.  This method had two parts.  The first part was a software patch we developed that reset the bus controller when the DHU’s main processor noticed it had stopped responding.  The second part was a procedure for quickly rebooting the whole DHU from the ground if the software patch didn’t catch the problem.

     

    We developed and tested the software patch just a few weeks before the payload was shipped to Southern California for integration with the rest of the spacecraft. From that point on, through the rest of our testing on the ground and in orbit, we didn’t see this problem again. That is, we didn’t see it until the morning of the impact.  That morning, the patch needed to reset the bus controller two dozen times.  The vertical green lines in this figure show when.

     

    Figure 6 During the final hour of the mission, Data Handling Unit (DHU) software detected and corrected an anomalous condition on a bus controller chip multiple times.  The green lines show when these events occurred.  The right end of this figure, starting at the label ‘I’ (the time of Centaur impact) corresponds to the time spans of the planned (Figure 1) and actual (Figure 2) performance plots. Earlier events happened while monitoring payload performance in the 56 minutes prior to Centaur impact.

     

    Once these events started, I was prepared, on a hair trigger, to start the process of rebooting the DHU if the patch didn’t work.  I was constantly checking and rechecking the fault response procedure I had developed for our payload.  The details of this procedure varied over time.  As the on-board sequence progressed and we got closer and closer to the Centaur impact, we had different decisions to make to recover if something went wrong.  This strategizing was being done over another voice loop with Kim and Tony Colaprete, the LCROSS Principal Investigator, in the Science Operations Center (SOC) which was not audible to the audience watching on NASA TV.  We had to keep track of a lot of independent data threads and contingencies simultaneously.  Our actual trigger for starting this recovery, a gap in numeric sequence of the data coming from the instruments, even occurred once, but it was unrelated and didn’t need a response.  In the end, our defenses worked.  The software patch performed exactly as intended and no crash occurred.  After the S-S/C impact, I breathed a sigh of relief and moved to the next room to start preparing for the press conference.

     

    Final Thoughts

     

    As I said above, we had challenges in all three areas critical to making our plan work: downlink bandwidth, camera exposure, and command timing.  Ultimately, all of the problems we had proved minor, and we collected the data we needed to draw conclusions about the presence or absence of water and other substances in Cabeus crater.

     

    We at NASA all too often strive to give the impression that complex, difficult missions are routine.  They’re not.  They’re complex and difficult.  What makes them possible is long planning, teamwork, and careful review by people both inside and outside the project.  One name for this process is “Systems Management”, which recognizes that

    people need backup just like the parts of a complex machine.  I personally made some mistakes and caught some mistakes.  Together, we caught enough of them that we were successful.  For me, it was a huge privilege and a wonderful experience.  I’m very grateful to have been a part of this mission.

    Recap of the Final Day,Part 1: Separation and Braking Burn

    LCROSS, the flight mission, is over, but we’re still analyzing the data that was collected on impact night.  Our team is very excited about the recent announcement – that LCROSS confirmed the presence of water at the lunar south pole.  But, there’s a lot more data to analyze, both for science and engineering purposes, and I’m guessing there will be more interesting announcements in the months ahead. 

    With the completion of mission operations, before concluding my account of the LCROSS journey, I wanted to add a few more posts relating our team’s experiences in the last day of the mission.  In this post, I’ll describe my last shift in the Mission Operations Control Room (MOCR), covering the Separation event and the final delivery of the Centaur impactor.  In a near-future post (I promise!), I’ve invited our Payload Engineer, Mark Shirley, to post with a detailed look at payload operations during the Impact event.  For all of you who watched the impact video and are wondering what happened that evening, Mark’s post should be very enlightening.

    Now, a note about terminology – the two major pieces of hardware on LCROSS are the Centaur (the impactor) and the Shepherding Spacecraft (abbreviated as “S-S/C”), which carries the science instruments and that has acted as the “living” part of LCROSS since the Centaur’s batteries ran out on Day 1 of the mission.  Up until now, I’ve referred to the combined Centaur and S-S/C as “LCROSS”.  But in describing what happened after Separation, when the two pieces decoupled, I’m forced to refer to each vehicle independently.  Finally, as a reference for timeline details, check out my post entitled “Brace for Impact: A Schedule of Events for the Final Day”.

    Shift A: Release of the Centaur and Slowing Down the Shepherding Spacecraft

    On October 8, I was the Flight Director in charge of the second-to-last shift (Shift A) of the LCROSS mission.  Shift A’s main task was to perform final targeting of the Centaur, via the separation of the Centaur from the S-S/C.  The Separation event was the final influence our team would have on the Centaur impact location and time.  Soon after Separation, we were to command the S-S/C to image the Centaur as it receded from view.  Shift A was also in charge of the “braking burn”, a delta-v maneuver performed after Separation that would decelerate the S-S/C relative to the Centaur to delay its own impact on the moon, thereby improving our view of the impact event during the collection of science data.  

    Figure 1: Diagram of Separation, Centaur Observation and Braking Burn

    In all, we could not have had a better shift.  Everything went incredibly smoothly.  Here are some highlights:

    Command Loads

    Once “on console”, Shift A’s first responsibility was to load all of the command sequences to the S-S/C that would govern the initial attitude change to Separation attitude, the Separation itself, the Centaur Observation and the Braking Burn events. 

    With so much at stake, the Flight Team was very focused, and we completed those command loads very early, with no problems.  The onboard commands were designed to execute the whole sequence through Braking Burn (minus the Centaur Observation) without human intervention.  Barring unforeseen problems, the Flight Team could have walked out of the control room at that point.  But in reality, there’s no way we’d leave the success of the mission to chance.  As I’ll describe later, Separation had a lot of potential risk.  If anything went wrong, our team needed to be there to take corrective actions and to set up for Impact.    Furthermore, the Flight Team had an integral role in initiating the Centaur Observation. 

    Reorienting to Separation Attitude

    To precisely target the Centaur into Cabeus crater, our plan was to use a fraction of the velocity change caused by Separation to push the Centaur into the right impact trajectory.  The interface ring between the Centaur and S-S/C contains springs that push the two vehicles apart as soon as they’re released.  They produce up to an estimated 500 lb of force, and were predicted to induce approximately 0.7 meters/sec of relative velocity between the two vehicles.  However, since the Centaur far outweighed the S-S/C, we expected only 0.15 m/s would be imparted to the Centaur, the rest to the S-S/C. 

    But we didn’t need all of that extra velocity for a precise impact.  To use only a fraction of this added speed, we’d have to perform Separation in a specific direction, so that only the needed component of the velocity change would affect the impact position, while the other components would have no effect on impact position and little effect on timing.  Our Maneuver Design team determined, coincidentally, that this direction was only 7 degrees away from our starting Cruise attitude. 

    So, what happened?  A portion of the commands we had just loaded automatically performed the change in orientation.  We needed to take every precaution not to disturb our orbit prior to Separation.  Rather than commanding the entire 7 degree change in one step, the command sequence performed the maneuver in a long series of small attitude updates.  This minimized the number of thruster firings required, though it took over 20 minutes to complete.   The whole sequence of commands executed perfectly and with one task completed, I prepared mentally for the next.  With the rotation completed, we were less than one hour from Separation.

    Separation

    Other than Impact, Separation was perhaps the most critical event of the mission.  Since the Centaur was not independently controllable after Day 1 of the mission, the de-coupling of the S-S/C from the Centaur was LCROSS’s last influence on the Centaur impact trajectory.  As Separation approached, my adrenaline started to flow.  Here are the reasons Separation made me a little nervous:

    • Centaurs typically separate from their payloads within just hours of launch.  LCROSS’s Separation was 112 days after launch, far longer than ever done with a Centaur before.  The space thermal environment often wreaks havoc on mechanical elements, with extreme temperatures and many swings between hot and cold.  Analysis showed that the mechanism could operate over a very wide range of temperatures and after many warming/cooling cycles.  Still, we didn’t know if something had been overlooked.  The good news was that temperatures from the separation mechanism had remained well within allowable limits for the entire mission, increasing our probability of success.  However, if Separation were to fail, LCROSS would impact in one piece, the science payload would be unable to watch impact, and our mission success would be entirely reliant on ground-based and other orbiting observatories to collect impact data.  
    • The Centaur separation mechanism was designed to very reliably push the S-S/C away smoothly, without causing the spacecraft to tumble.  To prevent attitude control from interfering with Separation, the command sequence would disable ACS thruster firings briefly through the transition.  But what if something snagged or re-contacted?  Disabling the ACS would allow rotations induced by separation to go unnoticed (and uncorrected) for a few seconds.  Again, analysis predicted success, but there were a lot of unknowns here.
    • In the instant of Separation, LCROSS would transform from a 2894 kg spacecraft to a 617 kg spacecraft (about 1/5 of the mass).  The moments of inertia (the spacecraft’s tendency to resist changes in rates of rotation) would also decrease dramatically.  To remain in control after this enormous change, the ACS would use new sets of control gains optimized for post-separation conditions.  But none of these controllers had been tested on the real spacecraft (only in good simulations – they wouldn’t have worked well with the Centaur attached).  Dynamically, it would be like getting a whole new spacecraft, but with only 9 hours to figure out how it really behaved before Impact.  To make matters more exciting, our command sequence utilized 5 out of 6 attitude control modes in the first 40 minutes after Separation.  Each mode employed an independent set of control gains.  One mode might work, and another might be flawed and cause an instability.
    • Though the Centaur and S-S/C would recede from each other at very slow speed (0.7 m/s, or about 1.6 mph) after Separation, it wouldn’t take very long for the Centaur to be quite far from the spacecraft (42 meters or half of a football field farther each minute).  The Science Team really wanted to film the departing Centaur to determine whether the separation had induced any tumble into the Centaur, and to use that knowledge to better understand the Impact behavior later.  To “see” any tumble, which might be very slow, we would need to be able to distinguish the Centaur long axis from its short axes.  To do that, the Centaur would have to be close enough to span at least a few pixels in the LCROSS cameras for several minutes.  But at Separation, the LCROSS cameras would be facing away from the Centaur.  The mission plan involved flipping the S-S/C 180 degrees to point them toward the Centaur as soon as possible after Separation.  But this ran counter to the desire of our engineers, who would have preferred to watch the ACS control behavior for a while.  As a compromise, we planned on rotating just one minute after Separation.  Good for Science, but a little uncomfortable for the Flight Team, which would have to confirm a good separation, then oversee a 180 degree flip on a new set of control gains!
    • Neither the S-S/C nor the Centaur could automatically confirm that separation had actually succeeded, so the Flight Team needed to confirm success from the ground via telemetry.  A set of three small wires would break at Separation, and enable us to determine a physical separation.  Then indirectly, we expected to observe a dynamic response from the S-S/C from the push and any resulting torques.  When designing this event, we had a choice: either have the Flight Team initiate the 180 degree flip from the ground on successful Separation (but risk being late on the Centaur observation if we lost communications), or execute the flip using onboard commands and have the Flight Team terminate the sequence if Separation failed (but, if we lost communications and the ability to command, risk performing the flip with the Centaur still attached).  Mission-wide, we had adopted a policy that assumed success, so we opted for executing these commands onboard.  This meant that within 60 seconds, the Flight Team had to identify whether Separation had been successful, and terminate the running command sequence if it had failed.  Sixty seconds may sound like plenty of time, but with multiple operators on console evaluating multiple telemetry indicators, allowing time for a possible termination of the command sequence, sixty seconds was barely enough. 

    So, what actually happened? At 10 minutes until the designated Separation time, the onboard command sequence started to run.  Game time.    Months of design, weeks of training would all come together now.  I counted down the last 5 seconds until the commands to fire the Clamp Band Ordnance Device (CBOD) would have issued from the onboard sequence.  Then we focused on telemetry.   Silence on the voice loop.

    Ten seconds after the separation time, the Flight Team broke into chatter over the voice loop.  I polled the System Engineer.  He returned with “three wires, Flight”, meaning that all three wires that indicate a clean break between Centaur and S-S/C all showed positive.  I polled ACS (Attitude Control System Engineer).  He announced “GO Flight”, indicating he had observed a significant attitude disturbance that could only have happened with a successful separation.  I polled the Flight Controller, and he confirmed “three wires”.  In the meantime, the control system had re-enabled, and quickly brought the attitude back under control.  Separation was a “GO”!  Relief.  No need to terminate the nominal sequence.  I instructed the Flight Controller to waive that option.  Then, at the expected time, the S-S/C began to perform its 180 degree flip. 

    With no time to celebrate, the Flight Team re-focused on the next task – Centaur Observation.

    Figure 2: Depiction of of Shepherd and Centaur, Post-Flip: The Centaur appears unrealistically close.  At 4 minutes after separation, the Centaur would have been about 170 meters away. Also, unfortunately, with the change in separation attitude from the original plan, the moon did not appear in the background of the Centaur images.

    Centaur Observation

    To ready for Centaur Observation, the Flight Team’s next job was to reconfigure the communications subsystem and payload to enable the spacecraft to collect imagery of the Centaur as it receded from view.  To enable the downlink of high-rate imagery, we first had to switch from our nominal downlink data rate of 64 kilobits per second (kbps) to 256 kbps.  Once the data rate had been increased, we’d then increase the allowable data output rate of the science payload, and switch from the low-rate to the high-rate imagery sequence designed for this event. 

    Data rate changes and antenna changes were two operations that we typically didn’t perform via onboard commands.  Each requires the DSN antenna providing the link to re-acquire the spacecraft signal.  The time needed to do that varied significantly, but fixed-time commanding doesn’t allow for that flexibility.  For this event, we had three Madrid DSN antennas watching simultaneously (two 34-meter dishes and a 70-meter dish), and each needed to re-acquire the LCROSS signal after the downlink rate change.  To complicate things, we knew we couldn’t achieve an adequate link margin at 256 kbps until the spacecraft had completed its 180 degree flip (to point the center of the secondary omni-directional antenna pattern towards Earth), the duration of which wasn’t known with high certainty given our lack of in-flight testing with the new gains.  By design, we planned to utilize ground-based commanding. 

    In actual flight, as expected, the science payload powered-on just as the S-S/C began to flip.  Payload Engineering reported good status.  Soon, the cameras were transmitting low-rate imagery, but we hadn’t rotated far enough to see the Centaur yet.  But without the Centaur on its aft end, LCROSS behaved less like a school bus and more like a sports car.  It was making short work of the 180 degree flip.  At three minutes, with the flip complete, we commanded the downlink rate change.  At about the same time, Science reported the receiving the first image of the Centaur, under the low sampling rate.  The DSN stations re-acquired the signal, and as soon as possible, we commanded the payload to begin the high-rate image sequence.  We has successfully configured the S-S/C for the Centaur Observation, and we now had a little time to breathe, and to watch the Centaur images flowing into the control room.    

    This link points to a movie made by the Science Team using the images collected during the Centaur Observation (with time sped up significantly). 

    https://www.nasa.gov/mov/392770main_mir1-zoomed-uncompressed-21fps.mov

    In the movie, you’ll notice it looks like the Centaur is moving all over the place, but really it’s the S-S/C rotating back and forth.  The attitude control system keeps the spacecraft pointed to within a small “deadband”, a small imaginary box around a target pointing direction.  The ACS only fires thrusters if the S-S/C is getting near the edge of the deadband.  What you’re seeing is the ACS “bouncing” on the edge of the deadband.  If you look carefully, you’ll notice that the ACS seems to change to a smaller deadband later in the movie.  This not an illusion.  The spacecraft switched from a wider deadband (1.0 deg) to a narrower deadband (0.5 deg) partway through the observation sequence, as part of a test of the new control gains.  You’ll also observe that the Centaur is in a very slow tumble.

    Braking Burn

    At the time of the Centaur impact, to meet the observation requirements, the S-S/C needed to be roughly 600 km above the lunar surface.  This was a trade-off between being too close (and possibly being hit by debris from the Impact), and being too far away (the spatial resolution of our imagery would decrease with distance from the surface).  The velocity induced by Separation between the S-S/C and Centaur (0.7 m/s) would be too little to achieve that distance in only 9 hours 40 minutes, so the Braking Burn was inserted into the mission design to slow down the S-S/C down by an additional 9.0 m/s (about 13 times the velocity change of Separation).   The Braking Burn was also designed to independently target the S-S/C impact point, also selected by the Science Team.

    In flight, nineteen minutes after Separation, the onboard sequence terminated the Centaur Observation and began configuring the spacecraft for the burn.  The Flight Team quickly commanded the return to 64 kbps downlink rate (necessary before changing our attitude again), and the DSN re-acquired the signal.  The onboard sequence performed another attitude change, this time to the optimal burn attitude.  Ten minutes later, after passing through two more new attitude control modes, the burn started on time.  Four minutes of firing the 22 N thrusters was predicted to be sufficient to slow ourselves down to meet the impact distance requirement.  With the burn over, we had crossed Shift A’s last major hurdle. 

    Shift A Cleanup and Farewell to LCROSS

    We had a few final things to do before our handoff to Shift B.  We biased our attitude to keep our thrusters warm, re-enabled payload heaters, and loaded a preliminary Impact command sequence to the spacecraft.  Planned before Separation using predictions of the post-Separation and Braking Burn orbit, those commands were designed to perform the full Impact sequence without the Flight Team.  They were an insurance policy to protect the team in the off-chance that we lost communications with LCROSS until the last few minutes.  However, without accurate knowledge of the S-S/C post-burn orbit, they wouldn’t point the cameras as accurately as the re-planned versions.  That planning was up to the next shift, using the actual, measured orbit after Braking Burn. 

    With that, Shift A was done.  In a rush all day, we sat on console for a few more minutes, soaked up our last experiences as members of the LCROSS Flight Team.  It was a bittersweet moment – so many experiences in those chairs in such a short time, and the culmination of three years of preparation.  Our final shift had gone perfectly – the result of months of thought and practice, and a great spacecraft design.  The moment we’d all been waiting for was only 8 hours away, and then the LCROSS flight would be over.  Officially, our time was over now.  Shift B folks began milling into the control room, with a combination of big smiles and game faces on, ready to get started on their last shift.  With just a little reluctance, we stood up from our chairs, and got on with shift handover.

    Figure 3: MOCR Shift A: The Ames-local part of Shift A that ran Separation through Braking Burn.  Pictured from R to L: Matt D’Ortenzio (Flight Controller, NASA Ames), Matt Reed (Attitude Control Engineer, Northrop Grumman), Tony Lindsey (Data Management Engineer, NASA Ames), Tony Colaprete (LCROSS Principal Investigator, NASA Ames), Darin Foreman (Systems Engineer, NASA Ames), and myself, Paul Tompkins (Flight Director, Stinger Ghaffarian Technologies at NASA Ames).  A lot of other people also worked this shift and contributed to its success!

    Remember, the next post will feature Mark Shirley, our Payload Engineer.  Stay tuned, and thanks for reading!