All of you keeping up with LCROSS have undoubtedly already heard the good news – that our team at NASA Ames successfully performed our capstone event on Day 5 of the mission – the Lunar Swingby. In execution, it worked just as we had hoped. But getting it to work wasn’t easy. It took us many months of development, testing and updates to get the procedures right, and throughout our months of training, our team rehearsed this event far more than any other.
Lunar Swingby (LSB for short) is one of two close lunar encounters during the mission – in many ways it was a dress rehearsal for impact. It’s an intricate coordination between orbit dynamics, attitude control, communications, thermal, power and of course, the science payload. In this post, I’d like to walk you through the evolution of the procedures and command sequences that we ultimately flew on Day 5.
An Introduction to Lunar Swingby
So, what was LSB designed to do? There were two primary goals – one, to throw LCROSS into its phasing orbit, called the LGALRO, or Lunar Gravity Assist Lunar Return Orbit. As that name suggests, our mission uses the gravitational well of the moon to drastically change the Transfer Phase orbit without any propellant cost. The post-LSB orbit is designed to hit the moon with one additional Trajectory Correction Maneuver (ideally), enabling the steep impact angle at the lunar south pole that otherwise would have been impossible to achieve using the outbound Transfer Phase (trans-lunar) orbit we were provided by the Atlas/Centaur.
The second goal of LSB was to calibrate the science instruments, taking advantage of the relatively close range of LCROSS to the moon. The calibration was divided into two phases: one to measure lunar surface targets, and the other to sweep the payload over the “limb” of the moon (the “limb” is the edge of the apparent disk of the moon). Looking at lunar surface targets with known mineralogy and optical properties (albedo) allows our science team to calibrate the cameras and spectrometers. Having “ground-truth” means they can compare the actual instrument outputs against expected outputs, knowing how the instruments should respond to the known surface properties. On the other hand, sweeping the payload over the lunar limb allows the science team to calibrate the payload alignment. The LSB incorporated “slews”, or changes of orientation, that alternately pointed the instrument boresights off the moon, then on the moon, back and forth, such that the boresight would inscribe a path perpendicular to the edge of the lunar disk. Since we know the spacecraft orientation accurately, and we know the precise positions of the moon and the spacecraft, we can predict, for a given alternating sequence of slews, the precise timing of when the boresight should cross onto and off of the lunar disk. A consistent bias between the predicted and measured timing indicates an alignment offset.
LSB was designed to measure three surface targets for 5 minutes each (three craters: Mendeleev, Goddard C, and Giordano Bruno), and two limb targets, one in a “left-right” slewing orientation, and a second using an “up-down” orientation to detect misalignments in both possible directions. Each of these limb sequences was also 5 minutes long. Sounds pretty straightforward, right? Well, the details make things tricky.
For reference in the next few sections, check out the link to an animation of the “as-executed” Lunar Swingby. The animation is sped up so that you can see the various sub-events (slews from Cruise attitude to initial sampling attitude, surface samples, limb slews, etc). Vectors coming out of the spacecraft model indicate the direction to the Sun (yellow) and Earth (Blue), and the body-fixed vectors representing the LCROSS body coordinate frame (Red). The projected square represents the field of view of the LCROSS visible camera, and the narrow cone projection represents the 1.0 degree field of view of the nadir-pointing Near-Infrared Spectrometer, the most important instrument for LSB science return. The little bullseyes are our surface targets. See if you can identify the “setup” and “cleanup” attitude changes from those between targets.
To get into LSB constraints, let’s continue with the discussion of pointing the spacecraft to the targets. The requirement was to point to within 1 degree of the intended target, and to slew no faster than 0.15 deg/s during measurements to avoid “smearing” our images (since we don’t have a super-fast effective “shutter speed”). For LSB, we designated an attitude control mode that would maintain our orientation to within 0.5 degrees of a given target orientation. The trick is that our spacecraft cannot autonomously identify the targets in an image, nor can it track them automatically once we point it in the right initial direction. So, we had to tell the spacecraft where to point each few seconds, relying on knowing the precise position and orientation of the moon, as well as the position of the spacecraft (via orbit predictions) to predict the right set of orientations that would point the science instruments to the targets. This means the whole sequence of orientation targets had to be planned in advance, but no earlier than several hours prior to the LSB event, since with TCM 3 happening the day before LSB, we couldn’t possibly know our orbit accurately enough to meet the pointing accuracy requirements.
To make matters slightly more complicated, instrument pointing was not our only concern with spacecraft attitude. LSB and Impact both call on our highest available communications downlink rate – 1 megabit per second (Mbps). This rate is only possible via one of two Medium Gain Antennas (MGA’s), and typically would require the use of a DSN 70 meter antenna. For lower rates, our Omnidirectional antennas provide adequate link margin in almost any orientation. The MGA’s, however, need to be pointed to within 20 degrees of the Earth to enable a strong signal to the ground, with more precise pointing providing better results. With two attitude constraints, one for the instruments and the other for communications, our attitude was fully constrained. Any other orientation requirement would have to be balanced against sacrificing one of the first two. So, for example, we’d have to deviate from our normal “happy” orientation pointing the solar array at the sun during all of these observations.
This brings me to thermal and power constraints. Both the Power and Thermal subsystems were designed to take only short deviations from the nominal spacecraft sun-pointed orientation. Prior to flight, our Power subsystem was predicted to allow two hours off-sun. Our Thermal subsystem’s constraints were a little more complicated, but it was predicted to enable up to one hour in some orientations, and two hours in other orientations. We’d have to follow the one-hour limit since we couldn’t rely on satisfying the two-hour orientation constraint.
What about the science instruments? The Data Handling Unit (DHU) is a separate computer that controls the science payload. The DHU includes 10 buffers in non-volatile memory to store instrument command sequences. The Science team defined different sequences for different purposes. Originally, the Science team defined three sequences for the nominal LSB – one for surface target sampling, and another two for the limb measurements. Later, as I’ll describe below, they had to define an additional three to cover contingency cases. Another job of the operations team was to perform power-up and power-down of the DHU and operational heaters. Thermal constraints dictate that payload operational heaters must be enabled at least 4 hours prior to using the science instruments to warm them to their operational temperature range. Furthermore, the DHU, running with the full set of instruments, cannot run for more than 60 minutes without risking overheating. Finally, the spacecraft command sequences, had to coordinate the execution of the LSB science command sequences with the commands to control the spacecraft attitude so that everything happened at the right times.
The following is a depiction of our surface and limb targets for LSB.
Wringing Out the Nominal Procedure
Last September, we thought we had a pretty good handle on flying Lunar Swingby (recall that our original launch date was October 28, 2008, so we had to be ready). Due to the precise timing required to get this right, all of the LSB commands were originally packaged into an onboard command sequence, including several hundred attitude commands and other commands to power the DHU on and off, to start DHU command sequences, and to configure our communications system. We had a nominal procedure that worked well in simulation, and both shifts had practiced it enough to gain confidence that we could do this in flight. We had already tackled many of the issues, but there were problems.
For one, we were exceeding our one-hour thermal time constraint. The science calibrations were predicted to require a total of 45 minutes (five targets, five minutes each, plus four five-minute slews between targets), leaving only 15 minutes for other preparation and cleanup steps. The initial reorientation from our Cruise attitude (which started the thermal clock ticking), the communications configuration steps, and the return to Cruise attitude (ending the thermal clock) required another 25-35 minutes, so we exceeded the limit by 10-20 minutes. Ultimately, this forced our lead Thermal Engineer to revisit the thermal subsystem analysis, and to prove that the Thermal system could actually handle up to two hours off-sun in almost any orientation.
Another LSB problem was with attitude control fault management. As I mentioned earlier, to achieve the accuracy needed for science calibrations, LSB needed to employ an attitude control mode that enabled ]0.5 degree pointing precision. Early tests of our command sequences worked just fine. However, this mode was not originally designed with LSB slews in mind. One day in the middle of an LSB simulation, our simulated spacecraft suddenly reset and dropped into Survival State, terminating LSB. As we later discovered, we had exceeded the maximum attitude rate limit for the ACS mode we were using (the limits were tailored for near-stationary operation), causing a significant spacecraft fault management response. Later versions of our procedures added steps to temporarily adjust the rate limits for that control mode to allow the spacecraft to slew a little faster without tripping fault management.
While testing LSB, the Flight Team also was temporarily stymied by a bug on the spacecraft that seemed to be activated when flowing high-rate data from the science payload (as happens during LSB). In some runs, we found that the bug prevented telemetry and science data flow, and even prevented most commanding – a really bad problem to have in the middle of such an important event. Over many trials, we found the only effective Flight Team response was to reboot our spacecraft processor – which would also put the spacecraft into Survival State and totally remove hope of successfully collecting our calibration data. This behavior had been observed in earlier, lower-level tests of the payload and spacecraft, but had never reared up to affect operations. Our operational discovery placed LSB in serious jeopardy. After a lot of very careful debugging, our Northrop Grumman partners ultimately discovered the cause of the problem – an FPGA (Field Progammable Gate Array) that encoded the communications protocol had a programming bug in it, sourcing from a faulty specification of the protocol. Once the problem was isolated, NG wrote software workarounds that enabled us to completely avoid the problem.
Once our original launch date had slipped briefly to December 2008, and ultimately into 2009, the Flight Team began focusing on “risk buy-down”. With every aspect of our mission design, we asked ourselves, “with this new-found time, how could we make our mission less risky?” With all nominal procedures worked out, we began to work on possible contingencies – things that might go wrong and cause LCROSS to have a bad day, either through loss of important data, or worse, a loss of spacecraft.
What were the big ones for Lunar Swingby? LSB payload calibration data is really important to the Science Team. I mentioned the high downlink rates required that we use our Medium Gain Antenna, and a DSN 70 meter ground antenna. What if one of those two elements failed? Could we still get enough data back to make our Science Team happy?
In November and December 2008, we worked on two low-rate contingency options, one assuming an MGA failure that utilized the Omnidirectional to 70 m antenna, the other assuming a DSN 70 meter antenna failure, utilizing the MGA and a 34 m antenna. We spent hours working through how to address these failures, and it became very clear that the structure of the nominal procedures and command sequences was not well-suited to the contingencies. Our challenge: at the time the failure is detected, the contingency response was to reconfigure the spacecraft for a lower downlink rate (256 kbps), then to run a specially-tailored low-rate DHU instrument command sequence that would approximate the higher-rate data return, with minimal sacrifice. The trick was that things had to be done pretty quickly (within 10-15 minutes in some cases) and, depending on when the failure happened, it required different responses. For example, if the DHU and instruments already running, the response was very different than if they were still several minutes from powering up. Doing the wrong thing could get the team tied in knots and make a bad problem worse. The nominal commanding structure made it very difficult to do this in a timely way.
We went back to the drawing board to re-structure our commands and procedures to suit both nominal and contingent execution. Ultimately, the changes we came up with were significant, but simple, and the result was far cleaner. However, in practice, things weren’t yet up to snuff. January Operational Readiness Tests (ORT’s) proved our initial contingency solutions were technically correct, but too slow – they required lots of human detective work and decision making under time pressure. The right decision made too late was not much of a help. In February, after multiple full-team tests, and hours more re-work, we finally arrived at a solution that addressed both the MGA and DSN 70 meter failures under a common approach that streamlined the decision process dramatically. As a side bonus, the same approach provided a contingency response option for similar failures for the Impact procedure. We finally had this one nailed down.
Aside from the low-rate contingency responses, we developed a few other major contingencies, including a general abort response that would enable our team to return to nominal Cruise state in the event of an unspecified problem aboard the spacecraft, and another response that would, in the event of a fire in our building or an evacuation, would enable the Flight Team to leave the control room and still collect LSB data successfully.
Details, Details: Deep Space Network Scheduling
As we waited for launch, now moved to April, May and then to June, our team worked with JPL to finalize our Deep Space Network schedule. Before launch, every mission that uses the DSN has to negotiate antenna time to support early-mission communications between the mission operations team and the spacecraft. Typically, negotiations happen between the new mission team and teams representing other spacecraft that are already flying. Newly-launched spacecraft tend to get high priority, since every team needs to have a lot of dedicated time at the beginning to get into orbit, for commissioning and to work out problems. Our launch was a little different because there were two new spacecraft (LRO and LCROSS), both going to the moon, and both relying on DSN for critical periods. As we worked through the details, we discovered that LRO and LCROSS were in conflict for two big LCROSS events: TCM 1 and, you guessed it, Lunar Swingby.
Lunar Swingby coincided with LRO’s Lunar Orbit Insertion burn (LOI), which would place LRO into orbit about the moon. Even though LSB was quite important to LCROSS, LOI was mission-critical for LRO. Missing LOI would result in a total mission failure for LRO, so LCROSS had to take back seat for DSN scheduling in this case. To ensure LOI success, LRO requested dual-antenna coverage for the burn, in case one antenna went down at a critical time (a standard request for mission-critical events). Each site has only two 34 meter antennas, so dual-coverage is pretty restrictive for other missions. Depending on our specific time of launch, different DSN sites – either Canberra or Madrid – would cover LOI and LSB. In addition to their 34 m dishes, both complexes typically operate a 70 meter antenna. However, to make matters more complicated, Madrid’s 70 meter antenna was scheduled for servicing over May and early June, making it unlikely that LCROSS could use it for LSB. Canberra also has a single LCROSS-compatible 26 meter antenna that could be brought to bear if needed.
Here’s how the negotiations ended up. Under most scenarios, during LRO’s LOI burn, the LCROSS Flight Team would have no direct contact with the spacecraft – our first loss-of-contact for the mission. If the LRO’s LOI went off successfully, LRO would release its backup antenna for use by LCROSS. With problems, LRO was entitled to hold onto its backup antenna far longer. The story was far more complicated because each launch time (three per day for our mid-June window) resulted in significantly different geometry and timing. But under many circumstances, an early release of the DSN backup would provide only 20 minutes before the required start of onboard command sequences – not enough time to load the sequences first. A late LRO release would mean the command sequences would have to start “in the blind”, before LCROSS actually was able to start them. This took weeks of planning meetings to work out, right up until the week before launch. Our JPL scheduling team deserves a lot of praise for supporting LCROSS and LRO in this intricate and time-constrained effort.
Our final negotiated schedules caused us to re-think how our team prepared for LSB. Originally, all of our command sequences were to be loaded in the two hours immediately prior to the LSB. Now, it was clear we’d need to load our command sequences, and potentially start them, prior to the LRO LOI, and hours before LSB. That way, even in the worst case, even if LRO’s burn took longer and overstepped the beginning of the LSB sequence, once we got an antenna, we’d have a shot at getting LSB data. This meant we’d have to have the full LSB command sequence much earlier than originally thought, and that significantly affected our Day 5 timeline and staffing plan. We made the procedural and timeline changes, and familiarized ourselves with all of the possible scenarios. Perhaps most complicated – what once was a single, complicated sequence now split into several possible sequences, depending on our launch time and on the success of LRO’s insertion burn. Some of these plans came into place in the week before launch, so we only had time to review them on paper!
Doing it for Real: Lunar Swingby in Flight
On Day 5 of the LCROSS mission, our team was getting the hang of flying this new spacecraft. All of the events had gone really well, and despite the various problems, the team was still going strong. Shift B was slated to perform Lunar Swingby, the capstone event for the week, and certainly the most difficult event to that point. We were all very excited, and a little nervous.
My Shift A team worked hard to get everything perfectly planned to give Shift B the best shot at success. Our Command Approval Meeting went off without a hitch, and soon we were handing the spacecraft off to our friends on Shift B. Timing for LSB command upload was extremely tight – as it turns out, for our launch time, we had just over one hour to get our LSB command sequence loaded to the spacecraft before a 5-hour DSN commanding outage over Madrid (consumed by LRO performing the LOI). Thankfully, the DSN team at Madrid worked hard to complete the refurbishing of the Madrid 70 meter antenna, and it was ready for unofficial service in time for LCROSS. It couldn’t provide commanding, but it enabled us to receive telemetry during the LOI.
To add to the complication and urgency, the Flight Team was still actively commanding the spacecraft to keep its thrusters warm. Recall that in previous days, some of the LCROSS thruster heaters were not activating, forcing our team to resort to commanding non-standard attitudes and unplanned thruster firings to keep them from freezing. Well, with the gap in commanding schedule to enable the LRO LOI, we’d have to endure several hours of cooling without commanding. We warmed the thrusters before going out of contact, and then relied fully on our biased attitude and sunshine to keep things warm. All we could do is watch.
I stayed past my shift to watch Shift B successfully load the LSB commands to the spacecraft, then left to catch some sleep. Even though I was scheduled to return hours after LSB, I knew that I’d be missing a once-in-a-lifetime experience if I slept through the night. I couldn’t resist, and planned to return early.
I returned with perhaps 30 minutes to spare prior to LSB. The spacecraft thrusters stayed warm, and LRO’s LOI apparently went off perfectly, because they released their backup DSN antenna even earlier than we thought possible! We had the best scenario possible. Shift B confidently prepared for and executed LSB. A number of us on the off-duty shift watched things unfold from a conference room outfitted with large-screen monitors and full voice loop audio below the control room. The onboard command sequence started right on time, and transitioned our attitude to the first target direction. Shift B commanded the communications reconfiguration and successfully achieved our 1 Mbps downlink rate via an MGA, the first major hurdle. Then, we waited on the edge of our seats as the command sequence neared the DHU power-up commands, and LCROSS’s first photos of the moon. It all happened just like it was supposed to – the DHU powered up, our Payload Engineer reported he saw good DHU telemetry, and then, our first image of the moon appeared on our monitors – not the cardboard image that we used for our spare simulator payload, but the real moon! Within minutes, we saw images from the Visible Camera, the Near-IR Camera, the Mid-IR Cameras, and spectra from our various other instruments. We sat rapt, quietly taking in this live data feed. Watching these images live from the moon was the culmination of all we had worked for – these photos were living proof that the year of work had paid off. We were absolutely thrilled!
Forty five minutes later, without fanfare, the DHU powered off again, the communication reconfigured for low-rate Omni downlink, and the spacecraft rotated back to Cruise attitude. Though the spacecraft returned to Cruise state after LSB, our team’s demeanor had changed subtlely. The pressure of Transfer Phase had lifted. Our success had set in and, exhausted from a week of very long hours on console, we celebrated with smiles and a shared sense of satisfaction. There was a lot yet to be done, but we’d made it through the most intense period of the mission…save Impact!
Incidentally, the Science and Payload team has volunteered to produce a guest posting on the details of the science findings, so I promise some details on Science findings in a future posting. Stay tuned!