Month: September 2009

With our first orbit around the Earth behind us, the Flight Team team looked optimistically ahead to a number of upcoming events, including two science instrument calibrations, and two Trajectory Correction Maneuvers, TCM 5b and 5c.  Thankfully for us, the schedule of planned events for the next 34 days to be a little more relaxed than for the previous 42 days.  The key distinction being the planned events.

August 1 (DOY 213): A Look at Planet Earth, and a “Discovery” of Water

Before the mission began, our team had identified key geometric opportunities for performing our Earth and moon-looking science instrument calibrations.  One of the important factors in determining the best time is spacecraft-to-target distance.  The closer the Earth and moon are to LCROSS, the better resolution we can attain, and the stronger the signals will be (hence why Lunar Swingby was so important).  The below plot shows the distance between LCROSS and the Earth (light blue) and LCROSS and the moon (dark blue), over the entire mission.  The labels indicate when specific LCROSS events happened (or will happen), including launch, Lunar Swingby, Earth Look Cal 1, Earth Look Cal 2, Moon Look Cal, and Impact.  As you can see, the Earth Look events coincide with low points on the curve, while Moon Look timing was influenced by other factors.

Another factor is “phase” with respect to sun illumination on the target.  Both the Earth and the moon go through cycles depending on the relative positions of the sun, the Earth or moon, and the observer, in this case LCROSS.  So at various times, the Earth will look like a crescent, at other times will be “full” (like a full moon) and still other times will be “new”, with its face completely un-illuminated.  For science measurements, our Science Team tended to prefer “fuller” views of the Earth and moon. 

As planned prior to launch, Earth Look started by rotating the payload boresight (pointed along the LCROSS –X axis) to point at the center of the Earth, then repeatedly “swept” the boresight in alternating East-West and North-South passes across the Earth’s disk.  The objective was to collect spectra of the known Earth signal (for spectral calibration) and also re-affirm the alignment of the instruments.  Similar to the limb crossings in Lunar Swingby (see “Lunar Swingby: Development of a Procedure”) by sweeping the cameras across the disk, the Science Team could compare the timing of the rising and falling signal with the well-known motion of the spacecraft relative to the position of Earth.  Consistent mismatches in timing would indicate a misalignment. 

We collected some cool photos of the event, included here.  Note that our instruments are optimized for lunar impact and finding water, so you can’t make out the landforms in the visible images, and the IR images detect thermal differences in the upper atmosphere.  It was really exciting to be looking back at our home planet from so far away.  In contrast to images taken from the Space Shuttle, International Space Station, or even Apollo, our images really show how insignificant the Earth is in comparison to the immensity of space.

LCROSS, via its spectrometers, also “discovered” water on the Earth – an event that didn’t exactly make headlines, but made the Science Team quite happy.  Detailed analysis of the spectra and imagery indicated that all of our instruments were all performing very well.

In an unrelated piece of good news, the Mission & Maneuver Design team determined that TCM 5a had been so accurate that TCM 5b would be unnecessary.  Originally, TCM 5b would have incorporated another Omni Pitch maneuver, but now with TCM 5b unnecessary, we’d perform a standalone Omni Pitch instead.

August 6 – 7 (DOY 218, 219): Re-Pointing our Antenna – Omni Pitch 3

Nearing the “top” of our orbit (see the flight path diagram), it was time again to perform an Omni Pitch maneuver to re-point the primary omni antenna towards the Earth.  These times are tricky because at the time of the maneuver, neither the initial nor the final orientation provide very good angles to the omni antenna.  You can plan to perform the maneuver early (favoring the initial attitude, but making the final attitude margins really bad), or perform the maneuver late (favoring the final attitude).  Omni Pitch 3 favored the final attitude.  We lost telemetry data sporadically before it began, indicating very poor link margins (downlink rate at our lowest rate of 2 kbits per second or kbps), but then had sufficient margin to boost our downlink data rate to 32 kbps for post-maneuver virtual recorder playbacks.  Throughout the mission, the Flight Team constantly has to evaluate LCROSS link margins to make sure we can command the spacecraft and monitor maneuvers via telemetry.

August 8 – 13 (DOY 220 – 225): Good and Boring

The mission operations philosophy states that “a boring spacecraft is a good spacecraft.”  This is certainly true because “exciting” usually means risky, dangerous and proximity to failure.  As a Flight Team member, you really don’t want to be responsible for a mission disaster!  Despite the boredom that sets in on those uneventful days, slow and steady is a good thing.  Thankfully for us, LCROSS was behaving very well, and occasionally we found ourselves in long stretches of time between planned events.  During those times, the Flight Team simply monitored spacecraft health, and hoped for boredom.

Cruise Phase health monitoring passes typically include the following steps:

August 14 (DOY 227): Opportunistic Science and Design on the Fly

During the mission, the Science team and Engineering team occasionally requested activities that were not on the original schedule.  The Science team wished to find an opportunity to image a point source with a strong infra-red (IR) signal in order to work on a solution to an Mid-IR 2 camera focusing issue. The considered Venus and Jupiter, and though both were very bright in the visible spectrum, neither provided a very good IR signature.  Working with the Mission & Maneuver Design team, the Science team discovered a golden observation opportunity – on August 16, just as LCROSS would be passing through orbital plane of the moon, the Earth and moon would appear only a few degrees apart from each other (see the plot of the angle between the Earth and the moon, as viewed by LCROSS).  The moon has known IR reflective properties, and because the moon would be on the opposite side of the Earth from LCROSS, it would approximate a point source (look very small).  About the only problem with the proposed timing is that LCROSS would be very far from Earth (see the “Special Earth/Moon Cal” in the above plot of Earth and moon distance.  However, our team considered the opportunity, and decided to go for it.

With a simple mission like LCROSS, a Flight Team is wise to pre-plan as much as possible before launch.  As the saying goes, “the devil is in the details”, and that certainly holds true when piecing together command sequences and operational procedures.  We worked very hard before launch, without the pressure of a flying spacecraft to care for, to pre-plan, construct and test commands and procedures for each event in the mission plan. 

This new activity, dubbed “Special Earth/Moon Look” didn’t match any of our pre-planned maneuvers, so we had to design it on the fly.  We designed the overall approach in a meeting between the Science team, Mission & Maneuver Design Lead, Sequencing Lead, Systems Engineers, Flight Directors and Flight Controllers.  To the credit of the team, especially the Activity Planning & Sequencing Lead (John Bresina), we were able to create command a sequence, adapted from our Star Field Calibration sequence, that could accomplish the goals of the activity.  One major difference was that we’d be looking at the Earth and moon (two targets) instead of a star field (one target), so it was like merging two Star Field Calibrations together.  There were a lot of other changes at the detailed level to make this work.  The Engineering Analyst had to extend his automated checking tools to work on this new command sequence.  The Simulation Engineer also simulated the commands on the LCROSS simulator to make sure things were going to work.  Within a few days we had a set of command products ready to go.

A day before the event, to save time on the scheduled observation DSN pass, we loaded the new command sequences to LCROSS and hoped we’d gotten everything right.

August 16 (DOY 229): Family Portrait: Observing the Earth and Moon

Our Special Earth/Moon Look Cal went off pretty much just as it should have.  We collected images of both the moon and the Earth on all of the cameras, and met the full objectives set out by the Science team.  There were some minor issues with payload heaters, and a small command sequence timing error, but we were satisfied with the results.

The photo here shows a family portrait taken by the LCROSS visible camera.  At the time of the calibration, the LCROSS was approximately 520,000 km from Earth and 881,000 km from the moon.

August 19 (DOY 231): Re-Scaling our Faulty Coarse Sun Sensor

Recall from my previous post (“Our First Orbit Around the Earth”), one of our Coarse Sun Sensors (CSS 1) had been producing low signal outputs since Day 1 of the mission.  Now “South” of the moon’s orbit plane, on the second half of our second orbit around the Earth, we took the opportunity of an open DSN contact period to finally load parameters to the spacecraft that would effectively re-scale the output of CSS 1 to have it match the output levels of the other CSS’s.  This closed out another of our original spacecraft anomalies, and left LCROSS even safer.

August 20 – 21 (DOY 232 – 233): Planning for Cold Side Bakeout 3, Omni Pitch 4 and TCM 5c

We had a challenging few days ahead of us.  The Navigation team had identified another good opportunity to perform a third Cold Side Bakeout (unfortunately on a Saturday).  Then on Monday and Tuesday, we’d have a back-to-back execution of TCM 5c and another Omni Pitch maneuver.  We generated, tested and reviewed commanding products for all three events on Thursday and Friday.  Excited to observe our next Cold Side Bakeout, I elected to take the weekend DSN pass so that I could oversee the maneuver.

August 22 (DOY 234): Innocence Lost

I came onto shift on Saturday morning, very excited for a difficult, busy shift in overseeing Cold Side Bakeout.  This third repetition was planned to look a lot like the first one, which ended up being pretty exciting and somewhat complicated.  After the standard coordination with our Canberra antenna, we prepared for Acquisition of Signal or AOS.

We acquired telemetry, and ran through our standard set of health checks, but it was immediately clear that LCROSS was in trouble.  Inertial Reference Unit (IRU) faulted, Kalman Filter faulted.  Attitude control system using the Star Tracker for body rate information.  Propellant tank low pressure fault.  Thrusters firing nearly continuously.    

It’s hard for me to believe, but LCROSS is now only 18 days from impact.  After a long time away from the LCROSS Flight Director’s Blog, I wanted to catch you all up in the amazing journey LCROSS and the Flight Team have been through in the time since I last wrote. 

On my last post (from July 22), LCROSS had not yet completed its first orbit about the Earth.  We are now on our third and final orbit.  Since then, we have observed the Earth and moon from various distant vantage points in the orbit to calibrate our science instruments.  We’ve baked a lot of ice off of the surface of the Centaur upper stage.  We orbited the Earth freely, with rarely a problem at all, and those only minor ones.  Then in late August, nearly tragically, LCROSS encountered a major problem, which resulted in the loss of most of its propellant reserves, and could very easily have resulted in the end of our mission.  You’ve read the headlines, but I’m happy to report that we’ve climbed our way out of that very precarious position, and are now poised to meet our full set of mission objectives, with less than three weeks remaining until impact.  The Science Team recently selected the impact target – Cabeus A1 – and the LCROSS team is now making final refinements in the orbit to make sure we hit that spot.  This is an exciting time.

In this post, I’d like to provide a summary of the events following Lunar Swingby on our first orbit around the Earth.  The next post will cover our second orbit and the lead-up to our “anomaly”.  Then in following posts I’ll detail the anomaly, and provide some running commentary in the lead-up to impact. 

By Deep Space Network convention, our mission uses a “day-of-year” reference for dates, rather than standard calendar months/days.  So, January 1 is DOY 1, and December 31 is either DOY 365 or 366 (depending on whether it’s a leap year or not). 

I’ll provide both for you here, so that you can interpret the orbit diagram I’ve included.  Each of the below events are shown along the flight path LCROSS took in its first trip around the Earth.  Each major “tick” mark is a 24-hour day.  Transfer Phase events are shown for reference.

June 23-25 (DOY 174 – 176): Taming our Thruster Thermal Control Problem

The Flight Team was working at full capacity during the first five days of the mission (Transfer Phase), with 24-hour operations, and major events happening daily.  It was an impossible pace to maintain indefinitely with such a small team.  But before we could ease off to our 4-out-of-72-hours contact schedule planned for Cruise Phase, we had to protect against some of the problems we encountered early in the mission, and build confidence that LCROSS would remain safe long-term. This took some extra work.

Just after Lunar Swingby, five days into the mission, we were still fighting our newly-discovered thruster thermal problem (see “Real-Life Operations: Day 3” post).  In a temporary move to keep them above the freezing point of our hydrazine propellant, our team watched thruster valve temperatures for T1 and T7 (both 5 Newton attitude control thrusters – see diagram), and fired specific sets of thrusters via ground commanding if we saw any of them get too cold.  This was not an effective long-term solution – we commanded the sequence from the ground many times per shift (very crew intensive), and this was also a waste of propellant.  But by the end of Day 6, we had loaded our first automated solution (no human intervention required) that performed the same sequence of thruster firings if either of the cold thruster valves fell below 7 degrees Celsius.  Though we refined the solution over the coming weeks, establishing this “safety net” was a major milestone in freeing up the Flight Team from having to staff 24 hours a day to keep LCROSS safe.

In a parallel effort, our team worked to find an orientation for LCROSS that would keep our cold thrusters warm using the sun, without firing thrusters at all – intended as a primary means of thermal control.  By June 25, we settled on a “yaw” bias of -20 degrees, tipping the “top” of LCROSS towards the sun slightly to warm the cold thrusters (T1 and T7 – see the figure).  We’re still using the same sun-relative orientation for LCROSS today, and plan to use it for the rest of the mission.

Thankfully, in the midst of this work, the Navigation and Mission & Maneuver Design teams informed us that we could skip our first post-Swingby Trajectory Correction Maneuver, TCM 4a.  It was always optional, but it was great to hear that Lunar Swingby was so accurate that performing this burn was unnecessary.

June 26 – July 14 (DOY 177 – 195): Gaining Confidence

Since our mission wasn’t certain how quickly we could transition to our Cruise Phase schedule, we grabbed as much leftover time on the Deep Space Network (DSN) as we could in the early days of the mission to allow a “gentle” transition to hands-off operations.  This wasn’t easy on the Flight Team – odd hours, numerous contacts, and lots of planning to do on the side.  We were anxious to scale back, but only when we could prove LCROSS would remain safe during long periods out of contact. 

In our new -20 degree yaw-biased sun attitude, LCROSS never triggered the automatic thruster firings to keep our two cold thrusters warm.  This was great news.  With automatic thruster thermal control showing promise, we moved to retire some of our other concerns with LCROSS.  Recall that the Centaur upper stage was leaking some of its residual propellant, and causing LCROSS to “fight” the resulting torque with thruster firings each day (again see “Real-Life Operations: Day 3”).  Well, our team observed a gradual but steady decline in the daily thruster firings to counteract the Centaur leak.  We were using only 0.24 kg/day at this point, easily sustainable for the remainder of the mission with our substantial propellant margin (warning: foreshadowing).  Good news for propellant usage, but the reduced thruster firing frequency also caused our cold thrusters to get colder!  Space operations are never simple.

We also got to skip Trajectory Correction Maneuver (TCM) 4b, scheduled for June 30 (DOY 181), our second skipped maneuver of the mission. Our orbit remained right on track stemming entirely from the accurate targeting of Lunar Swingby.

As we gained confidence, we eased off of our DSN contact schedule, de-staffing some contact periods, and releasing others.  This finally gave our team some much-needed down-time.  We also started figuring out our operational “rhythm” – what shift schedules worked and didn’t work, how to communicate information over the team without regular shift handovers, etc.  Simulations only go so far, and there’s nothing like flight experience to understand how things really work.  We were learning every day.

July 15 (DOY 196): Thawing the Centaur

As I describe in “Introduction to Cruise Phase”, we had anticipated that water ice might accumulate on the exterior of the Centaur prior to launch and remain there throughout the mission.  It was essential that we remove as much of that ice as possible prior to Impact, to avoid having it interfere with both our trajectory and water measurements at Impact.  Mid-July provided a great opportunity for our first attempt.

The plan for Cold Side Bakeout (CSB for short) was pretty simple.  From our sun-pointed (but yaw-biased) orientation, we would command the spacecraft to first remove the yaw bias, then to slowly rotate 180 degrees about the pitch axis (the long axis of the Centaur – see the figure) to point the solar array directly away from the sun, and the “cold side” of the Centaur directly towards the sun.  We would remain in that orientation for one full hour to warm the Centaur surface and to bake off as much water as possible.  At the end, we’d rotate back to our yaw-biased Cruise attitude and continue the mission. 

The risky part of CSB is that we’d be rotating the spacecraft in exactly the wrong direction for the Power and Thermal design, with the solar array dark and cold, and normally hot electronics equipment panels straight into sunlight.  Analysis indicated we could only tolerate this thermally for two hours.  We had to be on the alert to terminate the maneuver if anything heated or cooled more quickly than expected. 

The really interesting part is that our Navigation team selected July 15 for its particular geometry (with LCROSS passing through the ecliptic plane – the plane of Earth’s orbit about the sun) that would make it possible to actually measure the tiny change in LCROSS speed resulting from water escaping from the Centaur surface!  We were very curious to see what the measurements would tell us.

In practice, Cold Side Bakeout was a very interesting event.  Halfway around to the bakeout attitude, our Star Tracker (STA) suddenly dropped into Standby Mode just as it had before TCM 3 when we inadvertently pointed it at the Earth (read “Multi-Tasking: Day 4”).  We’re uncertain what caused the STA to trip to standby mode (it was not pointed at the Earth, moon or sun), but an enticing theory is that escaping water vapor from the Centaur scattered incoming sunlight and confused the sensor (though it may also have been sunlight reflected off the inside of the conical shroud protecting the STA).  Magically, as we pointed the Centaur cold side to the sun, the Navigation team noticed a Doppler shift in our ranging signal, indicating a change in our spacecraft velocity.  The Engineering team also noticed a change in the attitude control behavior, presumably due to a torque induced by escaping water vapor.   On its final assessment the following day, the Navigation had some startling news.  The impulse induced by the escaping water was actually 3.5 cm/second, nearly 3 times more than expected, and the effect did not dissipate over the hour-long bakeout.  This meant one thing – the Centaur was carrying lots of water, and we’d need to perform yet another Cold Side Bakeout in the future.

In a great act of forethought, one of our Flight Controllers, Matt D’Ortenzio, predicted that exposing all of our back-side thrusters to the sun during the CSB might actually prevent the thruster heaters from activating themselves later.  In our original implementation of the automated thruster thermal protection monitors, we only protected the valve temperatures for the two coldest of eight (T1 and T7) attitude control thrusters (all others were warming via heaters).  On the prediction that we might have trouble with others, we loaded similar monitors for the remaining six.  This proved to be a good call.

July 16-17 (DOY 197 – 198): Omni Pitch Flip and Thruster Thermal Flare-Up

On this day we made our second “Omni Pitch Flip” of the entire spacecraft (about the Pitch axis – see figure) to reorient the primary omnidirectional antenna toward the Earth again (see “Introduction to Cruise Phase”). 

Before we executed the Pitch Flip, however, we found that our automated thruster warming sequences had fired 70 times while we were out of contact with LCROSS.  The thruster 2 heater, which had up to this point properly activated to control the T2 valve temperature, had stopped activating.  The thruster now depended on our thruster firing “safety net” to avoid freezing.  Matt’s prediction of CSB had come true.  We were worried that perhaps we had induced a thermal instability – that the thruster would never settle back into its nice heater-based thermal control cycle.  It took another day before the T2 temperature equilibrated, allowing its heater to warm the valve without using additional propellant.  In the final assessment, we used on the order of 2.1 kg of propellant to fight the thermal issue after Cold Side Bakeout.  CSB had its cost.

July 18 – 20 (DOY 199 – 201): Steady as She Goes

LCROSS stayed healthy as we periodically monitored telemetry, downloaded and reviewed telemetry data stored onboard while out of contact, and performed a clock calibration.  Our thrusters continued behaving well, and we were burning less and less propellant in fighting the Centaur leak.  Things were looking great!

July 21 (DOY 202): Trajectory Correction Maneuver (TCM) 5a

As I described in “Welcome to Cruise Phase”, the LCROSS mission plan contained two major “delta-v” maneuvers, or changes in velocity, that had to be performed.  The remaining planned TCMs were “non-deterministic”, to be performed only as needed to clean up the errors stemming from previous maneuvers (e.g. minor mis-pointing, thruster performance modeling inaccuracies) and due to slight mis-prediction of effects on-orbit (e.g. solar radiation effects, effects of imbalance between attitude control thrusters).  We had been able to skip TCM 4a and 4b because of how well things had been planned and predicted by our Navigation and Maneuver Design teams. 

TCM 5a, however, was our second “deterministic” burn, meaning it had to be done.  TCM 5a, which used the larger 22 Newton thrusters, increased our velocity by 21.1 m/s, and was the biggest “delta-v” maneuver of the mission.  It went off just as planned, and put us on a collision course with the moon.

July 22 – 29 (DOY 203 – 210): Health Monitoring and Maintenance

Most of our time in this week was spent monitoring LCROSS health, reviewing both real-time stored telemetry data.  Because of onboard clock drift effects, we also re-calibrated the clock again to have it better match ground time.

Aside from monitoring, we loaded a new command sequence to the Data Handling Unit to control science instrument sampling for our upcoming Earth Look Calibration, our first payload calibration since Lunar Swingby.  We also downloaded the contents of all 10 of the DHU command sequences, to make sure none had been affected by radiation effects.  The Payload team determined everything was fine.

On Day 1 of the mission, we noticed that one of our Coarse Sun Sensors (CSS 1) was outputting abnormally low readings.  LCROSS uses the CSS’s to automatically point the spacecraft solar array toward the sun in Sun Point Mode (SPM), our “safe mode” for more severe fault management responses (and also the mode LCROSS “woke up” in on Day 1).  If you have a significant problem on the spacecraft, the first thing you want to do is switch to a safe power and thermal configuration.  The low output readings on CSS 1 meant degraded sun-pointing performance, potentially a risk if we ever had a serious problem.

Fortunately, LCROSS has two sets of CSS’s, called Primary and Redundant.  Upon entry to SPM, LCROSS defaults to the Primary set.  If the Primary set is malfunctioning, LCROSS will switch to the Redundant set automatically.  Unfortunately, CSS 1 was in the Primary set.  So, way back on Day 2, to avoid using CSS 1 by default, we designated the Redundant set as the default.  However, if the Redundant set ever malfunctioned, we wanted LCROSS to switch to the Primary set.  On July 23, we loaded a small flight software patch to allow LCROSS to switch automatically to the Primary set.  Meanwhile, we worked on an upload to actually re-scale the CSS 1 output to make it fall in line with the other CSS’s.  This is an example of the active maintenance that happened in various ways throughout the mission.

July 30 (DOY 211): Cold Side Bakeout 2 – Still More Ice on the Centaur

We executed a second Cold Side Bakeout maneuver at the end of our first orbit around the Earth, again near our passage through the ecliptic plane.  This time we rotated to the bakeout orientation more quickly (with high propellant expense), and faced the cold side of the Centaur to the sun for longer.  The good news is that Navigation saw a reduction of the delta-v effects due to escaping water – about 1/3 of what we saw for CSB 1.  However, there was still a substantial disturbance to our orbit, so we began looking for a third CSB opportunity in a future orbit.

All of the above happened in our first orbit around the Earth.  At this point, we were well on our way in the mission, having mastered our initial problems with LCROSS, having performed the largest TCM, and having removed a significant quantity of water from the Centaur.  However, we hadn’t operated the science payload instruments since Lunar Swingby.  This would be our first priority on the second revolution about the Earth.  In, my next post, I’ll pick up where this leaves off.