Multi-Tasking: Day 4

First, a note from the Flight Director…my apologies for not keeping up with my writing.  I’ve been extremely busy with flight duties, and am just now getting on top of things again.  This post was written in draft form a couple of days ago, but just now shaped up for posting.  I am enthused by all of your comments and encouragement!  Thank you for all the great notes.  I hope you enjoy this next installment.  More to come soon!

-Paul

The first week of the LCROSS mission plan is ambitious, even without problems to work through.  On Day 3 when we started realizing we had some thermal issues, it was a little daunting.  But now that I think about it, our team had met similar challenges prior to launch.  Over the LCROSS development, our team has had to work around a lot of surprises, under a very tight schedule and budget.  So, in a sense, development was training us for flight in ways we could not have anticipated.

At NASA Ames, after Day 3, Shift A (my shift) transitioned from “execution” to “planning”.  As part of our staffing plan, Shift B exchanged roles with us to take TCM 3, Star Field Cal  (Day 4) and Lunar Swingby (Day 5) execution.  The execution shifts end up being fully committed to commanding maneuvers and other spacecraft activities, so when we started having our cold thruster issue, our Shift A became the de facto shift for coming up with solutions to the problem.  You see, LCROSS was designed to manage itself thermally if we maintained its orientation correctly relative to the sun.  The trouble is that our thrusters are getting too cold, even when we’re sun-pointed. 

Operationally, this means until we design a good thermal control mode, we have a high-maintenance spacecraft.  Our goal was to find a control approach that would enable our spacecraft to manage itself again.  At first we had to change the orientation of the spacecraft every 2-3 hours, sometimes more.  Mixed in with our other numerous activities, this was really difficult to manage.  In collaborations between NASA Ames and our partners at Northrop Grumman, we started experimenting with different orientations, warming one side, warming the other side, entering control deadbands to fire thrusters more often, etc.  No strategy worked for more than a few hours.  Then we found an orientation that was more effective than the rest – where we tipped the “top” of our spacecraft a little bit toward the sun (negative yaw angle).  The thrusters at the bottom of the spacecraft were warmed by heaters, and the ones at the top, whose heaters don’t activate very easily, were warmed by the sun.  On my Day 3 post, we tipped 10 degrees to the sun (-10 deg in yaw), and things looked great.  I came back on Day 4, and found to my disappointment that the thruster temperatures fell too low after several hours.  But, this approach was a great discovery – maybe if we pushed it a little further, things would balance.

For Day 4, we had to improve our thermal performance, and also oversee the planners in creating commands for TCM 3 and Star Field Calibration.  We set to work.  Our first experiment was to nudge our attitude a little further in yaw, to -20 degrees.  Technically, this violated a flight rule that protects our spacecraft against thermal damage.  The LCROSS spacecraft was designed to fly with its solar array pointed directly towards the sun.  Outside the bounds set by our thermal flight rules, the concern was that perhaps the parts of the spacecraft newly exposed to the sun would overheat, and those newly shadowed from the sun would get too cold.  But confronted with an urgent problem, the flight team had to explore options that were outside of the nominal flight envelope.  Our hope was that the thermal subsystem was designed with some conservatism, and that it would allow us to off-point  from the sun just a little bit. 

So, we rotated to -20 degrees yaw, and watched, and waited.  The behavior we observed at -10 degrees on Day 3 improved, and magically, the bad effects we feared might bite us never emerged.  In the short term, the -20 degree yaw bias seemed to do the trick.  Thruster temperatures on the top of the spacecraft really stabilized, and those on the bottom continued to be warmed by their heaters.  From there, our engineering team wanted to explore the thermal sensitivity to these yaw changes, to understand how vulnerable we’d be if, for whatever reason we moved even further in yaw away from directly sun-pointed.  Would we get too hot, or too cold?  We needed to know, so we commanded the spacecraft just a bit further, to -25 degrees. 

Unfortunately we were so focused on thermal issues that we neglected to check the safety of this new attitude from other perspectives.  As soon as LCROSS began rotating, our Star Tracker (STA) “faulted” and dropped into its Standby Mode.  I knew immediately what we had done.  As we pointed our solar array down, the STA, mounted on the opposite site of the spacecraft, pointed more directly to the Earth.  The Earth is bright, and the STA cannot process star fields to determine our orientation if the sun, Earth or moon enter into its field-of-view.  So, our spacecraft reverted to relying only on its inertial reference unit (IRU) for forward-propagating our attitude.  No immediate threat.  Our IRU is quite capable of keeping track of the spacecraft orientation, so none of us were immediately worried.  However, I started worrying about TCM 3.  If, for whatever reason, we couldn’t get the STA re-initialized soon, I wanted to know whether it would suffice for accurately achieving the burn orientation for TCM 3.  As Flight Director, you always have to think ahead to the consequences of your actions.  I was disappointed that we hadn’t anticipated the entry of the Earth into the STA field-of-view, but we needed to proceed calmly and to right the wrong.

Our first action was to return to -15 degrees, to make sure the Earth would not interfere with the STA and to begin the STA recovery process (blinding is temporary, but requires a few minutes).  Of course by this time TCM 3 planning was going on, and the Flight Director on “planning” shifts has to help oversee the creation of new command sequences.  I missed the initial coordination meeting that nails down the precise set of activities we’d be performing later that night, but caught up with the process soon afterward.  Once I saw things were on track, I returned to the Mission Operations Control Room (MOCR) to continue working the STA issue. 

As it turns out, the STA recovery process is a little more involved from our standard Cruise configuration than from our initial Sun Point Mode from Day 1.  In Sun Point Mode, the process to initialize the STA takes only a few minutes.  In Stellar Inertial Mode, one has to re-initialize the spacecraft’s attitude estimator, a piece of software that merges the data from the IRU and STA into a single, optimal estimate of our current orientation.  Not a big deal, really, but time was beginning to get short.  As it turned out, with other spacecraft activities we’d had to perform, the team did not have time to re-initialize the STA before the Command Approval Meeting (CAM) for TCM 3 and Star Field Cal.  This meeting requires the presence of both outgoing and incoming teams, along with the entire planning and analysis teams, and of course our Science and Payload teams.  The meeting is important, since it is the venue where everyone gets their last view of the command sequences before execution, to make sure all is in order.  We suspended active spacecraft operations, and with a skeleton crew monitoring spacecraft health, we held the CAM.  In the CAM, armed with strong evidence that the burn could be safely and accurately conducted without the STA, we resolved that if time for STA recovery ran out prior to the burn, we’d continue with TCM 3 anyway and recover it later.

With the CAM completed, time was critical.  First, Shift A and the oncoming Shift B held a Shift Handover meeting to communicate all that had happened while B was gone, and to make sure Shift B was fully aware of the current state of LCROSS.  With little time to spare, Shift B took over operations in the MOCR, working towards getting the TCM 3 commands loaded in time.  I watched for a while, as Rusty, the Shift B Flight Director, moved to get everything done.  One concern was that our DSN site had had equipment problems on the Shift B from the night before, preventing commanding for a significant period.  Now we were over the same site, and we couldn’t afford an outage.  Things were going well and then bam!   Suddenly, there was a commanding outage, and things ground to a halt.  Never a dull moment!  Watching from the sidelines, I knew Rusty was in a real bind.  The station worked to resolve the issue with its ground equipment, and all of us in the MOCR started sweating.  But Shift B kept from panicking, and within minutes worked with DSN to get the problem resolved.  They really pulled TCM 3 back from the brink.  With only 10 minutes to spare (now too little time to recover the STA), Shift B got the commands into the spacecraft and the ball was rolling again!  I congratulated the team, and headed off to bed, confident that TCM 3 would go off without a hitch. 

It was a very satisfying moment, when the team really came together in a pinch to keep our operations on track.  Great job Shift B!

Real-Life Operations: Day 3

Written Saturday…

Before I begin, I want to let everyone know that our TCM 2 went off on-time tonight and just as planned (23:15 UTC, exactly 24 hours after TCM 1).  Another important goal met as we fine-tune our target trajectory leading up to Lunar Swingby.  Things continue to be a lot of fun, and we’re learning more each day we fly LCROSS. 

It is a rare event when a spacecraft launches and has no problems.  LCROSS is no different than other spacecraft in that regard, though it has certainly tended toward the problem-free end of the spectrum.  A Flight Team hopes for boredom in flight, because boredom can only happen when operating a perfect spacecraft.  The rosy picture I painted of LCROSS’s first two days of operations is largely accurate.  With so many possible failure scenarios to think of before launch, when our spacecraft displayed such good behavior in its first days, I breathed a huge sigh of relief.  We have discovered a few problems, and our team is really excited to work on them, because this is the challenging part, and exactly why we trained so hard.  I would be remiss if I glossed over the problems.  Another mini-novel for Day 3!

So, what are they?

1. Centaur Gas Leak

I tried to find the time to put a full, detailed posting on this prior to launch.  I never found the time.  I can’t wait any longer, since it’s important to my discussions.  This isn’t the kind of problem you can casually mention in a public e-mail, and then not clarify what you mean!  So here goes.

The Centaur upper stage has a distinguished history of successful launches and spacecraft deliveries.  Its missions typically last no more than a few hours – enough time to deliver a spacecraft payload into a target orbit, and then to shut down.  Well, LCROSS came along and proposed to use the Centaur as an impactor – a pretty out-of-the-box idea.  Great!… but at the time we didn’t know which rocket we’d be riding on and then once the Atlas was selected, very little information about Centaur long-duration behavior was known, due to the nature of its standard, short mission.  LCROSS really pushed the bounds on the Centaur design.  As an example, recall that LCROSS separates from Centaur only 10 hours or so before impact?  Well, the Centaur separation mechanism has to function 120 days after launch – unheard of in launch vehicle circles. 

We learned about Centaur’s potential to leak propellant gases out of several possible valves and vents after launch.  For most programs, this is not an issue, because by the time this happens, they’re separated from the Centaur, and unperturbed by the leaks.  The Centaur goes through all sorts of maneuvers to deplete itself of propellants.  Still, it is actually quite difficult to empty it completely. 

A couple of months before launch, we started getting a better picture of possible size of the leaks, what types of torques they might impart, and for how long during the mission they might occur.  The range was anywhere from virtually no effect, to mission-ending.  Basically, a constant leak could, in certain conditions, cause our LCROSS attitude control system to use all its propellant fighting the attitude disturbance.

This became one of LCROSS’s greatest risks, and the Flight Team spent most of its time in the month before launch preparing and testing special attitude control schemes to be able to mitigate the effects of such a leak, some of them significant departures from our normal operating configuration.  This had a lot of people worried, including us. 

So, what did we find in flight?  Well, we certainly display all of the symptoms of having a Centaur leak.  However, they are different than we expected, and overall a lot more benign.  First, following our pre-mission strategy, we have measured the disturbance the leak imparts by temporarily disabling thruster control and collecting attitude data to derive a torque magnitude and direction (assuming fixed mass properties).  Our torque seems to have changed direction more than once.  A discrete event, what we’re calling a “burp”, seems to have caused a dramatic improvement.  Second, we expected one of two scenarios – one where the leak would be very large but would decay quickly as gasses depleted, and another where the leak would have a mild effect, but would be long-acting.  The second was actually a more harmful case, since our attitude controller might have to fight it for months. What we actually seem to have is a benign combination – a relatively mild leak torque, but one that decays pretty rapidly.  We’ve seen noticeable decrease of the effect on each day, and already our spacecraft is almost acting like normal already.  I’ll bet by Lunar Swingby we’ll be done with this one.  I couldn’t have asked for a better leak outcome.

What did we do to fight this one?  Well, we used our pre-flight strategy to save propellant using some tricks of the trade.  First, we increased our minimum “pulse width” or thruster on-time for control, to improve the efficiency of each thruster pulse, thereby saving propellant.  We also tried firing thrusters in pairs rather than in fours.  As I alluded to in my summary of First Week Rehearsal, this ends up being more efficient, but also perturbs our orbit and makes our Navigation team unhappy.  But as it turned out, the efficiency had an unexpected consequence (see #2), and we couldn’t use it for long.  Thankfully, this leak ended up being quite benign. 

2. Cold Thrusters

This one has been a little more tricky. Our thrusters are mounted on four posts mounted circumferentially around the spacecraft, aligned with the axis of symmetry of the Centaur.  Each thruster has a pair of redundant heaters, a set of thermostats to control heating, and a thermistor, which is a temperature sensor that we read in telemetry to monitor thruster valve temperatures.  When the thrusters get cold (for example when they are shadowed from the sun behind other parts of the spacecraft), the thermostats are designed to activate the heaters to warm the thruster.  Well, for LCROSS, we’ve discovered that not all of the thermostats are activating their heaters before the thrusters fall to very low temperatures.  Thrusters are very important to our mission, and so keeping them warm is a big priority. 

Here’s where it gets interesting.  LCROSS is “three-axis stabilized” – it controls its orientation in all three of its axes to keep it in a fixed orientation in space.  As is typical for spacecraft, LCROSS is allowed to rotate a bit in each direction.  Beyond those limits, thrusters will fire to keep the spacecraft from rotating any further.  Inside those limits, the spacecraft is free to move unhindered.  That region is known in control circles as the “deadband”.  A normal spacecraft will tend to bounce from one side of the deadband to the other.  But since LCROSS is under the influence of a Centaur gas leak, it hugs one side of the deadband all the time.  Our deadband keeps our solar array pointed at the sun, and is supposed to keep our thermal conditions “happy”.  Hugging the side of our deadband means that one set of our thrusters tends to be in shadow, and other set tends to be sunlit.  The shadowed thrusters get cold, and the sunlit ones stay warm.  Our normal Cruise State uses an attitude control mode with a 10 degree deadband, so we can drift up to 8 degrees or so on a single axis before a thruster will push us back. 

So, what did our team do?  Our team tried using a smaller deadband so that our leak would only push us over by 1 degree instead of 8.  This caused our thrusters to fire, which warmed them up temporarily, but once we achieved the tighter pointing thruster firings didn’t happen frequently enough to keep the offending thrusters warm, and the sun orientation didn’t seem to help either.  Then we tried a new approach.  We rotated LCROSS about its long axis to expose the cold thrusters to the sun.  This had a nice stabilizing effect on the cold thrusters, but then the thrusters on the opposite side (the ones now pointed away from the sun slightly) began to get too cold, and the mirrored thrusters also did not activate their heaters.  Finally, at the end of tonight’s shift, our team rotated LCROSS in the other direction – a rotation that pointed the top of LCROSS slightly toward the sun (10 degrees), and the Centaur away from the sun.  When I left tonight, this was looking very promising.  The offending thrusters were exposed to sunlight, and they looked like they were keeping warm.  The shadowed thrusters had been demonstrating their automatic heating, and this trend continued in this new orientation.  So, the upper thrusters seemed happy in sunlight, the shadowed thrusters seemed happy under heater control.  And since we have such strong power margin, we can sustain this from a power standpoint.   We still need to experiment more, to make sure other things don’t get too hot or cold, but things are looking up here!

We have a few other issues, but this is all I can afford tonight!  I’ll hold here, and will keep you all posted on how things evolve.  Our team spirits remain high, and we’re looking forward to the problem-solving in the days ahead.  Finally, our team is recognizing the difference between how mission operations feel in simulations, and how they feel for real.  I am confident that we’ll make it through these problems.  We have a lot of smart, determined people, and these problems are not insurmountable.  More tomorrow!

First Step and First Glance: Day 2

Another late night for Shift A…we are staffing from around 11:30 AM to about 1 AM the next morning, so I don’t get to writing until the wee hours.  Today was another great day at NASA Ames as our team settled into our mission plan and continued through a number of “firsts”.  Each operations shift is trained in two roles – one with a “planning” emphasis, and the other with an “execution” emphasis. Shift B has taken the “planning” side of things for the first few days.  However, last night, in addition to planning, Shift B continued Activation & Checkout where Shift A left off.  They performed a full checkout of our attitude control system, among other things.  For the first few days, Shift A (my shift) gets the fun, with Activation & Checkout (yesterday), Trajectory Correction Maneuver 1 (TCM 1) and “Quicklook” today, and TCM 2 tomorrow.  Shift B takes over the fun for the remaining two days, with TCM 3, Star Field Calibration, and the capstone event for the week – Lunar Swingby.

Today, we had two major events.  TCM 1 is our most important “burn” for the first week.  It is intended to remove any targeting errors introduced by the Centaur upper stage as it injected us into our Trans-Lunar Orbit. Our navigation team determined our Centaur delivery was quite accurate, but it’s actually a very lucky day that a launch vehicle places a spacecraft precisely on its target orbit.  It’s not a knock on the launch vehicle industry, it’s just that they have a really difficult job.  Hey, it’s rocket science after all!  Our Mission & Maneuver Design team designed TCM 1 and arrived at a burn plan with 8.06 m/s “delta-v”, the resulting change in velocity, for a total commanded on-time of 9 minutes 40 seconds.  Coupled with the burn was a “pitch flip” to re-orient our primary omni-directional antenna towards the Earth. 

The burn went off better than in any simulation.  We had our command sequences loaded 30 minutes early, and the LCROSS spacecraft operated far more smoothly than our simulator ever did.  Our power system is working tremendously – our solar array was providing positive charge to the batteries while pointed 60 degrees off the sun.  Attitude control was smooth, and our 22 Newton thrusters performed as we had hoped.  Our mission-critical maneuver, one that we had spent so many weeks worrying about over the past few years, was over before we knew it, and our trajectory is now far better matched to the target.  Only time (and our Navigation team) will tell precisely how it went.  All initial signs are positive!

Our second goal was to perform “Quicklook”, an extension of Activation & Checkout, but for the science payload.  The LCROSS payload has a special place in the hearts of team members, since it was developed here at NASA Ames.  The scientists and payload engineers poured countless hours into instrument development, integration and testing.  All of the spacecraft team watched them do it, and know what effort they went through. 

Quicklook is a fairly simple functional test of the science instruments, the Data Handling Unit and of general science data flow from the spacecraft to Earth.  An onboard commmand sequence powers the DHU, which in turn powers the instruments, and commands each instrument to operate for a few minutes and to return data.   To simplify the test operationally, we make no attempt to point the payload to the Earth or moon yet, so to anyone but a scientist on our team, the results aren’t very exciting.  But to the Science Team, the data speaks volumes.  Quicklook might be a simple and mundane test, but the results are critically important to LCROSS.  Our mission is all about the science payload in the end, and problems here have a direct negative effect on our bottom line. 

The Flight Team kicked off the command sequence and watched as our payload successfully powered up, and began sending data.  One by one, the instruments delivered.  The Science team displayed the data in real time, and we watched the bit streams turn into images (very black, since we’re looking into dark space) and spectra.  Quicklook was finished in 23 minutes, and later this evening, the Science and Payload teams went home looking very satisfied!  Another milestone achieved, and a good result for the more interesting payload tests in the next few days.

Other than the planned events, our team has been busy characterizing how the spacecraft REALLY behaves.  It’s one thing to know the design, and how it SHOULD operate.  But models are only so good, but the actual physics of a spacecraft in its environment define how it will act.  It’s amusing because in several cases, “problems” we detected we later realized are intended or at least understood behaviors of the design, initially misinterpreted.  We’ve been working with LCROSS for years, and only now, as it recedes away from Earth, are we truly learning its personality. 

As a closing thought, I wanted to relate a common thread of thought amongst our team.  For those of us who have been in the same room as the LCROSS spacecraft so many days/weeks/months, it is a strange thought that we just shot that familiar piece of hardware off the Earth and sent it on its way to the moon, and despite that enormous distance, we can still “talk” to it like we did in the clean room.  You’d think years of preparation would dull us to the idea, but the Flight Team is struck by how similarly the flight is to those days in tests! 

As always, thank you for the great comments, questions and well-wishes.  More tomorrow!  GO LCROSS!  Now, time for sleep.

Fly Me to the Moon

I just got off my first shift of the flight about 25 minutes ago…this is a day I will never forget.  I was very confident we’d launch today, but then storm clouds advanced from the North, anticipated for arrival over Cape Canaveral at the time of our scheduled first launch opportunity.  My confidence dropped a notch or two.  Sure enough, with time to spare, the launch team opted to push to our third and final instantaneous opportunity to avoid the weather with greatest certainty.  Our launch hung in the balance, with weather on one side, and a the launch countdown progressing methodically on the other.  Magically, with about 13 minutes until the T-Zero time, the launch site weather classification changed from Red to Green, and the countdown marched without further hesitation.  We realized it was almost inevitable at that point.  Then…Liftoff! 

The Flight Team watched the feed from NASA TV from the Mission Operations Control Room (MOCR) at NASA Ames as we at once cheered this early success, and grew accustomed to the gravity of this new and suddenly very real responsibility – the success of LCROSS was certainly ours to keep, or to lose.  The cameras mounted on the Centaur provided excellent views of the receding Earth, our burst from the low-lying cloud cover into sunshine, our booster separation, and payload fairing jettison.  The forward-looking camera captured LCROSS and its prominent Star Tracker as the fairing opened and exposed it directly to space for the first time. 

At this point, time seemed to slow for a while as our primary focus became setting up our ground antennas for acquisition.  As we approached our expected power-up time, the DSN began tracking the Centaur carrier signal and we knew they’d be on target to find LCROSS, as long as it could begin transmitting.  Then the Centaur sent its commands and time accelerated.  Right on time, the DSN at Canberra found our carrier and locked on to our telemetry signal.  Data, and initial signs of life.  Everything indicated a healthy spacecraft.  Our training kicked in, and we were operating just as we had in all of our rehearsals of Activation & Checkout. 

The spacecraft has been a joy so far.  No major issues to report (if there were, I honestly wouldn’t be able to report them like this anyhow, but you’ll have to take my word for it).  We’re learning LCROSS ideosyncracies now – the realities that differ from the ideal behaviors models predict.  So, cautious optimism so far.  Our timeline this evening progressed a little more slowly than expected, though our final attitude control handover from the Centaur to LCROSS, triggered by Centaur hydrazine depletion, was a lot earlier than expected.  We rolled with the change, and kept on pushing.  Incidentally, preliminary indications are that our Atlas and Centaur placed LCROSS into a very nice orbit. Our Nav team will refine those initial estimates tonight.

Just an hour ago, Shift A left Shift B with a healthy spacecraft.  Shift B will perform a full shakedown of our attitude control system, and run through our available downlink communications rates.  Then they’ll oversee planning for our first Trajectory Correction Maneuver, TCM 1, and our first powerup and checkout of the LCROSS science payload.  And Shift A comes back in tomorrow to execute them both.  TCM 1: burn start at 4:15 PM Pacific.  Quicklook later the same day.

I will not likely be able to write like this every day, but I could not resist telling you all how this day went.  Thank you everyone for your pre-launch good wishes.  We got more than we bargained for today!  GO LCROSS!

First Shot

This morning when I woke up, I told myself, well another two days of preparation, and we’ll have our shot on Friday.  I woke to the news of the continuing problems with the STS-127 hydrogen leak, and all indications from KSC were that we’d have to slip to June 19 given the late hour of the discovery.  Then, I started reading e-mail – a firestorm of activity!  “We’re GO for June 18!”  Whoa!  The day sent me from zero to 100 in a few seconds, and it didn’t let up all day.  We held our last Flight Readiness Status meeting today.  With only a few items remaining until the evening, and now completed, I can honestly now say we are ready to go.  As good as we’ll ever be!

On the eve of our first launch opportunity, I marvel at this journey our team has taken so far – from LCROSS being a bunch of cool ideas on paper at NASA Ames, to now, as a functioning spacecraft encapulated atop an Atlas V launch vehicle, standing at Cape Canaveral, waiting for the signals that will send it from the Earth and to the moon.  From a scrappy, inexperienced team of largely intelligent systems researchers, systems engineers and once-operators, to a cohesive, well-trained, ready-to-fly team.  Hard to believe.  And this started for me as a side project I might devote a few hours a week to.  Life is amazing.

I can’t stay up too late writing tonight, but I did want to share a few things.  First, an update to the THE BANNER.  There will be only one better picture than this one (tomorrow’s), but I don’t know if I’ll have time to take its photo.  This one actually has me in it!  L minus one day!

Second, here is a picture of our launch vehicle, from earlier today, after roll-out to the launch pad.  I only hope we can get weather like this tomorrow!  When you see the scale of the Atlas V with people walking at the base, you realize this is no model rocket.  I can’t wait to watch the launch tomorrow!

Photo courtesy of NASA Kennedy Space Center

Finally, I have to say you readers out there are great!  You’ve written a lot of great comments and questions.  I only regret that I haven’t had the time to reply back to each of you.  Let me answer a few questions now.  I’ll paraphrase the questions:

1. During the First Week Rehearsal, your navigation team must have been going nuts trying to compensate for the changing thruster configuration.  Yes.  The person who asked this question must have Navigation experience!  LCROSS has no momentum wheels to maintain its attitude or orientation.  We fire thrusters instead.  Of course, we’d completely disturb our trajectory if we didn’t do this carefully.  LCROSS uses a well-proven technique of firing thrusters in fours, such that they induce a torque about the intended axis (to turn the spacecraft), but they cancel out their linear force components.  When our team was forced to adjust our thruster firings to avoid the hot thruster, we had to go to a paired configuration that HAD to affect the orbit.  Navigation wasn’t thrilled.

2. Why does LCROSS alternate between a North pole flyby and a South pole flyby?  It comes down to orbital mechanics and hitting the moon at the right angle and with the right lighting and observation parameters.  Imagine that when we fly out to the moon, to insert LCROSS into an Earth orbit that is highly inclined relative to the moon’s orbit, we can fly over the North pole or the South pole.  Once we perform the flyby, we can either try to hit the moon when LCROSS’s orbit intersects the moon’s on the opposite side of the Earth, or on the same side of the Earth.  If we fly over the North pole, then we swing over the top and then below the moon’s orbit.  In that case, to impact the South pole, we’d have to hit the moon on the opposite side of the Earth from where the flyby happened from below).  In the South pole flyby case, we’d swing below the moon and then up over the top of the plane of the moon’s orbit.  To hit the South pole of the moon, we’d have to fly all the way around the Earth at least once, so that we could come from underneath the moon again.  Either way works.  If each of LCROSS’s orbits is roughly one month long, then the North pole flyby leads to mission durations that are 0.5, 1.5, 2.5…months long, since we always need the half month to hit the moon from beneath.  For South pole flyby orbits, since we need an even number of orbits, our mission is an even number of months (1, 2, 3…).  Our mission designers are seeking a sweet spot in observation conditions, arising in October.  This means they have to chose either a South pole or North pole flyby so that they can match the impact timing they want.  Our launch opportunities are every two weeks (half a lunar orbit apart), and our flyby poles alternate each opportunity, so that we’re always hitting the moon on the same side of the Earth (roughly).

3. Finally, one of my readers suggested a really great idea.  He wanted to know if our Centaur would be venting gas as part of its sequence.  The reason he asked is that under the right lighting conditions, the sunlight high above Earth can actually illuminate the vented gases, making them visible from the ground.  He said he was able to observe this effect as a kid right after the Apollo 14 launch.  Cool!

So, I’ve tracked this down, and our Navigation team has provided some data for those of you with enough experience might be able to use to predict LCROSS’s early-mission trajectory.  Here is the information and some figures (attached).  Best for our friends in the Southeast…and by that I mean Australia, Indonesia, Singapore and Thailand…enjoy…and thank you for such a great idea!  From our extended Navigation team:

“I have used the P10 trajectory for 18 Jun in SOAP to assess visibility starting at separation from LRO. Attached are two SOAP screen shots showing links to some major cities. The yellow line running NNW to SSE is the terminator. Locations sufficiently to the left of this line (preferably with Sun elevation less than -12 degree though this limit occurs early) would be best situated to observe the blowdown event assuming it occurs before the sky gets too bright. Perth is most favored, with Sun El -6 degree at 22:49 UTC, almost an hour after LRO separation.

 

      The red line is the ground track (leading part only). The yellow line extending to the right is the link to the Sun (LCROSS in sunlight). The time ticks on the trajectory line (lower left, gray; upper right, light blue) are spaced at five minutes. There is also a light blue link from LCROSS to the current position of the ground track.

 

      We have posted the LCROSS trajectory on the JPL Horizons system (18 Jun P10):

 

telnet horizons.jpl.nasa.gov 6775

lcross

 

Availability of the LCROSS trajectory on Horizons has been announced on Seesat-L (where those interested in observing fuel dumps from Centaur rockets have been known to gather):

 

www.satobs.org/seesat/Jun-2009/0077.html

 

Horizons can be used to generate topocentric ephemeris data.”

A Window Into LCROSS

What a crazy week at NASA Ames.  I really thought I’d be writing more (though some of you are probably wishing I’d write a little less when I actually get a chance to write)!  But there’s so much to write about!

Here we are finally, on launch week!  NASA Ames has been really getting excited about flight missions, with Kepler, Pharmasat, and now LCROSS.  Lately, the Center has been posting countdowns for each mission on the auditorium that most of the Center’s employees drive by on their way off base.  Let me tell you, as someone on the Flight Team, this really drives my adrenaline through the roof.  To see that number get into the single digits, and now with only 2-3 days left!  Each time I see this sign, I say to myself “hoooahh, let’s go!”

Photo Credit: Paul Tompkins

Since our First Week Rehearsal, ending June 1, we’ve been pushing hard on improvements to make this mission go as smoothly as possible.  Aside from our self-induced tests, there have been a lot of hoops to jump through to be approved to fly  – for example, NASA Ames management, led by the Ames Office of Chief Engineer, has to sign us off on a process known as Certification of Flight Readiness, or CoFR (rhymes with “gopher”).  We’ve spent quite a bit of time with the review team, proving that requirements are verified, ground software and hardware to support the mission are in place, and things like procedures, processes, and flight rules are all ready to go.  Just like taking exams at the end of a semester, it’s a worthwhile process, but that doesn’t make it fun!

As everyone knows, LCROSS is sitting atop our Atlas V launch vehicle now, and our team at NASA Ames is waiting for its chance to launch.  We are all anxious to get our mission started.  We’ve trained for over a year, and now we’re ready to prove ourselves.  Due to the slip in the Space Shuttle launch to June 17 (for the latest information see https://www.nasa.gov/mission_pages/shuttle/main/index.html), our earliest date is now June 18, but only if the Space Shuttle decides they can’t make June 17 later on Monday (or according to some, as late as Tuesday June 16).  This is work in progress, so stay tuned while things get sorted out.

Our June 17 launch block has four days, each with three launch opportunities, for a total of 12 possible times we can launch.  With Space Shuttle moving into June 17, that leaves us with 9 more shots in this block, before we have to slip to June 30.  Here’s the rub – not all launch blocks are created equally.  This launch block is really good, the next one isn’t as good.  Here are the factors:


Photo courtesy of NASA Kennedy Space Center

First, the engineering perspective.  With each launch block, LCROSS alternates between performing a flyby of the moon over the North pole and South pole of the moon on Day 5 of the mission, in order to enter its Cruise Phase orbit that will ultimately intersect the Moon.  The June 17 block has us flying over the South pole, which is far less expensive in terms of propellant.  We end up having a lot more propellant to fight other problems that could occur, and that makes our mission less risky on this block.  Our earlier block at June 2, and our next one starting June 30 involve North pole flyby, which is more expensive.  Also, our June 17 block results in orbits that are far closer to the Earth at their furthest point (apogee) than for our June 30 block (570,000 km vs. 700,000 km).  This means we can avoid re-orienting our spacecraft to communicate with Earth on our Medium Gain Antenna on this block.   It means another propellant savings, and a less risky mission.

Second, what we’re really all about – Science.  Our impact geometry is excellent on this block.  One of our criteria for a good impact is the angle at which we can feasibly impact the lunar surface.  There’s actually a strong dependence of this angle on launch date and time.  For this block, our orbits enable impact angles of over 80 degrees, even though our requirement is for greater than only 65 degrees.  This means we can impart more momentum into the lunar surface, and will be able to raise the lunar regolith “ejecta” cloud that much higher into the sunlight where we can analyze it for signs of water.  Better for science.

One of the components of the LCROSS science campaign has ground-based observatories watching our impact from the ground.  The LCROSS science team at NASA Ames has worked with several observatories in the continental US on this end of the mission.  Sadly, as we slip later into this block, the observatories in the US begin to be too close to sunrise to “see” the impact.  By the 20th, there may be too much sunlight at impact time in the US, outside of Hawaii, to get good observations here.  So, there are consequences to moving later.  Of course, our primary science observations, straight from LCROSS itself, are going to be great either way.   If you have time, check out the LCROSS science briefing, scheduled for 6/16 at 9:00 AM Eastern.  Tony Colaprete, our Principal Investigator, may address this issue in his talk.  Our team will certainly make the best of what we have!

NASA has a difficult job balancing the priorities of missions competing for the same launch slot.  Space Shuttle is also very important, and without knowing the constraints and pressures of STS-127, I am unqualified to judge the decision to slip LCROSS and LRO to June 18.  Regardless of what happens, our Flight Team is anxious to get started, and will be ready when we’re given the “GO”. 

Our hope is that we’ll be allowed to take full advantage of this special opportunity we have over next few days to fly LCROSS to its fullest potential.  The window is opening…wish us luck!

Lessons Learned: A Recap of First Week Rehearsal

The LCROSS First Week Rehearsal was all it was cooked up to be.  With a fully realistic operations timeline, long hours, and a representative dose of problems inserted by our beloved Test Conductor, “FWR” was all-consuming.  Our team emerged successful, having achieved all of the major objectives of the first week.  However, this test was not a walk in the park.  Under the stress of continuous day-and-night operations and contending with the challenge of various anomalies, the team learned a number of valuable lessons that we’ll certainly carry into flight. 


Our Goals


What was our aim?  We’d already practiced all of the events we plan to execute in flight many times over.  We’d rehearsed many “contingency responses”, the pre-planned reactions we’d developed in response to a variety of off-nominal conditions on the spacecraft and ground data system.  We’d been pummeled by our Test Conductor, who dealt out nasty fault conditions in the middle of important science events. 

What we hadn’t done is operate the spacecraft for a full week, during the most intense period of the mission, exercising our process to the fullest.  This wasn’t about torturing the team with one evil spacecraft problem after another, nor was it about drilling on a particular event over and over to hone it to perfection.  This test was designed to wear us down.  To see how we’d act on the fourth and fifth stressful day.  To see whether we’d loosen our grip, or keep focused and stay fully aware of the pitfalls of the mission.  The honest truth is that we’re likely to find our spacecraft is pretty healthy after launch.  It may have its share of problems, mostly little, devious ones, and hopefully none that incapacitate us.  But, if we’re not disciplined, the Flight Team could be its own worst enemy. 

So, how did it go?  Very well overall, but not perfectly (thankfully – if it had, what would we have learned?).  Here’s a summary:


Day 1: Pre-Launch, Launch and Activation and Checkout

Objectives:

1. Verify ground systems and flight team are “GO” for launch

2. Oversee LCROSS activation, and verify all spacecraft subsystems are fully functional

3. Transition to Cruise State, our nominal operational configuration


For the first week, we’ll be operating with two shifts of operators – “Shift A” and “Shift B”, 13 hours each with overlap for handovers, for 24-hour coverage.  I’m on “Shift A”.  The long hours are part of the challenge of this mission in the first week.

On launch day, our “Shift B” takes care of final testing of the ground system prior to liftoff.  They test to make sure all of the pieces of hardware and software that support the mission are ready to go (computers, voice loops, networks, DSN antennas, people).  On the first morning of our simulation, “Shift B” determined ground systems were a “GO” for launch.  “Shift A” came on shift to staff the liftoff, which for our simulation was nothing more than a countdown timer that finally reached “zero” and then began counting up.  Using our imaginations, we tracked the event timing of the Atlas, as we will in flight, with help from our operators sitting at Kennedy Space Center.  The ascent was flawless.  Before long we were on the way to the moon, separated from LRO, and ready to wake up.  Our first operational hurdle – activation – was upon us.  The Centaur sent its discrete commands on time to power LCROSS.  Then, the horrible wait – when LCROSS is powered up, but not yet transmitting telemetry.  The mission rides on that moment.  If DSN doesn’t detect a telemetry signal after a few minutes, we know we have problems.  But finally, DSN reported detecting the “carrier” frequency, and soon afterwards locked on to the “subcarrier”, the signal carrying our data.

We were relieved to find LCROSS mostly healthy.  After the tense moments as DSN “locked up” on telemetry, the engineering team performed an initial checkout, and all systems looked more or less nominal.   Soon, we established our “uplink”, the commanding signal, and we spent the next few hours performing tests and gradually building up to our full operational configuration.  Everything was going great!  But then came our first setback.

Having reached “Cruise State”, we expected our spacecraft to behave fairly predictably.  What we noticed is that our thrusters were firing far more frequently than they should to keep us in a stable orientation.  In fact, we were burning propellant a LOT faster than we should have been, and this posed a great long-term threat to our mission.  To be honest, this was not altogether unexpected.  I won’t get into details here, but the LCROSS team has been working hard for weeks to address this very problem.  Call this blatant foreshadowing of my next post!

With this problem detected, “Shift A” handed off to “Shift B”, who took the reins to begin assessing prop usage, and to drop into a special attitude control configuration designed to save as much propellant as possible.   Their main goal, though, while “Shift A” slept, was to plan TCM 1, our first and most important trajectory correction.  The Navigation team processed simulated orbit tracking data, and determined that our Centaur provided us with a very accurate insertion, which proved to knock down the size of TCM 1, intended mainly to remove errors in our delivery orbit.  The planning team made it through its first day, almost without a hitch.  They missed something important, and later so did “Shift A”. 


Figure: Depiction of LCROSS First Week Rehearsal events imposed on trans-lunar orbit.  The LCROSS trajectory is green, and the Moon’s is blue.  Each “tick” on the trajectory is 6 hours.  Dates are in “day-of-year” and hours:minutes.  Note how we cover a lot of distance each six hours in the beginning of the mission, but then slow down dramatically by the time we reach the Moon.  Kepler’s laws in action!  The inset shows the overhead view.  Courtesy of NASA Ames Research Center


Day 2: TCM 1 and Payload Quicklook


Objectives:

1. Estimate the Centaur injection orbit

2. Plan and create command sequences for TCM 1, designed to correct errors introduced by the Centaur

3. Execute TCM 1

4. Perform checkout of science payload (Quicklook)


“Shift A” returned to the MOCR (Mission Operations Control Room) to find the TCM 1 burn plan ready to go.  Everything looked fine.  Exactly like we’d practiced so many times.  Almost.  And that’s exactly where we all discovered our first systemic mistake. 

“Shift A” sat down on console and prepared for the “burn”, loading the command sequences that would control the finely-tuned steps that safely configure LCROSS for a “delta-v” maneuver.   Suddenly, our attitude control engineer noticed we were in a non-standard configuration, designed to fight the propellant usage problem I referred to earlier.  As Flight Director on duty, I realized we could not perform TCM 1 in this configuration, and our time to change back to normal configuration had run out!  I ordered the termination of the command sequence.  This was not a happy moment.  TCM 1 aborted. 

How had we all missed this?  Basically we had so much practice under nominal attitude control, and our new attitude control strategy for dealing with the propellant usage is indistinguishable in telemetry, and must be recorded and remembered.  Lack of situational awareness.  Lessons: always maintain situational awareness, and always anticipate how new conditions will affect upcoming activities, and never get complacent.

With no time for emotions, we set our backup plan in action – our standard procedure is to generate commands for a backup TCM 1 plan, four hours later than the main opportunity, just in case.   We had never practiced this process in a rehearsal, so now we set the plan into motion with our confidence shaken.  The planning shift stayed extra hours, but we efficiently developed and checked a new set of commands, and voila, four hours later we redeemed ourselves (slightly) by executing TCM 1, re-engaging our trajectory target.  One of the most poignant lessons of the rehearsal.

We also found that one of our attitude control thrusters was running hotter than it should.

To make up our schedule, we swapped TCM 1 with Quicklook, our first major science payload activity, so that we continued to make progress during the backup burn planning.  “Quicklook” powers all of the science instruments and runs through a quick sampling sequence on each to determine whether they are operating properly or not.  Unfortunately, our Test Conductor dealt us another blow by disabling one of our Near-Infrared cameras (NIR 2).  As far as we knew, that instrument was dead, and prompted the Science team to order another Quicklook the following day to investigate further.

“Shift B” returned again to take control of LCROSS, evaluate our fuel usage problem some more, and to plan TCM 2, our next major event.


Day 3: TCM 2 and Quicklook 2


Objectives:

1. Estimate orbit from TCM 1 and evaluate TCM 1 performance

2. Plan and create command sequences for TCM 2, whose job is to remove errors from TCM 1

3. Execute second Quicklook payload checkout, to investigate NIR 2 camera malfunction


Thankfully, this day was less exciting than the previous day.  “Shift B” gingerly planned another TCM, as well as another Quicklook, being very careful with our hard-learned lesson from Day 2.  “Shift B” also recommended we configure our thrusters to avoid firing the hot thruster I mentioned earlier, to see if that would cause it to cool off.   In flight, we’d have tested this sort of a change on our LCROSS simulator before running it on the spacecraft.  Unfortunately, we were using the simulator to represent the actual spacecraft for the rehearsal, and re-configuring it is not a quick operation, so we couldn’t test the command product.  After some hand-checking, we decided to accept “Shift B’s” advice, knowing that in flight we’d have tested the product in simulation first.

With the sting of our missed TCM 1, “Shift A” came in, checked the command sequences thoroughly, and then proceeded very carefully to make sure we wouldn’t repeat the oversights of the previous day.  That caution was healthy.  TCM 2 went off without a problem. 

Our second Quicklook confirmed our findings from the previous day – NIR 2 was non-functional.  This prompted the Science team to do some in-flight replanning.  Error messages from the NIR 2 were causing the payload to drop some data from the Visible Spectrometer (VSP).  Rather than run future calibration activities with instrument command sequences that included the NIR 2 camera , they developed new sequences that left it out. 

Finally, we uploaded and began using the thruster table that avoided the hot thruster.  Wham!  Within minutes, our spacecraft attitude was drifting off its target.  We quickly reverted to our original table, and we recovered our attitude.  The table was faulty, and hand-checking had not caught it.  This was a lesson we’d already learned, but strongly confirmed again:  always, always test fresh parameter tables before putting them on the spacecraft.  This was a little less painful than the other mistake, since there’s no doubt we would have performed the tests before allowing anything near the spacecraft.  Even so, it didn’t feel good to have that happen.   

“Shift B” returned from their day’s rest to monitor the spacecraft again.  Here in the mission, our roles swap – “Shift A” becomes the planning shift, and “Shift B” becomes the execution shift. 


Day 4: TCM 3 (or not), Star Field Calibration and Payload Sequence Load


Objectives:

1. Estimate orbit resulting from TCM 2, and evaluate TCM 2 burn performance

2. Plan and create command products for TCM 3, whose job is to remove errors from TCM 2, to target our Lunar Swingby trajectory

3. Plan and create command products to execute Star Field Calibration, designed to find the alignment between the payload boresight and our star tracker

4. Execute TCM 3 and Star Field Cal


Our shift oversaw the planning team in the creation of command sequences for TCM 3, our last “burn” prior to our first close encounter with the Moon, as well as for our first payload calibration, Star Field Cal.  The planning shift was very rushed, since the team had to integrate a complicated series of steps to configure for different control modes, different communications rates, and different DSN antennas to support these two events.  What is more, I had arranged to make the LCROSS simulator available to the planning team so that they could check some of their products as we would in flight.  This introduced another layer of complexity.  Feeling particularly rambunctious, our Test Conductor decided to arrange a fire drill with the NASA Ames Emergency Services.  They came in with full firefighting gear, and needless to say, caused some chaos in the midst of planning. 

Thankfully, fortune played in our favor.  The Navigation and Mission Design teams determined that our TCM 1 and TCM 2 plans and executions had gone so well that TCM 3 became unnecessary.  We waived the maneuver.  This discovery bought us more time to complete the command generation for Star Field Cal and to test the products fully.  The tests confirmed the commands, and “Shift B” ended up successfully executing the Star Field Calibration.


Day 5: Payload Sequence Uploads and Lunar Swingby


Objectives:

1. Load new sampling command sequences to the science payload, to avoid use of NIR 2 camera

2. Refine orbit estimate

3. Plan and create command sequences for Lunar Swingby

4. Execute Lunar Swingby and collect first major payload calibration data set


“Shift A” came in the morning with work to be done on the spacecraft.  We performed a load of the new science instrument command sequences, which had been tested on the payload simulator the night before.  After a few other operations, we got to work on planning Lunar Swingby, the most complex of the events in the first week, and perhaps the entire mission. 

Recall that LRO and LCROSS are both going to the Moon simultaneously.  For the most part, we keep out of each other’s way, but at certain times, we both have important things to do at the same time, using the same Deep Space Network resources.  As we approach the Moon, LRO needs to perform its “Lunar Orbit Insertion”, or LOI, the maneuver that puts that spacecraft into orbit around our silvery celestial neighbor.  Meanwhile, LCROSS performs its Lunar Swingby calibration, a lengthy data capture sequence as we slingshot right by the Moon into our inclined phasing orbit.  Ideally, the two events would happen pretty much right on top of each other.  But they can’t, since both missions use the Deep Space Network, and there aren’t enough antennas to serve both missions at once.  LRO is the older brother of the two missions, and under the agreement between LRO and LCROSS, it takes precedence in situations like these.  The LOI burn is critical for LRO – the mission would fail completely if this didn’t work.   The LCROSS Lunar Swingby Calibration is important, but is not mission critical.  So, LRO takes two antennas at once (a primary, and a backup, just in case the primary fails).  LCROSS has to wait for LRO to release its backup antenna to enable the “downlink” of science data for the calibration. 

We worked as carefully as possible to ensure we’d generated all of the command sequences correctly, and had considered every eventuality.  “Shift B” would come in with very little time to adjust to the operational pace in order to load the commands before the LOI burn.  A planning mistake would likely mean problems.  The command approval went off smoothly, and “Shift B” quickly took over from “Shift A”, and soon they were off, loading their commands with fervor.  They finished ahead of schedule, then released their DSN antenna to LRO.

In this point in the rehearsal, my job was done – I’d completed my last shift.  Rather than leave to grab sleep, I decided to make some mischief as an assistant to the Test Conductor!  We schemed a bit, and decided that I’d play the role of LRO Operations Manager.   We devised a scenario in which LRO would need to hold on to their backup DSN antenna for an extra 40 minutes (entirely possible, though unlikely in flight).  This would cause stress with “Shift B”, who would need to reconfigure their communications very quickly to enable them to capture the science data from Lunar Swingby.  Just a little fun for the last day!

Happily, “Shift B” did great. They made the right choices, and had no trouble collecting all of the science data for the calibration.  As a reward, our Test Conductor decided to fail one of the LCROSS primary heater circuits, causing the entire team to scramble to avoid freezing propellant in the lines that connect the propellant tank to the thrusters.  A little slow to react, but they did fine.   The event marked the end of the First Week Rehearsal, and the beginning of the time remaining until launch.


Synopsis


We successfully met the challenge of FWR.  All of our ground data systems worked beautifully.  We practiced every aspect of our operations concept.  Our team succeeded in planning and executing all of the maneuvers to place LCROSS into its Cruise Phase orbit, and performed all of the planned science calibrations.  Our procedures had very few problems. We managed investigations on several minor anomalies, and mitigated the fuel usage issue.  We communicated our daily status to Center management, and to stakeholders at LPRP and Headquarters.  This was as close to real as we could have done.  I want to thank and congratulate my team in reaching this important milestone!

What this test made clear are the human hazards of prolonged operations.  We must maintain our operational discipline day in and day out.  We’ll have to remain vigilant and keep our guard up, even when the spacecraft is operating perfectly.  We’ll need to communicate our observations effectively, and maintain operational logs, so that we don’t let anything fall through the cracks.  And we cannot allow ourselves to become over-confident with LCROSS as we grow more accustomed to flying it.  With these lessons fresh in mind, I’m confident we can succeed.