Month: September 2013

I am starting the writing of this article on the first day of the month so, “rabbit, rabbit, rabbit.”  There, now that’s done with (a silly, harmless old-world superstition shared with me by my mother – look it up!).

Meanwhile back at the ranch, Auntie Jane fell down a well and Jake the hound dog led the sheriff to Granddad’s moonshine still in the barn…

Okay, okay, I’ll stop stalling.  This is the third and final article about the in-flight anomaly on STS-104.  In the first article, we talked about how everything apparently went so well for the first launch of the Block 2 configuration of the Space Shuttle Main Engine.  This was the culmination of over a decade of incremental work to transform the SSME into a safer, more reliable engine.  In the second article, we talked about the “uh-oh” moment when we found pressure rises that, at first, just seemed a little unusual and then, upon further research, were found to be extreme as compared to what we had seen before in the flight program.  In this article, we will discuss why this anomaly happened, why we missed predicting that it would happen, and what we did to keep flying safely beyond that point.

Why it happened, part 1: the rotor
Did you ever notice how much coordinated effort it takes to slow down an airplane as you’re landing?  They drop the landing gear to increase drag.  They slow the engines and drop flaps and tip the nose up, getting more lift but at the price of additional drag.  And when you’re finally on the ground, they throw up more flaps and sometimes use loud thrust reversers and, lastly, they use the brakes on the wheels.  All this effort is necessary because you’ve got this great big thing with lots and lots of energy and momentum and you’ve got to bring it to rest.  That’s a great deal of energy to dissipate.

Now, think about the rotor in the SSME high-pressure fuel pump.  After engine shutdown, it’s spinning down, decelerating, from over 30,000 rpm to zero in just a few seconds.  Just like the airplane in a relative sense, that’s a great deal of energy to dissipate in a very short time.  So, where does the energy go?  Remember, short of Einstein’s relativistic effects (not relevant here), energy is neither created nor destroyed.  It is only transferred.  When we say that was dissipate the energy from the rotor, what we really mean is that the energy comes out of the rotor and into, well, um … what?

Some of the dissipation is due to mechanical friction.  But we’ve got really, really good bearings in that turbopump, and there aren’t any brakes (i.e., the energy dissipation tools used for your car), so friction is a very small piece of the process.

The only other thing that you have is the fluid in the pump, the residual liquid hydrogen left there after shutdown.  Think again about the plane landing.  Many of the things done to slow the plane rely on drag, which is basically relaying on putting energy into the working fluid, i.e., air.  We do kind of the same thing with our working fluid, the residual hydrogen.  That’s why we close the valves the way that we do and effectively lock up the fluid in the line rather than just let it all drain out of the pump portion of the turbopump.  Because the turbine end of the turbopump is no longer being powered and because the rotor is continuously transferring energy to the fluid, the result is that the rotor slows down.  Ta-da!  The plane has landed and the rotor is slowed.

But what happens when you put energy into a fluid?  In the case of the landing plane, the reservoir of Earth’s atmosphere is so huge that there’s basically no effect.  But in the case of the pump, it’s more like the boiling, covered pot on the stove discussed in the previous article.  It is a fixed, trapped volume into which you are putting energy.  Thus, the pressure rises.

“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.”  Well, sort of.  We always expect a pressure rise.  That’s part of the process.  If you go back to the previous article, you’ll see that there was a pressure rise for all three engines.  It’s just that the pressure rise for the Block 2 configuration engine was so much greater.

Remember back to the first article when I was explaining that the Block 2 fuel turbopump was safer, in part, because it was heavier?  We were able to allocate more weight to the designers and so they used that extra weight (along with many lessons learned) to increase the safety margins.  A heavier rotor means that when it’s spinning, it has more energy than a lighter rotor spinning at the same speed.  Thus, a heavier rotor should mean more energy dissipation/transfer and therefore more pressure rise.

“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.”  Again, not quite.  Note that this was practically the same thought path that we followed as we unraveled the STS-104 anomaly.  By this time, we were deep into our analytical modeling efforts.  We had models for the engine transients – in other words start up and shutdown – but we had not predicted this effect to any great detail.  And even when we carefully accounted for the greater mass of the Block 2 rotor, we could not entirely recreate the STS-104 anomaly.  Yes, we did indeed get higher pressures in the line, but not as high as we saw in flight.

Why it happened, part 2: “thermal mass”
Here’s an experiment.  Heat your oven to 350 degrees.  Put potato and a radish, each wrapped in aluminum foil, on a shelf in the hot oven.  An hour later, take them out.  Now, the experiment part is drop them into ice water and observe how much time it takes before you can comfortably pick up and hold each foil-wrapped vegetable with your bare hands.  While I cannot say [confession coming] that I’ve actually done this experiment in preparation for this article, I am quite confident that the radish would cool more quickly.  Why?  Well, there are all kinds of heat transfer equations related to conduction and convection that we could review, but that’s not really necessary.  It’s just common sense.  A hot potato stays a hot potato because it’s a heavy, dense thing.  It’s got what you could call “thermal mass.”  It stores a lot of energy when hot.  Something less weighty like, say, a radish, has less mass and so even if it starts at the same temperature, there just isn’t as much energy to dissipate to bring it back down to temperatures that allow for handling.

Now, the high-pressure fuel turbopump is not a potato, but it does have significant thermal mass.  It’s about the size of an automobile V-8 engine.  Put your hand on the hood of your pick-up truck an hour after you’ve parked and you’ll get a sense of thermal mass.  And just as with the rotor, when we went to the Block 2 configuration, we were able to provide for a larger weight allocation for everything as part of the effort to make a safer component.  So, in addition to the rotor, the housing and all of the internal flow-path elements of the pump were a bit more meaty than previous designs.

But how does this relate to the STS-104 anomaly?  That’s really a good question because it’s not really obvious that it should.  If the pump was full of liquid hydrogen for all that time prior to launch and during ascent, then the pump ought to be the same temperature as the liquid hydrogen.  Just like water flows downhill, heat only flows when there’s a difference in temperature.  If the metal is liquid hydrogen temperature and the liquid hydrogen is liquid hydrogen temperature, then you ought to have a thermal standoff with no heat transfer.  Ahhh, but here is where you have to think in terms of other worlds with respect to the notion of “liquid hydrogen temperature.”

Think about water boiling on the stove.  How hot is it?  It’s just about 212 degrees.  If you turn down the burner so that it’s boiling less, it will still be at 212 degrees.  If you turn up the burner so that it’s boiling violently, it will still be at 212 degrees.  At regular, atmospheric pressure you cannot make water hotter than 212 degrees.  If you add any heat to water sitting at 212 degrees, that extra heat will be released by boiling but the temperature remains unchanged.  However, if rather than our normal 14.7 pounds per square inch we increase our pressure up to 1,000 pounds per square inch, then we can heat the water up to over 500 degrees before boiling starts.  In other words, the boiling point is dependent upon the pressure.  I mentioned this in a previous article in this series.

Prior to launch, we chill down the pump to liquid hydrogen temperatures at near atmospheric pressure conditions.  That’s means about 37 to 39 degrees above absolute zero (or more than 420 degrees below zero Fahrenheit).  While the engine is running, however, because we’re adding lots and lots of work into the fluid as part of the pumping up to over 5,000 pounds per square inch of pressure, the temperature of the fluid coming out of the pump is somewhere between 90 and 100 degrees above absolute zero.  Well, if the fluid is up around 90 to 100 degrees, then the metal of the pump is going to be up around that temperature as well after eight and half minutes of power ascent.  So what happens when we shut down?  The spinning pump rotor slows, comes to a stop, and the pressures through the whole system drop.

Think about that water sitting at 500 degrees Fahrenheit and at 1,000 pounds per square inch pressure.  Now imagine that you gradually decrease the pressure back down to normal atmospheric pressure.  What happens?  The water is going to boil.  It is going to release energy as it cools down to the normal boiling point of 212 degrees at 14.7 pounds per square inch.  This same phenomenon happens in the hydrogen side of the engine during shutdown.  The pressure drops during shutdown and this warmer liquid needs to expel energy to get down to lower boiling point temperatures corresponding to lower pressures.  And, as the liquid temperature drops, then there is a temperature difference between liquid and the metal of the pump.  And what happens when you have a temperature difference?  That’s right.  You get heat transfer.

All of this is the normal process.  It’s all expected.  So, what made the STS-104 experience different with the Block 2 engine?  Answer: The hot potato, i.e., thermal mass.  As mentioned, the Block 2 high pressure turbopump was intentionally meatier than its predecessor.  This additional meatiness helped with the reliability and safety of the unit but it also added more thermal mass and this meant more energy being put back into the fluid.  More energy into the fluid translates to higher pressures in the trapped hydrogen between the main fuel valve and the prevalve.

“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.”  Okay, yes, in combination we have.  Simply put, it was the combination of greater kinetic energy from the heavier rotor plus the additional heat due to the greater thermal mass of the pump structure that led to the elevated pressure between the main fuel valve and the prevalve.  And this elevated pressure, in turn, due to the release mechanism built into the prevalve, is what caused the elevated pressure in the 17-inch manifold.  We modeled all this analytically, both at a higher level for the purposes of running through many “what-if” scenarios and also at a very detailed, multi-node level embedded within the accepted transient model for the engines.

“Why didn’t you see this coming?”
So, in gracious and humble gratitude for solving the riddle of the STS-104 in-flight anomaly, the management community for the Space Shuttle demanded an answer to the following question: “Why didn’t you see this coming?”  And, sometimes, this question was asked with slightly more colorful language.  After all, the Block 2 engine configuration had been in development for over a decade, so it really was a perfectly fair question.  And, with the perfect hindsight, perhaps it was something that we could have or should have predicted.  Fine, I’ll accept that.  But here is the biggest part of the reason why we didn’t really think of it: We never saw any difference on the test stand.

When you set up a test stand to test an engine, there are many competing factors to take into consideration.  Obviously, you want to protect the very expensive engine hardware so you include a multitude of safety provisions.  You also want to get useful data from the tests (remember: you only test engines for two reasons – to impress your friends and to get data).  But how do you define useful data?  Most people would say that useful data is defined as data from testing that most looks like and feels like the actual mission.  But you can’t make the test stand look exactly or entirely like the vehicle and, even if you did, it’s not like you can make the test stand fly through the upper atmosphere at Mach 10 during the test.  In other words, the ground test stand will never be a perfect reproduction of the flight mission.  So you make compromises based upon practicality, cost, and safety.

On the SSME test stand, the prevalves are higher up in the feed line as compared to the Shuttle orbiter.  This means that the trapped volume between the main fuel valve and the prevalve is much larger.  Also, in the shutdown sequence, the test stand prevalve shuts down later than on the vehicle.  And, very shortly after the prevalve shuts down, we bleed the line since the prevalve is not a fancy flight unit with a built-in relief functionality.  All these things combine to provide a very safe, steady shutdown for the engines in the test stand.  Pressure is maintained, but not allowed to rise very much and we have a larger volume of liquid to keep the pump loaded and to absorb any energy input.  It wasn’t just for the Block 2 configuration that the test stand was different from the vehicle.  It had always been different, as in for nearly 30 years.  Thus, for nearly 30 years the test stand was different from the vehicle and the shutdown data between tests and flight looked different, but the differences were understood, accepted, and, because there had never been an issue, largely ignored.  The Block 2 tests looked just like all the previous configurations on the test stand with regards to the shutdown pressures and so a flag was never raised.  It was only when we got to the different geometry and procedures of flight that an issue appeared.

Okay, now what?
We now understood the issue and we understood why we hadn’t predicted that the issue would come up.  We now had to resolve the issue.  We had a number of possible choices, but the most obvious came from the fact that we hadn’t predicted the issue based upon test results.  So we figured that if we made the vehicle act a little more like the test stand, then we could make the anomaly go away.  We couldn’t move the prevalves or make the feed lines any longer, so we instead recommended delaying the prevalve closure by 2 seconds.  In terms of physics and thermodynamics, what is this doing is allowing more of the energy being released from the spin down of the turbopump to leak upwards, out of the engine, and even back into the External Tank before we lock up the system.  Less trapped energy, less pressure rise, but we still spun down the turbopump safely as shown repeatedly on the test stand.  According to our analytical models, this worked well for the engine and the feed lines.

However, the prevalve closure is just the first step in a series of actions taken by the Shuttle in preparation to separate from the External Tank.  So, if we recommend that the prevalve close 2 seconds later, then the whole sequence has to change by 2 seconds including the disconnection of the External Tank.  That required an assessment from our trajectory analysis friends at the NASA Johnson Space Center.  They had to see if mission performance was impacted or if the External Tank reentry footprint (i.e., the place in the Indian or Pacific Oceans where the External Tank comes crashing down into the sea) was altered in an unacceptable manner.

The trajectory analysis results showed that the changes were acceptable and so, on 5 December 2001, less than five months after STS-104, we launched STS-108, Shuttle Endeavour, with one Block 2 configuration engine.  The results were exactly as predicted.  There were no unacceptable pressure rises.  In April 2002, on STS-110, we successfully launched a complete cluster of three Block 2 configuration engines, on Shuttle Atlantis, and had no post-shutdown pressure issues.  The STS-104 in-flight anomaly was officially considered to be resolved.

No, the story of the STS-104 in-flight anomaly will never be made into a movie starring Tom Hanks a la “Apollo 13,” and certainly the stakes that we faced were never that dramatic in finding a resolution (thank goodness).  Nevertheless, it was an excellent example of an investigation in which we used the available data, constructed suitable analytical models, and found a solution to an engineering problem in a manner such that the ongoing sequence of Shuttle launches was never really impacted.  It was an interesting problem and a complete success and I am still, to this day, proud that I had the opportunity to play a role.  [Note, however, that if there are any movie producers reading this and it they want to start casting the parts, Brad Pitt would be an excellent choice to play me.]

 

To begin the last previous article, we discussed water, steam, and ice.  Here, for this article, I want you to think about a boiling pot of water on the stove.  Now, put a lid on the pot.  What happens?  Usually, the lid rattles around, rapping out an irregular rhythm (in my house, that usually gets the dogs barking), and it lets puffs of steam escape with each little wobble.  But what if we made the lid really tight?  Well, in that case, if we kept applying heat, the pressure in the pot could eventually rise high enough that the lid could pop off with a good bit of energy.  If I wanted to exaggerate, then I would say that the pot and lid combo could explode, but that’s probably being just a bit overly dramatic when talking about a normal lid.

 

Now, rather than water, think about liquid hydrogen.  Yes, I know, LH2 is not something that you encounter in your daily life.  Shoot, I’m in the business of rocket engines and I don’t deal with LH2 much other than on paper (or as mathematically represented in computer models).  But just think of it like any other liquid such that, if it is warmed and starts to boil in a closed container, then the pressure rises.  Something to keep in mind.

Now, let’s get back to STS-104.

I returned from my short vacation in Boston back in 2001 and I got down to work by working with the team analyzing the STS-104 data.  After every flight, we get data from measurements on the engine and the Shuttle orbiter and we effectively reconstruct the engine performance to make sure that everything did what was expected.  Please note that the engine folks are not unique in this respect.  The team responsible for the boosters scours the post-flight data from the boosters (and, on the Shuttle Program, they also examined the recovered booster shells).  The team responsible for communications examines all of the communications data.  The team responsible for the External Tank studies, in detail, the tank data and reconstructs the draining of the tanks as another means for confirming that the engines performed as predicted.  For us engine guys, the initial conclusions were, yes, the engines performed as expected and as predicted.  In particular, the new Block 2 engine performed perfectly.  No obvious anomalies on the engine.  Pop the corks because all was good!

However, during the data review there was one moment of, “Huh, what’s that?”  It was over the data shown in the following plot (this is not the actual plot – I reconstructed it for this article using the original data):

What you have in that plot is data from two Shuttle missions.  STS-098 was a launch of Shuttle Atlantis that took place in February 2001.  STS-104, our launch being reviewed, was the next launch of Atlantis.  So, this is a comparison between two back-to-back missions of the same orbiter and that’s important since the measurement in question is on the orbiter side of the interface rather than on the engine side.  Indeed, it’s not a measurement that usually concerns us much.  This is the pressure in the shared manifold from which the three engines are fed liquid hydrogen.  The time period is after all the engines have shutdown.  The thing that was kind of odd was the pressure rise at right around 11 seconds (please note that the data looks “boxy” for a variety of factors including, primarily, data rate and, secondarily, the fact that we sometimes sacrifice fidelity for robustness depending on relative importance).

But, hey, the pressure rise only about 10 psi.  We’re rocket guys.  We deal with thousands of psi all over the place.  This is a blip.  A nothing.  No big deal.  That was, I have to admit, the initial response.  But, of course, we still have to figure out how or why it happened.  The teams looking at post-flight data are always extremely meticulous since it is often this data that tells you that you’re safe to fly for the next mission.  However, there is still a prioritization of issue in your head.  Our biggest concern for this mission was how the engines, particularly the new Block 2 configuration, performed during powered ascent.  Nevertheless, we did come back and respond to the simple question asked, “Huh, what’s that?”

Above is a simplified, cartoon schematic of the liquid hydrogen feed system from the External Tank to the engines.  The “Disconnect” is, exactly as it sounds, the place at which the External Tank disconnects from the Orbiter.  Remember, the sequence at the end of the powered ascent of the Space Shuttle is that the engines shutdown and then, a few seconds later, the External Tank is dropped off from the Orbiter.  For our purposes here, you can think of the Disconnect as one of three “on/off” kinds of valves in the system with the other two being the prevalves and the main fuel valves on the engines.

From the schematic you can see that the “17-Inch Manifold” (cleverly called that because, well, it’s a pipe that’s seventeen inches in diameter) is in the Orbiter.  As I said, it’s not even part of the engine.  So, why would we get an unusual pressure rise in that manifold after engine shutdown and does it have anything to do with the engines?

The first thing that we did was go back to other launches, both on Atlantis and on other the other Orbiters since they were all configured the same way (though each had slight quirks and differences).  Frankly, we wanted to show, “Well, we’ve seen this before.  It’s no big deal.”  That way we could get back to patting ourselves on the back about our great success.  But what we found was that this pressure rise was not normal.  If we went far enough back in the flight history, we found examples of the manifold pressure rising, but these early launches used a different sequence of shutting down the engines and dropping the tank.  So, more and more, this rise in pressure was looking to be unique and unexplained.  Please realize that this is taking place over a couple of days.  It started slowly, as a “get back to us” action in the data review.  But it really gained momentum with the next series of revelations.

It turns out that the manifold has a relief valve that “cracks” open at anywhere from 40 to 55 psi.  Remember the pot of boiling water and the tight lid?  Well, everywhere in a cryogenic system that you can trap fluid – like, for example, between valves – there has to be a relief valve.  Otherwise, because cryogenic liquid has such a propensity to boil, you have the possibility of, well, “popping the lid.”

So, on STS-104, we had a pressure rise in the 17-Inch LH2 Manifold that could have opened the relief valve.  It turns out that it didn’t (based on other data), but it could have.  It also turns out that the relief valve is actually single-string, meaning that there is no redundant, backup pathway should the relief valve fail to open.  In other words, consistent with the broad redundancy philosophy pertaining to human space flight, we should not rely on the relief valve to guard against over-pressurizing the manifold.  That relief valve ought to be only a tool in the case of an emergency.  What we need to do is avoid unintended pressure rises in the manifold.  Thus, we needed to figure out why the pressure rise happened.  We could not pooh-pooh the situation.

Now, before anyone thinks that we nearly had a catastrophic event on STS-104, please let me assure you that this was not the case.  We just had a 10 to 12 psi pressure rise in the manifold.  The structural limit of that pipe allowed for another fifteen or twenty psi of pressure before any analytical limits would have been reached and there’s additional margin designed in over that. However, also in keeping with the philosophy and thought processes for human spaceflight, we can’t just look at the 10 psi increase and believe that this represents the worst case.  If you don’t know why it happened this time, then you certainly don’t know that it couldn’t be worse next time.

What does this all mean?  Well, it means that we started with “Huh, what’s that?” and, over the course of a few days, got to the point of formally declaring the situation an In-Flight Anomaly (IFA) and a possible impediment to future flights.  Let me tell you, there is not much that is more motivating than knowing that your subsystem could be what holds up the next launch simply because you can’t figure out what’s going on.  Oh my, things got busy.

Here was another piece of data that jumped out as odd.  Note that we saw this right away during the data review, but we didn’t think too much of it at the time because, by itself, it didn’t seem to endanger anything.  In the plot below, I’ve shown again the pressure in the 17-Inch Manifold after engine shutdown and along with that I’ve plotted the engine inlet pressures.  These measurements are below the prevalves in the schematic shown up above.

 The thing that was obvious was that the inlet pressure related to the Block 2 engine was higher than the two inlet pressures for the non-Block 2 engines.  It wasn’t until the whole “oh-my-goodness” momentum of realizing the manifold pressure rise was important did we go back, plot up every flight ever flown, and discover that not only was this inlet pressure high, it was the highest inlet pressure ever seen.  Now things were getting interesting.

 Let’s pause for a moment and talk about the process shown in the plot above.  Remember, the time marked as zero on this plot is the point at which the engines are commanded to shut down.  The main fuel valves closing past five seconds marks the end of liquid hydrogen flow through the engines.  Next, the prevalves on the Orbiter are closed.  That traps liquid hydrogen between the main fuel valves and the prevalves.  Trapped fluid with any energy input is like the boiling pot of water and so the pressure rises in that volume as you can see on the plot.  A pressure rise to some degree is expected and normal.  Next, the disconnect valve is closed between the Orbiter and the External Tank.  Here again, you trap a volume of fluid.  But usually, except for STS-104, there isn’t much if any pressure rise in this volume.  Why?  In part, it’s because it’s a much larger volume with, therefore, more compliance.  But also because there’s less energy input (that’s a key point to put away and remember for the next article).  Once the disconnect valve is closed, that allows for the tank to be separated from the vehicle.  Then, after the tank is safely away, dump valves and bleed valves are opened to start draining the system.  Here you see the pressures drop because the fluids are allowed to leak out in a controlled manner.  Thus, the whole thing is a process of shutting down flow, systematically locking up two volumes of fluid, letting go of the tank, and then releasing the pressure and fluid in the trapped volumes as part of the initial preparation for having the Shuttle vehicle stay on orbit.

So, this was where we were in terms of what we knew and the data that we had a few days after the launch of STS-104.  Our task was to understand and reconstruct why the In-Flight Anomaly happened.  Let’s review what we know at this point just as if we were working through the STS-104 investigation together:

• After engine shutdown, the pressure rose in the 17-Inch LH2 Manifold and we figured out that such an occurrence, while not terrible by itself, was indeed something that we should avoid
• During roughly the same time period, the pressure in the lower branch of the LH2 feed system, the section below the prevalve feeding the new Block 2 engine, also rose and did so to a level higher than for any previous Shuttle flight
• This was the first flight of the new Block 2 configuration which featured a new high-pressure fuel turbopump design
• As discussed in the previous article, the Block 2 high-pressure fuel turbopump was beefier and more robust than the previous version

In addition to this, I will give you one more key piece of information (something that I learned as part of the original investigation): The prevalves are designed such that they have their own internal relief system.  If the pressure in the volume trapped between the prevalve and the main fuel valve of the engine reaches anywhere from 35 to 45 psi higher than the pressure in the 17-Inch LH2 Manifold, then the prevalve allows leakage “backwards” across the valve.  The differential pressure where this relief function kicked in was different for each prevalve, but it was an intentional design feature that influenced the STS-104 anomaly.

So, that’s it for now.  In the first article, I gave you some background.  In this article I told you about the observed data and how it became to be understood that this was an issue rather than just an oddity.  In the next and final article on this subject, I’ll tell you about what determined was the actual scenario, the models that we developed to reconstruct the anomaly, and how we fixed the problem such that we could continue to fly the Shuttle for almost another decade.

I leave you with two questions to contemplate until you hear from me again: