I am starting the writing of this article on the first day of the month so, “rabbit, rabbit, rabbit.” There, now that’s done with (a silly, harmless old-world superstition shared with me by my mother – look it up!).
Meanwhile back at the ranch, Auntie Jane fell down a well and Jake the hound dog led the sheriff to Granddad’s moonshine still in the barn…
Okay, okay, I’ll stop stalling. This is the third and final article about the in-flight anomaly on STS-104. In the first article, we talked about how everything apparently went so well for the first launch of the Block 2 configuration of the Space Shuttle Main Engine. This was the culmination of over a decade of incremental work to transform the SSME into a safer, more reliable engine. In the second article, we talked about the “uh-oh” moment when we found pressure rises that, at first, just seemed a little unusual and then, upon further research, were found to be extreme as compared to what we had seen before in the flight program. In this article, we will discuss why this anomaly happened, why we missed predicting that it would happen, and what we did to keep flying safely beyond that point.
Why it happened, part 1: the rotor
Did you ever notice how much coordinated effort it takes to slow down an airplane as you’re landing? They drop the landing gear to increase drag. They slow the engines and drop flaps and tip the nose up, getting more lift but at the price of additional drag. And when you’re finally on the ground, they throw up more flaps and sometimes use loud thrust reversers and, lastly, they use the brakes on the wheels. All this effort is necessary because you’ve got this great big thing with lots and lots of energy and momentum and you’ve got to bring it to rest. That’s a great deal of energy to dissipate.
Now, think about the rotor in the SSME high-pressure fuel pump. After engine shutdown, it’s spinning down, decelerating, from over 30,000 rpm to zero in just a few seconds. Just like the airplane in a relative sense, that’s a great deal of energy to dissipate in a very short time. So, where does the energy go? Remember, short of Einstein’s relativistic effects (not relevant here), energy is neither created nor destroyed. It is only transferred. When we say that was dissipate the energy from the rotor, what we really mean is that the energy comes out of the rotor and into, well, um … what?
Some of the dissipation is due to mechanical friction. But we’ve got really, really good bearings in that turbopump, and there aren’t any brakes (i.e., the energy dissipation tools used for your car), so friction is a very small piece of the process.
The only other thing that you have is the fluid in the pump, the residual liquid hydrogen left there after shutdown. Think again about the plane landing. Many of the things done to slow the plane rely on drag, which is basically relaying on putting energy into the working fluid, i.e., air. We do kind of the same thing with our working fluid, the residual hydrogen. That’s why we close the valves the way that we do and effectively lock up the fluid in the line rather than just let it all drain out of the pump portion of the turbopump. Because the turbine end of the turbopump is no longer being powered and because the rotor is continuously transferring energy to the fluid, the result is that the rotor slows down. Ta-da! The plane has landed and the rotor is slowed.
But what happens when you put energy into a fluid? In the case of the landing plane, the reservoir of Earth’s atmosphere is so huge that there’s basically no effect. But in the case of the pump, it’s more like the boiling, covered pot on the stove discussed in the previous article. It is a fixed, trapped volume into which you are putting energy. Thus, the pressure rises.
“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.” Well, sort of. We always expect a pressure rise. That’s part of the process. If you go back to the previous article, you’ll see that there was a pressure rise for all three engines. It’s just that the pressure rise for the Block 2 configuration engine was so much greater.
Remember back to the first article when I was explaining that the Block 2 fuel turbopump was safer, in part, because it was heavier? We were able to allocate more weight to the designers and so they used that extra weight (along with many lessons learned) to increase the safety margins. A heavier rotor means that when it’s spinning, it has more energy than a lighter rotor spinning at the same speed. Thus, a heavier rotor should mean more energy dissipation/transfer and therefore more pressure rise.
“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.” Again, not quite. Note that this was practically the same thought path that we followed as we unraveled the STS-104 anomaly. By this time, we were deep into our analytical modeling efforts. We had models for the engine transients – in other words start up and shutdown – but we had not predicted this effect to any great detail. And even when we carefully accounted for the greater mass of the Block 2 rotor, we could not entirely recreate the STS-104 anomaly. Yes, we did indeed get higher pressures in the line, but not as high as we saw in flight.
Why it happened, part 2: “thermal mass”
Here’s an experiment. Heat your oven to 350 degrees. Put potato and a radish, each wrapped in aluminum foil, on a shelf in the hot oven. An hour later, take them out. Now, the experiment part is drop them into ice water and observe how much time it takes before you can comfortably pick up and hold each foil-wrapped vegetable with your bare hands. While I cannot say [confession coming] that I’ve actually done this experiment in preparation for this article, I am quite confident that the radish would cool more quickly. Why? Well, there are all kinds of heat transfer equations related to conduction and convection that we could review, but that’s not really necessary. It’s just common sense. A hot potato stays a hot potato because it’s a heavy, dense thing. It’s got what you could call “thermal mass.” It stores a lot of energy when hot. Something less weighty like, say, a radish, has less mass and so even if it starts at the same temperature, there just isn’t as much energy to dissipate to bring it back down to temperatures that allow for handling.
Now, the high-pressure fuel turbopump is not a potato, but it does have significant thermal mass. It’s about the size of an automobile V-8 engine. Put your hand on the hood of your pick-up truck an hour after you’ve parked and you’ll get a sense of thermal mass. And just as with the rotor, when we went to the Block 2 configuration, we were able to provide for a larger weight allocation for everything as part of the effort to make a safer component. So, in addition to the rotor, the housing and all of the internal flow-path elements of the pump were a bit more meaty than previous designs.
But how does this relate to the STS-104 anomaly? That’s really a good question because it’s not really obvious that it should. If the pump was full of liquid hydrogen for all that time prior to launch and during ascent, then the pump ought to be the same temperature as the liquid hydrogen. Just like water flows downhill, heat only flows when there’s a difference in temperature. If the metal is liquid hydrogen temperature and the liquid hydrogen is liquid hydrogen temperature, then you ought to have a thermal standoff with no heat transfer. Ahhh, but here is where you have to think in terms of other worlds with respect to the notion of “liquid hydrogen temperature.”
Think about water boiling on the stove. How hot is it? It’s just about 212 degrees. If you turn down the burner so that it’s boiling less, it will still be at 212 degrees. If you turn up the burner so that it’s boiling violently, it will still be at 212 degrees. At regular, atmospheric pressure you cannot make water hotter than 212 degrees. If you add any heat to water sitting at 212 degrees, that extra heat will be released by boiling but the temperature remains unchanged. However, if rather than our normal 14.7 pounds per square inch we increase our pressure up to 1,000 pounds per square inch, then we can heat the water up to over 500 degrees before boiling starts. In other words, the boiling point is dependent upon the pressure. I mentioned this in a previous article in this series.
Prior to launch, we chill down the pump to liquid hydrogen temperatures at near atmospheric pressure conditions. That’s means about 37 to 39 degrees above absolute zero (or more than 420 degrees below zero Fahrenheit). While the engine is running, however, because we’re adding lots and lots of work into the fluid as part of the pumping up to over 5,000 pounds per square inch of pressure, the temperature of the fluid coming out of the pump is somewhere between 90 and 100 degrees above absolute zero. Well, if the fluid is up around 90 to 100 degrees, then the metal of the pump is going to be up around that temperature as well after eight and half minutes of power ascent. So what happens when we shut down? The spinning pump rotor slows, comes to a stop, and the pressures through the whole system drop.
Think about that water sitting at 500 degrees Fahrenheit and at 1,000 pounds per square inch pressure. Now imagine that you gradually decrease the pressure back down to normal atmospheric pressure. What happens? The water is going to boil. It is going to release energy as it cools down to the normal boiling point of 212 degrees at 14.7 pounds per square inch. This same phenomenon happens in the hydrogen side of the engine during shutdown. The pressure drops during shutdown and this warmer liquid needs to expel energy to get down to lower boiling point temperatures corresponding to lower pressures. And, as the liquid temperature drops, then there is a temperature difference between liquid and the metal of the pump. And what happens when you have a temperature difference? That’s right. You get heat transfer.
All of this is the normal process. It’s all expected. So, what made the STS-104 experience different with the Block 2 engine? Answer: The hot potato, i.e., thermal mass. As mentioned, the Block 2 high pressure turbopump was intentionally meatier than its predecessor. This additional meatiness helped with the reliability and safety of the unit but it also added more thermal mass and this meant more energy being put back into the fluid. More energy into the fluid translates to higher pressures in the trapped hydrogen between the main fuel valve and the prevalve.
“Eureka!” you say. “We have identified the source of the STS-104 pressure rise.” Okay, yes, in combination we have. Simply put, it was the combination of greater kinetic energy from the heavier rotor plus the additional heat due to the greater thermal mass of the pump structure that led to the elevated pressure between the main fuel valve and the prevalve. And this elevated pressure, in turn, due to the release mechanism built into the prevalve, is what caused the elevated pressure in the 17-inch manifold. We modeled all this analytically, both at a higher level for the purposes of running through many “what-if” scenarios and also at a very detailed, multi-node level embedded within the accepted transient model for the engines.
“Why didn’t you see this coming?”
So, in gracious and humble gratitude for solving the riddle of the STS-104 in-flight anomaly, the management community for the Space Shuttle demanded an answer to the following question: “Why didn’t you see this coming?” And, sometimes, this question was asked with slightly more colorful language. After all, the Block 2 engine configuration had been in development for over a decade, so it really was a perfectly fair question. And, with the perfect hindsight, perhaps it was something that we could have or should have predicted. Fine, I’ll accept that. But here is the biggest part of the reason why we didn’t really think of it: We never saw any difference on the test stand.
When you set up a test stand to test an engine, there are many competing factors to take into consideration. Obviously, you want to protect the very expensive engine hardware so you include a multitude of safety provisions. You also want to get useful data from the tests (remember: you only test engines for two reasons – to impress your friends and to get data). But how do you define useful data? Most people would say that useful data is defined as data from testing that most looks like and feels like the actual mission. But you can’t make the test stand look exactly or entirely like the vehicle and, even if you did, it’s not like you can make the test stand fly through the upper atmosphere at Mach 10 during the test. In other words, the ground test stand will never be a perfect reproduction of the flight mission. So you make compromises based upon practicality, cost, and safety.
On the SSME test stand, the prevalves are higher up in the feed line as compared to the Shuttle orbiter. This means that the trapped volume between the main fuel valve and the prevalve is much larger. Also, in the shutdown sequence, the test stand prevalve shuts down later than on the vehicle. And, very shortly after the prevalve shuts down, we bleed the line since the prevalve is not a fancy flight unit with a built-in relief functionality. All these things combine to provide a very safe, steady shutdown for the engines in the test stand. Pressure is maintained, but not allowed to rise very much and we have a larger volume of liquid to keep the pump loaded and to absorb any energy input. It wasn’t just for the Block 2 configuration that the test stand was different from the vehicle. It had always been different, as in for nearly 30 years. Thus, for nearly 30 years the test stand was different from the vehicle and the shutdown data between tests and flight looked different, but the differences were understood, accepted, and, because there had never been an issue, largely ignored. The Block 2 tests looked just like all the previous configurations on the test stand with regards to the shutdown pressures and so a flag was never raised. It was only when we got to the different geometry and procedures of flight that an issue appeared.
Okay, now what?
We now understood the issue and we understood why we hadn’t predicted that the issue would come up. We now had to resolve the issue. We had a number of possible choices, but the most obvious came from the fact that we hadn’t predicted the issue based upon test results. So we figured that if we made the vehicle act a little more like the test stand, then we could make the anomaly go away. We couldn’t move the prevalves or make the feed lines any longer, so we instead recommended delaying the prevalve closure by 2 seconds. In terms of physics and thermodynamics, what is this doing is allowing more of the energy being released from the spin down of the turbopump to leak upwards, out of the engine, and even back into the External Tank before we lock up the system. Less trapped energy, less pressure rise, but we still spun down the turbopump safely as shown repeatedly on the test stand. According to our analytical models, this worked well for the engine and the feed lines.
However, the prevalve closure is just the first step in a series of actions taken by the Shuttle in preparation to separate from the External Tank. So, if we recommend that the prevalve close 2 seconds later, then the whole sequence has to change by 2 seconds including the disconnection of the External Tank. That required an assessment from our trajectory analysis friends at the NASA Johnson Space Center. They had to see if mission performance was impacted or if the External Tank reentry footprint (i.e., the place in the Indian or Pacific Oceans where the External Tank comes crashing down into the sea) was altered in an unacceptable manner.
The trajectory analysis results showed that the changes were acceptable and so, on 5 December 2001, less than five months after STS-104, we launched STS-108, Shuttle Endeavour, with one Block 2 configuration engine. The results were exactly as predicted. There were no unacceptable pressure rises. In April 2002, on STS-110, we successfully launched a complete cluster of three Block 2 configuration engines, on Shuttle Atlantis, and had no post-shutdown pressure issues. The STS-104 in-flight anomaly was officially considered to be resolved.
No, the story of the STS-104 in-flight anomaly will never be made into a movie starring Tom Hanks a la “Apollo 13,” and certainly the stakes that we faced were never that dramatic in finding a resolution (thank goodness). Nevertheless, it was an excellent example of an investigation in which we used the available data, constructed suitable analytical models, and found a solution to an engineering problem in a manner such that the ongoing sequence of Shuttle launches was never really impacted. It was an interesting problem and a complete success and I am still, to this day, proud that I had the opportunity to play a role. [Note, however, that if there are any movie producers reading this and it they want to start casting the parts, Brad Pitt would be an excellent choice to play me.]
8 thoughts on “LEO Extra: Coming to a Resolution / STS-104 Part 3”
Awesome! The amount of engineering that goes into the creation of a space vehicle is . . . above my head. However, it is interesting how such an anomaly can be solved by 2 seconds in time. Thank you for this explanation. A humble view of true science is truly enlightening!
Very interesting article
Very interesting. Having studied thermodynamics in my chemical engineering classes it all makes sense and seems so simple though I’m sure the idea to try a 2 second delay was thought at first to be a hairbrain idea. then to see results would be from mathematical models is never easy either, let alone convincing top managers to try the idea and that it won’t blow up every thing is no easy job either. That’s why we leave all this to “rocket scientists” to figure every thing out!
very simple and very concise, even with all the technicalities… great read, great read…
really precise but really detailed
Is it anywhere close to feasible, currently, to engineer frictionless magnetic-bearing turbopumps? If yes, then using them would allow you to keep the rotors spinning throughout the orbital phase of the mission. They’d function as CMGs for orbital attitude adjustment. You could even mount them such that the three axes of rotation for the three turbopump assemblies in the three engines are all orthogonal.
This’d offer a fair number of benefits – you’d save on RCS propellant, and, attitude adjustment can PRODUCE power (harnessing rotational energy for electrical power: spin-down torquing, as opposed to spin-UP torquing). Plus, you’d have components with larger MTBF – given zero friction.
Trade-off, of course, is the cost and the expense in developing this, and all the complications that the detailing of such a system will bring out. Plus, a business case. Well, the latter’s not too bad. I know the shuttle’s have been retired, but people are still pursuing that ‘holy grail’ of cryogenic/liquid SSTOs.
@AJA: It took a little while, but I found someone who could respond to your questions and input. The following is from an expert in the area of turbomachinery here at NASA MSFC named Matt Marsh: “Magnetic bearings have been used by NASA primarily in ground test rigs where the rotor is supported by basically frictionless bearings. They have the capability to not only control the position of the rotor both axially and radially, but also move the rotor rapidly and in a specified orbit. These bearing have been used to study the capability of fluid film bearings. However, the magnetic bearings require a large power source. The power source could be reduced if superconducting magnets were used and cooled in a liquid hydrogen turbopump. Probably the best application for magnetic bearings would be in a liquid hydrogen turbopump designed for a Nuclear Thermal Rocket Engine used as the main propulsion system needed for transportation to Mars. If a Bimodal Nuclear Thermal Rocket Engine was used, it could generate electricity and the thrust for the Mars mission. Now, if the turbopumps were connected to an alternator/ generator / motor the pumps could be used as a free spinning CMG, or better yet as a large fly wheel to store the energy. Many of the benefits you mentioned could be used to help reduce the mission system propellant needs and help extend the life of the bearings for turbopumps. The technologies are available and would need to be integrated into the electrical and control systems. The orientation of the turbopumps in three orthogonal axis may give the Engine systems team a little heartache, but not impossible. Thanks for the suggestion.”
Thanks for the answer Matt, and for pursuing it Bill. Much appreciated 🙂
It’s great that the technology’s already been tested.
I looked up a list of high-temperature superconductors with their associated transition temperatures; the operating pressure of the Shuttle’s ET (Wikipedia), and increase in boiling point of Oxygen with increase in pressure (http://www.uk.airliquide.com/en/products-and-services/bulk-gases/bulk-liquid-oxygen.html). 22 psig operating pressure of the ET LOX tank = ~2.6 bar absolute => the BP of LOX at tank pressure is <<130.15 K (LOX BP at 20 bar).
This gives us a decent number of options as far as superconductor materials are concerned – to implement magnetic bearings in the LOX lines too… even if we just chill it down to the bulk temperatures of the LOX tank.