LEO Extra: “Huh, what’s that?” / STS-104 Part 2

To begin the last previous article, we discussed water, steam, and ice.  Here, for this article, I want you to think about a boiling pot of water on the stove.  Now, put a lid on the pot.  What happens?  Usually, the lid rattles around, rapping out an irregular rhythm (in my house, that usually gets the dogs barking), and it lets puffs of steam escape with each little wobble.  But what if we made the lid really tight?  Well, in that case, if we kept applying heat, the pressure in the pot could eventually rise high enough that the lid could pop off with a good bit of energy.  If I wanted to exaggerate, then I would say that the pot and lid combo could explode, but that’s probably being just a bit overly dramatic when talking about a normal lid.

 

Now, rather than water, think about liquid hydrogen.  Yes, I know, LH2 is not something that you encounter in your daily life.  Shoot, I’m in the business of rocket engines and I don’t deal with LH2 much other than on paper (or as mathematically represented in computer models).  But just think of it like any other liquid such that, if it is warmed and starts to boil in a closed container, then the pressure rises.  Something to keep in mind.

Now, let’s get back to STS-104.

I returned from my short vacation in Boston back in 2001 and I got down to work by working with the team analyzing the STS-104 data.  After every flight, we get data from measurements on the engine and the Shuttle orbiter and we effectively reconstruct the engine performance to make sure that everything did what was expected.  Please note that the engine folks are not unique in this respect.  The team responsible for the boosters scours the post-flight data from the boosters (and, on the Shuttle Program, they also examined the recovered booster shells).  The team responsible for communications examines all of the communications data.  The team responsible for the External Tank studies, in detail, the tank data and reconstructs the draining of the tanks as another means for confirming that the engines performed as predicted.  For us engine guys, the initial conclusions were, yes, the engines performed as expected and as predicted.  In particular, the new Block 2 engine performed perfectly.  No obvious anomalies on the engine.  Pop the corks because all was good!

However, during the data review there was one moment of, “Huh, what’s that?”  It was over the data shown in the following plot (this is not the actual plot – I reconstructed it for this article using the original data):

What you have in that plot is data from two Shuttle missions.  STS-098 was a launch of Shuttle Atlantis that took place in February 2001.  STS-104, our launch being reviewed, was the next launch of Atlantis.  So, this is a comparison between two back-to-back missions of the same orbiter and that’s important since the measurement in question is on the orbiter side of the interface rather than on the engine side.  Indeed, it’s not a measurement that usually concerns us much.  This is the pressure in the shared manifold from which the three engines are fed liquid hydrogen.  The time period is after all the engines have shutdown.  The thing that was kind of odd was the pressure rise at right around 11 seconds (please note that the data looks “boxy” for a variety of factors including, primarily, data rate and, secondarily, the fact that we sometimes sacrifice fidelity for robustness depending on relative importance).

But, hey, the pressure rise only about 10 psi.  We’re rocket guys.  We deal with thousands of psi all over the place.  This is a blip.  A nothing.  No big deal.  That was, I have to admit, the initial response.  But, of course, we still have to figure out how or why it happened.  The teams looking at post-flight data are always extremely meticulous since it is often this data that tells you that you’re safe to fly for the next mission.  However, there is still a prioritization of issue in your head.  Our biggest concern for this mission was how the engines, particularly the new Block 2 configuration, performed during powered ascent.  Nevertheless, we did come back and respond to the simple question asked, “Huh, what’s that?”

Above is a simplified, cartoon schematic of the liquid hydrogen feed system from the External Tank to the engines.  The “Disconnect” is, exactly as it sounds, the place at which the External Tank disconnects from the Orbiter.  Remember, the sequence at the end of the powered ascent of the Space Shuttle is that the engines shutdown and then, a few seconds later, the External Tank is dropped off from the Orbiter.  For our purposes here, you can think of the Disconnect as one of three “on/off” kinds of valves in the system with the other two being the prevalves and the main fuel valves on the engines.

From the schematic you can see that the “17-Inch Manifold” (cleverly called that because, well, it’s a pipe that’s seventeen inches in diameter) is in the Orbiter.  As I said, it’s not even part of the engine.  So, why would we get an unusual pressure rise in that manifold after engine shutdown and does it have anything to do with the engines?

The first thing that we did was go back to other launches, both on Atlantis and on other the other Orbiters since they were all configured the same way (though each had slight quirks and differences).  Frankly, we wanted to show, “Well, we’ve seen this before.  It’s no big deal.”  That way we could get back to patting ourselves on the back about our great success.  But what we found was that this pressure rise was not normal.  If we went far enough back in the flight history, we found examples of the manifold pressure rising, but these early launches used a different sequence of shutting down the engines and dropping the tank.  So, more and more, this rise in pressure was looking to be unique and unexplained.  Please realize that this is taking place over a couple of days.  It started slowly, as a “get back to us” action in the data review.  But it really gained momentum with the next series of revelations.

It turns out that the manifold has a relief valve that “cracks” open at anywhere from 40 to 55 psi.  Remember the pot of boiling water and the tight lid?  Well, everywhere in a cryogenic system that you can trap fluid – like, for example, between valves – there has to be a relief valve.  Otherwise, because cryogenic liquid has such a propensity to boil, you have the possibility of, well, “popping the lid.”

So, on STS-104, we had a pressure rise in the 17-Inch LH2 Manifold that could have opened the relief valve.  It turns out that it didn’t (based on other data), but it could have.  It also turns out that the relief valve is actually single-string, meaning that there is no redundant, backup pathway should the relief valve fail to open.  In other words, consistent with the broad redundancy philosophy pertaining to human space flight, we should not rely on the relief valve to guard against over-pressurizing the manifold.  That relief valve ought to be only a tool in the case of an emergency.  What we need to do is avoid unintended pressure rises in the manifold.  Thus, we needed to figure out why the pressure rise happened.  We could not pooh-pooh the situation.

Now, before anyone thinks that we nearly had a catastrophic event on STS-104, please let me assure you that this was not the case.  We just had a 10 to 12 psi pressure rise in the manifold.  The structural limit of that pipe allowed for another fifteen or twenty psi of pressure before any analytical limits would have been reached and there’s additional margin designed in over that. However, also in keeping with the philosophy and thought processes for human spaceflight, we can’t just look at the 10 psi increase and believe that this represents the worst case.  If you don’t know why it happened this time, then you certainly don’t know that it couldn’t be worse next time.

What does this all mean?  Well, it means that we started with “Huh, what’s that?” and, over the course of a few days, got to the point of formally declaring the situation an In-Flight Anomaly (IFA) and a possible impediment to future flights.  Let me tell you, there is not much that is more motivating than knowing that your subsystem could be what holds up the next launch simply because you can’t figure out what’s going on.  Oh my, things got busy.

Here was another piece of data that jumped out as odd.  Note that we saw this right away during the data review, but we didn’t think too much of it at the time because, by itself, it didn’t seem to endanger anything.  In the plot below, I’ve shown again the pressure in the 17-Inch Manifold after engine shutdown and along with that I’ve plotted the engine inlet pressures.  These measurements are below the prevalves in the schematic shown up above.

 The thing that was obvious was that the inlet pressure related to the Block 2 engine was higher than the two inlet pressures for the non-Block 2 engines.  It wasn’t until the whole “oh-my-goodness” momentum of realizing the manifold pressure rise was important did we go back, plot up every flight ever flown, and discover that not only was this inlet pressure high, it was the highest inlet pressure ever seen.  Now things were getting interesting.

 Let’s pause for a moment and talk about the process shown in the plot above.  Remember, the time marked as zero on this plot is the point at which the engines are commanded to shut down.  The main fuel valves closing past five seconds marks the end of liquid hydrogen flow through the engines.  Next, the prevalves on the Orbiter are closed.  That traps liquid hydrogen between the main fuel valves and the prevalves.  Trapped fluid with any energy input is like the boiling pot of water and so the pressure rises in that volume as you can see on the plot.  A pressure rise to some degree is expected and normal.  Next, the disconnect valve is closed between the Orbiter and the External Tank.  Here again, you trap a volume of fluid.  But usually, except for STS-104, there isn’t much if any pressure rise in this volume.  Why?  In part, it’s because it’s a much larger volume with, therefore, more compliance.  But also because there’s less energy input (that’s a key point to put away and remember for the next article).  Once the disconnect valve is closed, that allows for the tank to be separated from the vehicle.  Then, after the tank is safely away, dump valves and bleed valves are opened to start draining the system.  Here you see the pressures drop because the fluids are allowed to leak out in a controlled manner.  Thus, the whole thing is a process of shutting down flow, systematically locking up two volumes of fluid, letting go of the tank, and then releasing the pressure and fluid in the trapped volumes as part of the initial preparation for having the Shuttle vehicle stay on orbit.

So, this was where we were in terms of what we knew and the data that we had a few days after the launch of STS-104.  Our task was to understand and reconstruct why the In-Flight Anomaly happened.  Let’s review what we know at this point just as if we were working through the STS-104 investigation together:

• After engine shutdown, the pressure rose in the 17-Inch LH2 Manifold and we figured out that such an occurrence, while not terrible by itself, was indeed something that we should avoid
• During roughly the same time period, the pressure in the lower branch of the LH2 feed system, the section below the prevalve feeding the new Block 2 engine, also rose and did so to a level higher than for any previous Shuttle flight
• This was the first flight of the new Block 2 configuration which featured a new high-pressure fuel turbopump design
• As discussed in the previous article, the Block 2 high-pressure fuel turbopump was beefier and more robust than the previous version

In addition to this, I will give you one more key piece of information (something that I learned as part of the original investigation): The prevalves are designed such that they have their own internal relief system.  If the pressure in the volume trapped between the prevalve and the main fuel valve of the engine reaches anywhere from 35 to 45 psi higher than the pressure in the 17-Inch LH2 Manifold, then the prevalve allows leakage “backwards” across the valve.  The differential pressure where this relief function kicked in was different for each prevalve, but it was an intentional design feature that influenced the STS-104 anomaly.

So, that’s it for now.  In the first article, I gave you some background.  In this article I told you about the observed data and how it became to be understood that this was an issue rather than just an oddity.  In the next and final article on this subject, I’ll tell you about what determined was the actual scenario, the models that we developed to reconstruct the anomaly, and how we fixed the problem such that we could continue to fly the Shuttle for almost another decade.

I leave you with two questions to contemplate until you hear from me again: