I believe it was General Norman Schwartzkopf who said: “Arm chair generals study tactics; real generals study logistics”.
One of the first lessons I learned in program and project management is that attention to the details of supplies, vendors, and parts manufacturers will determine success or failure more than anything else that management does. They are not glamorous, Hollywood does not make movies made about them, but logistics and supply chain are the unsung pillars on which every major project rests.
It is nice to have eloquent oratory and high flown philosophical statements, but the real way that real programs are really controlled is through the money. When the logistics and supply budget is stopped, the program is over. Momentum and warehoused supplies can carry on for a short period, but when those are exhausted, its time for the museum.
Starting four years ago, the shuttle program in its various projects made “lifetime buys”. That is, we bought enough piece parts to fly all the flights on the manifest plus a prudent margin of reserves. Then we started sending out termination letters. About two years ago, we terminated 95% of the vendors for parts for the external tank project, for example. Smaller, but still significant, percentages of vendors for SSME, Orbiter, and RSRB have also been terminated.
A lot of things that go into the shuttle build up are specialty items. Electronics parts that nobody makes any more (1970’s vintage stuff). Hey, if it works, why invest money in certifying new parts? Certifying new ones would be even more costly! Specialty alloys to meet the extraordinary demands of space flight, parts that are made by Mom and Pop shops mostly in the LA basin are norm rather than the exception. You might think that simple things like bolts and screws, wire, filters, and gaskets could be bought off the shelf some where, but that thinking would merely prove how little you know about the shuttle. The huge majority of supplies, consumable items, maintenance items, they are all specially made with unique and stringent processes and standards.
Our shuttle history tells us that when we try to cut corners, trouble results. Small, even apparently insignificant changes have caused big problems. For example, the unheralded end of production of a solvent caused enormous complications for the SRB folks a few years back when things started falling apart unexpectedly. It took a huge engineering detective effort to determine that small chemical changes in the new solvent were the culprit. Anything coming apart in the SRB is not good. There are hundreds of similar examples.
There is a long and arduous process to certify a vendor to produce the logistical parts for the shuttle. Not many companies do this work. Almost all of them are extraordinarily proud of the role they play in America’s space program. A lot of them have been there from the beginnings in the middle 1970s. So when a Mom and Pop specialty shop gets a termination letter from the shuttle program after 35 years of production and they have other customers, guess what happens? Mom and Pop decide to close the shop, pension off their highly skilled workers, and then Mom and Pop move out of LA to their retirement cottage in the mountains or at the sea shore.
A lot of this has been happening over the last four years; most of it over two years ago.
So, just for the sake of argument, lets see what would happen if somehow we decided to fly the shuttle some more flights?
From time to time a vendor of specialty parts for the shuttle has gone out of business. Our experience then is that we have immense problems getting anybody to even bid on making replacement items. Sometimes, with hat in hand, we have to knock on doors. Always, we have to offer premium payments to get those exotic, small production run parts made.
Given time and money, anything is possible. But we are always short on time and money. Life seems to be like that.
To take one little example: if we started today to build another external tank at MAF, there are probably enough parts on the shelf. But very shortly we would exhaust supplies of some parts. Maybe on the second tank — which we need to start in 3 months or so — would have to get a new supply of specialty parts. Sometimes the old vendor is still there and could be persuaded to make more of the old parts. But in many cases, a new vendor would have to be found. Since the production run would be small, a premium price would have to be paid; and a certification effort requiring 6 to 12 months would start. Initial production likely would have a number of rejects as the workers learn the process. Hmm. In probably 15 to 18 months would would have the parts to build that second tank — only a year or so later than we needed them. So a new gap would form. Not between shuttle and orion but between shuttle and shuttle.
And what would we get: even higher price per flight of an old technology which is not nearly as safe as we would like . . .
Hey, I am the biggest shuttle hugger there is. I think it is the best spacecraft ever built. But I also deal in the real world.
Where does the money come from? Where do the people — who should be working on the moon rocket — where do they come from?
We started shutting down the shuttle four years ago. That horse has left the barn.
Hi Wayne,
Are not the individual orbiters as wholes due for airframe recertification actions/activities around 2010 as well?
Unfortnately, I believe this country will pay dearly for the action of shutting down the shuttle 4 years ago in many-many ways over the long run.
Playing some hypotheticals for a moment, what happens if the situation in Georgia (or even, heaven forbid, the Ukraine) worsens and Congress decides (rightly, IMHO) not to exempt NASA from INA and all of a sudden no one can get to Station except the Russians? Money can only accelerate Constellation so much, and no amount of money, IMHO, can close the gap. Do you have any proposed ideas for how to solve that problem?
My personal suggestion is to accelerate COTS-D, but I’d rather hear yours…
Insightful, thanks for this clarification.
Given the recurring problems with Ares development, some questions should be asked, at least for historical perspective: In your opinion, would have re-certifying the shuttle been a wiser move with regard to shortening the gap?
If not re-certifying the Orbiter, then why not many of the existing Shuttle based concepts (dare I say derived) hardware such as Shuttle-C?
Seems such a shame to be in the position we’re in.
I love reading your blog. It is informative. It is brief, yet thorough.
Great insite. I love Wayne Hale. I love watching the mission status briefings when he is there. You can tell by his (sometimes long) answers to the press that he loves the program and his decisions are well taught out. If I got my geek wish to talk to someone at NASA it would be him.
So give NASA a lot more money to continue the shuttle or deal with the multi year gap.
What I see here is history repeating itself- there was a stunning lack of continuity between Apollo and STS, and now we’ll have the exact same result between STS and whatever comes next.
Like it or not, the stark reality is that any space endeavor does not lend itself to lean manufacturing and “just-in-time” inventory.
Four years ago, a decision to move to the next generation was made; world events since then have cast that decision in a questionable light.
But it all ultimately boils down to a question of money, and from what I’ve read, Congress never gave NASA the $2B promised in the wake of STS-107 and Katrina.
A friend recently asked if I knew the difference between prudence and politics. I replied that prudence comes with money to make it happen.
Politics is the birthplace of empty promises. Empty promises is something Congress excels at, so don’t expect enough money to smooth the transition…or to pay for orbiter recertification.
And just like Apollo, all of the specialized knowledge and machine tooling winds up on the scrap heap.
The facts keep getting stated over & over, yet the internet has a very short memory. In a few weeks this blog post won’t show up any more, anywhere, & we’ll be back to reading shuttle termination rants.
Very refreshing blog – just stumbled upon this today. Thanks for sharing your perspective. I think NASA, and Dr. Griffin in particular, made a valiant effort to shout from the rooftops many years ago, that “Hey, we are going to put an end-date on this thing! This is for real! Here are the reasons…” But of course when the time comes, some people act surprised or even shocked. I wish Congress could have been a better partner with NASA to enable the organization to actually be able to do what was asked of it. I will admit to being a critic of the current VSE CLV/CaLV implementation, however in this case NASA has done the right thing with respect to closing down the Shuttle after a prudent number of ISS construction/logistic flights.
How NASA management has handled the gap is another matter, however, but that is a topic for another day…
So the Shuttle program can’t be extended, not without at least some gap “between shuttle and shuttle” and a lot of money.
Given the current geopolitical situation we can’t, or at least shouldn’t, rely on the Russians to ferry US astronauts to and from ISS during the gap between shuttle and Orion.
So what are we to do? Sit back and watch other nations launch their citizens into orbit while The United States of America has no means to do so? That to is unacceptable.
There is an alternative. The SpaceX Dragon and Falcon 9. No, they’ve not flown yet. But they’ve got a good design and a plan. All they need is a little more money, much less than what extending shuttle would cost.
So why doesn’t (and I know you don’t decide these things Mr Hale, but you have the ear of those who do) NASA appropriate 100, 200 million to SpaceX to speed the development of Dragon and Falcon? Reduce the shuttle to Orion gap to 1 year, 6 months or eliminate it all together.
It is the only alternative that makes sense, both from a practical/financial standpoint and from a national pride standpoint.
An excellent discussion, of the issues in restarting Shuttle.
There is a corrolary in that a number of workers have also
left the STS Program. The people are at least as important
as the parts. As the program winds out, younger folks
stop training and begin looking for new work. Mid-career
folks begin transitioning into new programs and
senior people are held on to try and keep the program
going until the end.
Meanwhile the support contractors start spooling down and
the whole mess loses institutional memory and resilience.
Now perhaps some of these problems would be a bit less intense
if the STS had at the beginning used common infrastructure
from other platforms such as the F-16 computers or standard
aerospace bolts, etc… Also if on an ongoing basis,
the key parts were upgraded, so the bird is less of a
flying museum. That would have required managers in
the 1980’s and 1990’s to invest into sustainment.
That isn’t practical at this point.
“And that comment about professionals talking logistics”
goes back at least to the First WOrld War.
Mr. Hale:
Thank you for the nice comments on logistics, especially since I work in Shuttle logistics! I agree it would be too expensive to retain the current vendor base, but what about replacing obsolete parts with new technology through Orbiter upgrades? For example, replacing the APU/Hydraulic and pneumatic systems with electromechanical actuators? EMA technology has come a long way since the shuttle was designed, and even being considered for the Ares 1 thrust vector control. LI-900/RCG tiles could be replaced with AETB-8/TUFI for a hardened TPS. There are so many other upgrades that be incorporated into the Orbiters (integrated modular avionics, PEM fuel cells, SSME channel wall nozzle, non-toxic OMS/RCS system, etc.) to make the shuttle safer, supportable and less costly than the current configuration. The shuttle would also have 120 flights of experience and two tragedies of lessons learned behind it.
Any new parts, whether for Orion or Shuttle will require same long certification process, and since the lot size will be low for either program, the cost will be the same. There is also a very good chance Constellation will be delayed beyond the official 2015 initial operating capability.
In a perfect world, I wish we could continue to improve and operate the shuttle, and develop the Orion capsule, since a capsule can play a role in missions that require a spacecraft to remain in orbit for an extended duration. One final thought, if we take a real world view, what are our actual chances of returning to the moon and going to Mars, considering the state of world we live in today?
I truly hate to say this, but going back to the Moon and an expedition to Mars is not important to Americans right now with everything else that is going on. If it is not important to the American people, it is not important to Congress. Without the support of Congress, the President will not get the space exploration program that he wants, which means NASA doesn’t get to go to the Moon and Mars. In addition, since we have let the the space shuttle supplier base go, the space transporation system we did have ends up in museums. I am a spaceflight-hugger, and I do not want to see this happen, but I fear it is our destiny.
Hello Wayne: I’ve been enjoying your blog and support what you are doing. Congress is directing one final flight after STS-133 to deliver the Alpha Magnetic Spectrometer. It would use the Orbiter, ET and SRB’s already prepared for Launch on Need. A crew of three would return by Soyuz, and the Orbiter commanded to reenter tail-first. This avoids the risk and expense of recovering a Shuttle on Earth. This seems to be a safe and inexpensive way to orbit AMS while retiring Shuttle. Given your experience with STS, your opinion would be appreciated.
I worked for a short while in the early 1980’s at one of those LA area Mom and Pop shops. Sorry I can’t recall the name, but they made ‘O’ rings, not the big ones for the SRBs, but smaller ones for other applications. To this day I am immensely honored to have played even the smallest part in the supply chain of the program. And like you said, those folks were very proud, too, and were unerring in their dedication to meeting the specifications.
The supply chain is critical to our near term future success for Constellation as well. If we are to fill the gap quickly, we need an Ares launch vehicle design that uses hardware and people based more closer to shuttle or EELV hardware, (ie two 4 seg SRBs, RS-68s, ET, etc). Those horses may have left the barn, but they are closer than the other horses which are not even born yet.
So we need to restart key items for the boosters (just not the orbiter unfortunately, which is the best spacecraft ever made). If there are JOFOC issues, lets just try National Security as simple justification.
Given the supply chain of funds, we can probably only afford one booster development, so if the Ares X is oversized for CEV, so be it. We can come up with a lander that will fit on it. We could also put the LOI burn back on a block II lunar service module where it should be, if Ares X is bigger. Then we will have two launches for lunar missions, rather than 1.5, but it would be the same vehicle. Again supply logistics favors this approach rather than trying to maintain two vehicles.
ps. How are we doing on the 5 seg booster supply chain, is it the same as the 4 seg? How are we doing at the MAF making the Ares I upper stage, is it similar in processes and suppliers to the ET? How is our goal/directive to use shuttle hardware as best as possible going.
Wayne: Another consideration here, albeit somewhat more subtle than logistics, is operability. More than 2/3 of aviation accidents are caused by “pilot error” which is, in turn, typically a product of excessively high workload and lack of situation awareness, often brought on by poor interface design. The current shuttle cockpit is notorious for the operational difficulties it creates for crewmembers, and if aviation is any guide, it is only a matter of time before “crew error” results in a loss of mission (or loss of crew) situation.
As you know, the shuttle operational community attempted to rectify these problems, and build a much more user-friendly cockpit, through a cockpit avionics upgrade project that kicked off in the 1990’s. The crucial point here is that after the Columbia accident, it was decided not to implement the new improved cockpit that emerged from the project, because there weren’t enough flights on the new schedule to justify it.
Fair enough. On the current schedule, the number of remaining flights is probably small enough that we’ll squeak through without a crew error. But if the decision was made to increase that number, and fly beyond 2010, the risk associated with the decision NOT to implement the cockpit upgrade would increase as well.
Just another consideration to throw into the mix!
After reading the current article in the Orlando Sentinel, perhaps now might be a good time for some old-fashioned foot-dragging before the train gets too far down the tracks.
Wayne, you talk about ’70s style electronics? The company I work for specializes in “legacy” electronics and instrumentation. A lot of the stuff we have actually has old “NASA” stickers on it.
Yes, that “horse” has left the barn…but a good whistle can often bring it back home again.
I was a product line mamager for Hagan Ring Balance instruments once, and am quite familiar with getting things made that hadn’t been made in years.
You tell me what you need made, and I’ll find someone to make it for you. I’m not kidding, either.
As I stated earlier, unless Congress steps up to the bar and gives you the necessary funding, nothing’s going to happen.
The Geopolitical events that have transpired in Georgia may determine that this would be the prudent thing for the US to do. Higher premiums to smaller vendors seem like a viable alternative if the only other option the US has is allowing Moscow to dictate ISS personnel transfer scenarios. Would you prefer the Space Station occupants to be Cosmonauts exclusively selected by Moscow?
“Hey, if it works, why invest money in certifying new parts?”
This made me LOL…
“Certifying new ones would be even more costly!”
This made me LOL harder…
Shouldn’t you be asking these questions of the Ares configurations? Is there ANYTHING on Areas that ISN’T new?
Didn’t the Congress direct NASA to re-use as much as the Shuttle infrastructure and workforce as possible?
Real generals must also study politics back home, of course, as any Roman consul could tell us. When you say “a prudent margin of reserves”, that was really a judgement call as to whether there would be a second Bush administration (yes), whether the American public would be much swayed by President Bush’s “vision” (not so much), and whether the White House would continue to pursue a post-Shuttle agenda with any real energy (no).
If Congress were given the easy option of doing nothing, and continuing Shuttle at four missions a year to do nothing but resupply ISS, they would settle for it. What you’ve done is to take that option off the table.
Which is a good idea, I think. But it didn’t just happen – you chose it, and although this is an admirable blog posting, I think it’s just a tiny bit disingenuous. NASA wants to go beyond low orbit, and it finally has the chance to get working on that without Congress zeroing out the budget for it. Logistics are one of the ways you can, with luck, force the next (clearly far more Democrat-leaning) Congress to stick with this line.
The art of administering NASA isn’t getting the politicians to make a decision. It’s making it hard for them to change their minds back again when the budget runs over, the delays stretch out, and the project loses all focus in an outwardly opaque series of design reviews and sub-contracting exercises. (The really hard part, in my view, is going to be getting Congress not to cancel the Ares V.)
I’m going to repeat a comment I made at NASA Watch related in a post on extending the Shuttle to 2015. I should note that from my perspective the way we should look at NASA programs like the Shuttle (or generally crew and cargo transportation to LEO) and ISS (or generally space stations) is that NASA has paved the way, and now that NASA has demonstrated these capabilities, NASA should move on to the next thing, and completely hand over the job to someone else — using “someone else’s” services, and prodding them along into existence if needed. In these cases “someone else” is the U.S. commercial space industry. This is the same kind of approach that’s taken with a new remote sensing or other satellite capability that NASA pioneers. They get handed over to private industry or an operational agency like NOAA. This is the way for NASA to contribute to the nation’s needs and space capabilities and to grow the economy. It’s also the only way NASA is going to be able to afford to go to the “next level” (like humans to the Moon and beyond). What advances would the NASA Earth Science area be able to afford if they had to keep making all of the operational Earth-sensing satellites?
Anyway, here’s my NASA Watch comment:
Extending the Shuttle to 2015 strikes me as an idea with a lot of potential ways of turning into a disaster:
– This would give the Shuttle many more chances to have a third accident.
– It wouldn’t do a whole lot to increase independence from Russian vehicles, since the Soyuz is needed for emergency crew return when the Shuttle isn’t docked.
– The Wayne Hale post demonstrates just how expensive this would be. Think about the current enormous Shuttle costs, and then add CAIB-recommended flight recertification and restarting production for items that have already been cancelled.
– At least now the Shuttle flights are bringing up major pieces of the ISS. What would the Shuttle do through 2015? It would be a great waste to go to that expense to just fly a couple times per year.
– Related to the last point, it would also be a great waste to just fly crew. So … would the Shuttle be used to send lots of supplies and parts to the ISS? It seems like this would compete with, and possibly kill, the commercial COTS ISS cargo efforts, as well as kill any U.S. commercial crew transport to ISS capabilities that might be forming. That would be a true policy disaster – what’s the point of going to all of this effort and expense with Shuttle and ISS and Constellation if commercial suppliers don’t take over eventually? Even going through the exercise of planning a 2015 extension for the Shuttle must have a chilling effect on SpaceX, Orbital, and any COTS-D efforts.
– A Shuttle extension is likely to cost a lot of money, and it’s likely to have to come mainly from other parts of NASA. That would be a disaster if it came from productive parts of NASA, like Science, COTS, etc. If it came from Ares, and forced Constellation to come up with a better plan that serves U.S. interests rather than NASA interests, it might be all for the best. However, it seems like the idea is to continue Constellation while flying the Shuttle, so it seems like a bad scenario is much more likely than a good one.
If any money can be scraped together to help solve the ISS gap problem, it should be invested in some kind of incentives to get U.S. commercial vendors (perhaps using some foreign components, perhaps not) to pitch in their money and capabilities to get crew access to the ISS. The latest model of that is COTS, but generally something along those lines should be done. This would bring the benefits of technical and business competition, open the “gap” solution space to everything available (EELV, SpaceX, Orbital, Shuttle-derived … everything), bring in additional effort in the form of commercial investment, bring in the additional incentive of potential non-NASA business, and bring the added chance of success that comes with more than 1 attempt (assuming there’s more than 1 winner like in COTS-cargo).
Now … if some compromise can be found, where the Shuttle does productive work that helps open LEO to commercial space, like bringing commercial Bigelow modules to the ISS, that might cancel at least one negative part of the plan. One part of the U.S. commercial space industry gets crushed by NASA, but another one starts, and maybe in this probably unrealistic example the Bigelow-type industry eventually becomes a customer for the commercial launchers. However, I doubt that such activity would be part of the Shuttle plan.
Straight talk from a straight shooter. Thanx Wayne.
One question, when you say LA Basin, do you mean LA the city, or LA the State?
I feel that shutting down the shuttle program is going to be sad, and I’v seen orine tests video’s, they were all failers! I saw the parachute tests the parachutes didn’t deploy right, and the emergency cutteres to cut the chute in case of emergency cut them before the fully deployed! Then it smashed to the ground, and left a huge crater. I think the orine program is going bad, and if I grow a little more I won’t be able to fit in the capsule! I’m just 11!
Great insight into the difficulties of keeping ancient technology alive that long. I helped design a leading edge X-Ray machine eight years ago and although it is still the best there is, some redesign is needed. I can’t imagine something of a gazillion times more complexity operating in infinitely more difficult environment still going after 40 years.
But I do disagree that techniques such as “lean manufacturing” cannot help. One of the technologies that has progressed the most in the past half century is quality. A risk-based design and innovatively encouraged thinking outside of the box may get us where we absolutely have to be. The “mom and pop” statements seem to indicate that we have not written down the real recipe. I saw that here on Long Island when concise records of the LEM were only available by accident from a “historian” who had squirreled away a collection of design documentation.
But I do understand what happened in the two generations since we designed and built the Shuttle. Katrina is an example of thousands of political decisions made instead of turning our prototype designs and ventures into dull, reliable, thrilling steps forward. it is estimated that in three years we spent $300 billion — more than half as much as the entire NASA budget in 50 years! And the French Quarter was barely touched and virtually none of the submerged residences have been reoccupied. Instead of spent, possibly “flushed” should be a better verb, while NASA can chart contributions to national and local economies that far exceed its annual budget.
And based on the rhetoric in the political conventions, any interest we have in space exploration is going to be tossed aside as we buy food and shelter for folks who could care less about achieving escape velocity.
Wayne,
Thank you very much for addressing this topic! I am suspect many of us readers have been curious as to what the implications would be were the Shuttle program extended, especially now that 2010 draws near and the voices to continue the program will likely get louder over time.
Mr. Hale,
I appreciate that this is a painful subject for you, you especially. I have always admired and respected your honesty, your dedication to human spaceflight, your practicality and, as much as one can without sitting in your chair, the constant struggle you have against demanding and often conflicting necessities.
I, for one, applaud your willingness to tell us bluntly your view on this subject and I would caution anyone to be very leery about dismissing it too quickly. Few people understand the efforts already made to make the Shuttle flights since Columbia a reality, the sacrifices, the compromises, the tough choices, the budgetary impossibilities. I know that you do.
Thank you, for your honesty and for your efforts in taking the remainder of the Shuttle program as far as it can go.
Very insightful article. In regards to Shuttle avionics and possible pilot’s errors, does Shuttle still land on manual control? Does not it have any landing automatics? The Russian Buran, which flew only four years after the first Shuttle, did it up there and back on full auto and landed in 10 m/s crosswind in dusty Kazakh steppe with only 1 m deviation from the landing field center line. How come we cannot do what Russians did 20 years ago with technology that already was behind ours? If the Shuttle is such a close system that it cannot be upgraded to full auto control, than it indeed has to be scraped.
Hello there Mr. Hale…
Thank you for the very informative article!!!
Like you sir, I’m a shuttle hugger myself (though self proclaimed that is…) but like you said, when logistics and funding fails, STS is dead in the water…
the orbiter is definitely the most complex manned spacecraft to fly… but let’s face it, if nobody’s left to support it, its no use…
I can see that you probably have some heavy feelings while typing this article, being in the program that has been operational for years now…
I just hope that, when NASA decides to make a new spaceplane in the future, they would look back to STS and learn from it…
Yeah it really is a catch 22 with specialty parts versus long term contract agreements. The smaller the pool you have to draw from the less quality product you can expect, but if you don't give them a contract they cant stay in business. End result? Everything gets more expensive because of less competition. It's a real fine line, hopefully things get straightened out.
thanks you very good
see you later
http://eftelya.org
Given the current geopolitical situation we can’t, or at least shouldn’t, rely on the Russians to ferry US astronauts to and from ISS during the gap between shuttle and Orion.
http://eftelya.org