Lack of Discipline = Slaughter of the Innocents

I would love it if we could cost-estimate these projects accurately! However, I feel the reality is that research and development (R&D) are difficult to accurately cost-estimate. If someone has R&D experience in the commercial sector, I'd love to hear whether they have the same opinion or not.

I wonder if someone has tallied up the cost-overrun of Hubble compared to its original projections, especially when you factor in the Shuttle mission to fix its flawed mirror. These days Hubble is often held up as one of NASA's greatest successes, but in the months after Hubble was launched it was ridiculed as one of NASA's biggest failures.

My point is that a project's potential return may be worth the extra cost to complete it. Another project may not. The agency's management must judge a project considering its potential and the consequences of their decision. That includes the impact to other projects. That seems fair enough to me already. I am concerned that a "fair policy", where if a project exceeds X% then it is no more, would lead to an even bigger waste of resources as it would remove judgment from the decision process.

As others have pointed out before, cost-overruns are not just a NASA problem. If there is a systematic issue with how the federal government as a whole manages R&D, I suggest it worth considering how to best identify and solve that. Then figure out how to best apply that fix to NASA.

There is plenty of cost estimating talent in NASA. The problem is arrogant porject managers who ignore the independent cost estimates of IPAO. IPAO estimated this mission to cost 3 times what was budgeted/planned by project management and was promptly told to disappear. Same for JWST, just multiply project management's estimate by pi and you get a more realistic cost. Once again, our illustrious Administrator has put controls in place to keep project management in check, and then proceeded to ignore or denigrate those people tasked to provide a sanity check. Independent cost estimates are like the Agency's governance structure and technical authority, something put in place to create the appearance of trying to do something right and then ignoring or hobbling that independent capability to the point of being ineffective. As Griffin has made very clear on numerous occassions, project management is King and all others serve them, no exceptions. If you are not here to do the bidding of project management, then you are not a team player and there is no place in NASA for you. We cannot get rid of this guy soon enough. He's a loser.

The basic problem of estimating and cost control is that engineers and scientists dominate NASA and government contractors. These individuals regard financial management as "bean counting" and routinely show their contempt for financial management. When Congress cuts their budgets due to schedule slips and cost overruns, they wonder why. These comments are based experience with Apollo, Shuttle, B-1 and B-2 Bombers.

Richard Olsen

>As others have pointed out before, cost-overruns are not just a NASA problem.

In fact, it is worth keeping in mind that, compared to the frequent cost overruns at DOD and many other agencies, NASA programs are actually pretty good in terms of keeping budgets under control.
http://www.thespacereview.com/article/1248/1
http://gwotbudget.blogspot.com/2008/04/defense-acquisition-problems.html
http://www.nytimes.com/2008/06/25/us/25engineer.html
http://www.rawstory.com/news/afp/US_weapons_budget_comes_under_fire__04012008.html

Solving MSL's problems is a tough problem. It makes no sense to not fly it (NASA won't get back the money already spent, and the remaining money needed to fly it is small compared to the total) and spending a few hundred million dollars to delay it two years is a solution that makes the budget problems worse, not better.

So the real question is, how can we keep this from happening again?

Mariner is right that cost estimating is hard, but I think that NASA's cost people have actually gotten pretty good at it in recent years. I think possum is correct in saying that the problem isn't that NASA can't estimate costs, its that the project managers will run three cost models and pick the one that they want, instead of going with the high one. (or at least justifying - with analysis - why the high model is wrong)

GL--

NASA's good performance relative to other Agencies is no excuse to continue bad cost control practices (just as it is no excuse for the rapist to ask for a pass simply because his cell mate the murderer's crime is worse).

And I think you miscalculate the benefit of canceling a thing or two: The shock wave it would send in changing the culture would more than pay for the sunk cost. Saying that the only "real question is, how can we keep this from happening again?" simply enables further bad performance, because that recipe always results in the same thing: no penalty for projects that overrun.

The Hubble Space Telescope was approved for a development budget of $435M. Actual costs including operations were much more than ten times that amount. And yet because Hubble has been so incredibly successful, it has produced a very good return of science/$. Many of NASA's most successful mission had large amounts of cost growth. However, that doesn't mean that if a program overruns, it's guaranteed to do better science. And there were ways that Hubble could have been developed for less than what it ultimately cost.

The key thing is to focus on specifics rather than generalities. Why is a given program's costs growing? Are there valid reasons based on new information? Are there systemic flaws in design or management approaches? If you descope, what does that do to the science per dollar? It would be possible to fashion a space program with very low risk missions and guarantee that they would not overrun - but that kind of conservatism may also eliminate some of the most productive missions. Almost all of NASA's most successful missions had significant cost growth.

There's a lot of factors at play that makes "cost control" difficult. It's not a matter of management processes (has EVM helped? nope). It's a combination of:

1) large uncertainty in cost to do the work (assuming that scope is fixed) -> the cost to design and build something never been built before, even if similar, is hard to estimate. It's not like construction, where even though the projects differ, you're using processes and materials for which there are centuries of experience. It's not like making cars or widgets on an assembly line, where you amortize the "tooling cost" over many units.

2) changes in scope -> Sure, a lot is political in origin, but it's there none-the-less, and adds to the problem, in non-obvious ways (more cost exercises, more strawman designs to base cost estimates on, etc.) Cassini did dozens and dozens of cost estimating exercises.

3) changes in requirements -> differing from a scope change.. you're doing the same thing, but something has changed.. like maybe the new thermal estimates mean that you need to redesign the electronics to keep junction temps reasonable. Or maybe, the project's risk acceptance posture changes (oh, that instrument can fail changing to that instrument MUST work.. that triggers a lot of design changes)

4) Changes because of bets going wrong - Oops, that component you were planning to use isn't available, because the mfr's fab line was busy doing something else, or it failed the radiation testing. So you redesign and spend time and money qualifying a substitute. The long design/build cycle for spacecraft compared to the commercial world really hurts. Either you use 10 year old technology (and 15 year old parts) or you gamble on technology insertion to meet the mission goals.

Since these things are inevitable, you'd need some way to track the cost impact of them occurring, so that you could fold that into your next estimate. However, the current way of doing things doesn't really provide or collect easily extractable cost impacts. You're doing lots of things and making multiple changes together, to meet the launch date determined by the inexorable motion of the planets. Who's to say that the redesign was triggered by a change in parts availability, new thermal requirements, a new instrument, or whatever.

Maybe the only way, is to start taking money away from NASA. Every time a project runs double the estimated cost, you penalize NASA by reducing its budget the next year. I know people will scream about this comment, but I'll bet that if this is done only once or twice, the cost estimating will suddenly become more realistic.

Remember the old policy (Russian?) of having the aircraft designers fly on the first flight of the aircraft they design? Is there any way to work this into the budget design world? Say, the folks responsible for crafting a project's budget get their salaries docked by the same percentage that the actual budget goes over their estimate? Obviously the project manager's salary would have to be part of the arrangement...and maybe the folks down in the trenches who offered up bogus numbers in the first place?

AND, to prevent them from OVER-padding their budgets, they get docked a certain (smaller) percentage of any OVER-estimates as well.

Maybe engineers would start to care more about that financing stuff then.

Just a thought...

I work exclusively on private sector projects and estimating the time it will take to develop something new is part experience and part guess work. There will always be a percentage of projects that have moderate to major overruns no matter what estimating technique you use. The only way to 100 percent avoid them is to cancel every project when you reach the budget limit (a not very realistic approach). If you could up front know every issue you are going to run into and budget for it, it wouldn't be R&D.

I'm just finishing up a project my company did for a fixed price. Today we ran into an issue that could have been a complete show stopper for the project in the final week or two. It was an issue no one out of a group of a dozen very experienced engineers had any inkling would pop up. Fortunately with a group brainstorming session a simple quick fix was found. I have been on other projects where near this stage major issues have popped up near the end and no simple quick solutions were arrived at. Difficult, time consuming fixes were devised that worked. The customers are never happy about the delays and extra costs no matter how happy they are with the capabilities delivered with the final results.

Until we have enough standard rovers and standard delivery methods for them that you can clone for a variety of missions, overruns like the MSL will happen. Until you get to that stage you have to research new ways to deliver a rover. You have to advance the software that runs the rover. You have to improve the rover design so it can travel farther and do more science than the previous rovers on more challenging terrain. If you want to be on the cutting edge and advance science and technology you have to be willing to risk overruns.

During every project two of the factors that can drive delays are scope creep and overreach of the initial project goals. Any project that pushes a number of technologies to get the desired results are the most likely to have massive overruns. The Lockheed VentureStar was prime example. The Space Shuttle was another. One flew, the other didn't.

Scope creep is natural because as a project moves along people start figuring out more items that would make sense to include. There needs to be very strong reasons to allow new goals to be added to a project. As much as the new features might be very good to have, they are quite often the reason a project goes over budget and is then killed resulting in a net gain of nothing for all parties involved.

I remember reading in a trade journal that 70 percent of IT projects are terminated before completion. Exploding budgets and technical hurdles are quite often the reason. The private sector faces this all the time. If you don't want any cost overruns the answer is simple, don't try anything in the first place. If you put in big contingency funds to start, most projects won't see the light of day. Can we do better? There is always room for improvement.

Several year ago I did a software project for a large Fortune 500 company that was on the other end of the spectrum. It came in thirty percent under budget and about a month ahead of schedule with several extra features beyond what the project specification called for. The engineering manager was not happy. He was afraid that his boss would start expecting results like this all the time.

While trying to improve, accept the fact that there will almost always be problems in any R&D project.

There were independent cost estimates done by IPAO early in the MSL project life cycle that are very close to what reality is today. They were, of course, much higher than the available budget or project-generated estimates (same goes for the schedule, by the way). They were ignored, as is always the case. It is not that difficult to get in the ballpark with a parametric cost estimate which takes into account size, complexity, level of technology development, etc. We have not done this specific project before, but we have done projects that are comparable to the point where a parametric cost estimate can be within 20% (not off by 200% or 300%).

In spite of my previous railing on project management, I do understand that they are hobbled from the get-go in the environment we work in. If they had provided a more realistic estimate showing the cost as $2 billion up front, the project would have been cancelled on the spot. That is the biggest issue. If an accurate estimate is given up front, the project never gets approved because those making the decision cannot swallow the true cost of doing the business that we do. NASA tends to develop an estimate that it think will be accepted to get the project approved. We are ever the optimist when it comes to thinking that this time it will be different. Or we could be collectively insane, repeating the same thing over and over expecting a different result.

Welcome to the aerospace reinvented. Back when functional organizations held a balance of power with the program offices, programs, even big ones, did not have the political power to stomp on all other programs. Today programs are king. You work for a program, you get your raise from a program, everyone serves the program. This gives programs the kind of political power within NASA to do whatever they want whenever they want without regard to the larger objectives and goals of the organization. You want to reign in the almight program? Give back to the functionals the power to control raises and hiring and firing. Do that and you go back to one NASA instead of a loose collection of programs under the NASA banner.

I think Dr. Stern is engaging in a bit of “DO AS I SAY NOT AS I DO – OR DID”.

Let me expand: Dr. Stern is correct in that the majority of NASA space science missions face an overrun to one degree or another. This is endemic and happens at all NASA Centers and with all contractors. One conclusion is that NASA is filled with bunch bungling idiots. This is the conclusion that Dr. Stern jumps to – holding himself out as an exception of course.

There is at least one other explanation. NASA builds complex first of a kind one of a kind missions not pre-fab houses. Even so, Centers and contractors have to commit to a cost when NASA has spent only a few percent of the cost of the mission in the definition phase that is often so short and so under funded that it fails to fully expose the complexity of the mission that is to be developed. Ask most experienced project managers and they will point to this factor as one of the most significant root causes of cost growth in the later part of the mission implementation.

So what did Dr. Stern do about this during the short time that he was in charge and could actually do something to change things? He indulged in Excel sheet engineering planning missions that were poorly funded early (even more so than usual) and then a balloon payment at near the end. (That brings up banking and mortgage analogy but I won’t go there). As an example, he put on the books a Mars Sample Return mission and an Outer Planet mission (both among the most challenging missions in NASA space science history) and put them on a cost profile that would provide token budgets at the start and then provide most of the funds so late that the missions would have been in deep trouble by then. And, had he lasted, undoubtedly he would have started beating up on project managers for his own follies. It isn’t that he was not told about the dangers of these factious budget profiles. It is that then, as now, he thought he knew better than anyone else as to what the answers were.

Dr. Stern you had your chance to reform the budgeting process but you didn’t or you couldn’t. In fact it has taken quite a bit of time to clean up the smoke and mirror budgets that you left behind. So please spare us the lecture on cost containment.

Having said that, let me end this on a conciliatory note. In the recent Space News issue when asked if you would want to be the next NASA Administrator you said “in my eyes there are better candidates”. I fully agree. Although I don't think you meant it. In your Op Ed in the NY Times you say that: “This is a challenge that can be met with appropriate leadership”. Did you have anyone special in mind?

been.there.done.that

"If they had provided a more realistic estimate showing the cost as $2 billion up front, the project would have been cancelled on the spot."

That's absolutely correct. This is standard car salesman "bait and switch" tactics on the showroom floor, with the inevitable sticker shock when all the nice-to-have options are added and it's time to write the big check. The wise consumer would just walk away from such blatant dishonesty.

Congress would never have approved $2 billion for a program that "will assess whether Mars ever had an environment capable of supporting microbial life" (http://marsprogram.jpl.nasa.gov/msl/science/). That's mildly interesting, but so what? Is that worth $2 billion to the taxpayer?

It's difficult for the average Joe or Jane to get excited about a question where either a Yes or No answer is irrelevant to them being able to pay their mortgage next month or find affordable health insurance coverage.

NASA knows that, so they play their project low balling cost estimating games, and then feed the hungry, bloated, half-completed missions the seed corn for the future, hoping that Congress will have amnesia come next fiscal year and provide a fresh and possibly larger corn sack.

Oddly enough, this strategy works much better than it should.

Here are some ways to get it somewhat under control:

- Use something like the COTS model more frequently. Encourage capable commercial partners to implement something NASA wants by sharing the costs in cases where the commercial partners see NASA and non-NASA markets. The added commercial funding, and "skin in the game" on the commercial side, will help share and lower the risk. Follow commercial demonstrations with fixed-cost contracts.

- Increase Centennial Challenges types of rewards for achieving goals. Follow up similar private-sector efforts like the Google Lunar X PRIZE with NASA business for the resulting commercial systems.

- Reduce emphasis on large programs like Ares, Mars Sample Return, and so on. Increase emphasis on programs of a more manageable size: commercial suborbital missions, smallsats, hosted payloads, instruments on foreign space agency missions, etc.

- Increase use of program "lines" that use incrementally improved hardware (i.e. numerous rovers with similar size and delivery mechanisms, numerous satellites from the same "mold", etc).

- When a new "line" of missions, or the occasional unique mission, is made, make sure the technology at least is ready before starting the program. Have an adequate, separate budget for technology development and technology demonstration (New Millenium style) in space. Such cutting edge R&D and demonstration efforts might very well have cost overruns; have ample buffer funds, and don't be afraid to cancel them when needed.

- Focus more NASA effort on efforts to generally reduce the costs of doing business in space so any cost overruns occur in a lower-cost environment. This might include encouraging commercial suborbital and orbital low-cost space access efforts through NASA business, NACA-style R&D in space access, demos of orbital propellant depots and reusable space tugs, etc. Forget new space access development and operations plans that are high-cost.

I'd like to understand John Smith's claims about "spreadsheet engineering" a bit better. As I understand it, the budget process at NASA requires multiple levels of technical review internally, most notably by hard-nosed PA&E, followed by OMB review externally, before any new proposed mission and mission budget can become part of the President's budget. Are you saying that all of these reviews failed to spot budget bull**it? That would be pretty scary. Or could it be that in fact the budgets submitted actually did make a fair amount of sense, and it your information that is flawed?

Simple, no more than one low TRL item per project. Maturing TRLs on different bits of technology become separate projects. Smaller cost caps on lower TRL projects.

Behemoth hundred million dollar deals would only be undertaken when everything that needs to go into it is above TRL7

This would essentially result in way more smaller tech demo projects to do tech shakeout and R&D.

Something like X-33 would never be approved to begin with with such controls in place.

red wrote:
...Increase emphasis on programs of a more manageable size: commercial suborbital missions, ...

It's worth pointing out that NASA already has a suborbital program, and had had one since the very first day of its existence. Sounding rockets are being launched with little fanfare and almost no public attention out of Wallops. You're basically saying "increase the emphasis on a program that has been going on continually for fifty years and nobody pays any attention to." There's not a whole lot of cheap, simple suborbital science that's there to be done-- if there were, the flights are available; the rockets are cheap and reliable and are well proven.

You use the phrase "commercial suborbital missions" as if a commercial market for suborbital science flights actually existed, but that market is so small as to be essentially nonexistent-- there's just not that much that can be done in a few minutes of microgravity; not enough to sustain any kind of commercial market. The real market in suborbital flight is going to be tourism, which could be quite a significant market... but that's something that really doesn't need NASA; it's a market that is developing on its own quite nicely. There's no reason to put NASA into the commercial suborbital tourism business.

GL, we shouldn't prop up the commercial suborbital tourist market, but we should make sure those flights are available to scientists to use if desired and appropriate. (I agree, though, that "red" seems to overemphasize the importance of those flights to NASA.)

A previous poster commented:

"Use something like the COTS model more frequently. Encourage capable commercial partners to implement something NASA wants by sharing the costs in cases where the commercial partners see NASA and non-NASA markets. The added commercial funding, and "skin in the game" on the commercial side, will help share and lower the risk. Follow commercial demonstrations with fixed-cost contracts."

Except that there's really not much commercial demand for a lot of the technology that goes into a MSL sort of mission. How many commercial customers are there for deep space radios that work at 8 bits/second? What about radiation tolerant electronics? (yep, big market for that iPod-RH..) Flight qualified X-ray spectrometers?

Then, the next question... are you willing to bet your billion dollar farm on COTS components and assembly? You only get one shot, remember. not like you can return it for warranty or say "well.. it has 99.9% reliability, and too bad you got the 1 in a 1000 that died"

The one-off nature of what we do for space exploration is very, very different from a COTS product development model, where you spread the NRE over many units. The design/build/use cycle is an order of magnitude longer, if nothing else. I'm not sure that technology developed and proven out for a flight application actually has much commercial applicability, at least in a "saving money on product development" sense. Sure, there's fundamental advances in technology that come out of what we do for exploration, and there are examples of commercialization, but I'd venture that most of those "forked" between space/commercial back at the TRL 2 or TRL 3 level, when the costs are still pretty small (a few hundred k$).