|
Preface: The following is from the Executive Summary of "Mars Climate Orbiter Mishap Investigation Board 1 Report".
Root cause: Failure to use metric units in the coding of a ground software file, "Small Forces" used in trajectory models.
 Mars Climate Orbiter Mishap Press Conference Summarycopyright Keith Cowing, Editor, NASA Watch 10 November 1999 The press were briefed in the NASA Headquarters Auditorium by Ed Weiler, Associate Administrator for Space Science, Art Stephenson, MSFC Center Director, and Ed Stone, JPL Center Director. Weiler opened by stating that the Mars Climate Orbiter (MCO) either burnt up and hit Mars or was thrown into a solar orbit "we're not sure which". He described the review of this accident (or "mishap") as having 2 phases. The first (the topic of the day's press conference) was to determine the root cause of the failure of the mission and to recommend preventative measures to preclude problems from occurring in the Mars Polar Lander (MPL) mission. The second phase, with a report due on 1 February 2000, was to examine the processes used to manage MCO, develop lessons learned, and to suggest corrective actions needed to fix identified problem areas. Weiler then sought to provide a summary of the upcoming information that would be presented later in the press conference by saying that the direct cause of the failure was the use of metric units Vs British units. According to Weiler "our check and balance process did not catch this." Weiler went on to say that no individual is to blame - but rather that the entire MCO team - NASA, JPL, Lockheed Martin - "shares the responsibility for this mishap". Art Stephenson then presented a summary of the root cause and the contributing causes. Noting that "going into space is not easy - it is hard" he stated that "two good spacecraft were built" and, in alluding to the accident, noted that "little things can come back and make a big difference". Stephenson described the basic design of the MCO noting that its design symmetry required periodic adjustment burns to adjust for the cumulative effects of solar pressure on the satellite's photovoltaic arrays, and the effect that this had on the satellite's axis over time. These perturbations were supposed to be taken care of with periodic angular momentum desaturation maneuvers (AMD) done 10 to 14 times per week. While the program requirements asked for the AMD data to be provided in metric units by Lockheed Martin, these numbers were actually provided by Lockheed Martin to JPL British units for quite some time. The JPL Navigation Team expected to get metric units and treated the data they got as being metric. Over a long period of time these small incremental errors became magnified. Eventually, as MCO got close to Mars, the changes in the trajectory were 4.4 times higher than the Navigation Team was thinking they were. Stephenson went on to note that problems were further exacerbated by the fact that the Navigation team was not familiar with this spacecraft and had only stared working on it 2 months before launch. Further, the team had not participated in the PDR (preliminary design review) or CDR (critical design review) for the spacecraft, and that they had erroneously assumed that the operations of MCO would be similar to the Mars Global Surveyor, a wholly different spacecraft. Stephenson then described the TCM 5 burn which was listed by the project as being a "contingency". As the MCO got closer to Mars, the Navigation Team saw that the spacecraft was coming in too low. They asked for approval to plan for and perform the TCM 5 burn, but their request was denied - the reason being that the team was not ready to actually do the burn in time to be ready for the Mars Orbit Insertion burn shortly thereafter. Another problem arose when a systems engineering requirement to turn the spacecraft on a regular basis was modified (without consultation with systems engineering) so as to simply operations and save power. Stephenson noted that had this "barbecue mode" been kept in place, as had been the expressed intent of the designers, the small forces that served to compound MCO's trajectory variance would have been balanced out. Stephenson noted that there was not adequate communications between various project elements, that the Navigation Team had inadequate operations team staffing, that a number of individuals had received inadequate training, and finally, that the overall mission validation process did not adequately address ground software. Regarding recommendations, Stephenson noted that close attention to the proper use of units of measurement was important. He added that software audits need to be done, that close attention needs to be paid to verifying and correcting for small forces, and that future missions should expect to be ready to perform a TCM 5 burn and not hold it as a contingency activity. He then touched on the one main engineering concern expressed on the MPL mission by the report - the propulsion system. The team became concerned that temperatures on the spacecraft might be low enough such that frozen fuel might exist and serve as potential source of clogging of fuel lines. This concern was heightened by the fact that the pulsed mode engine concept was making its first appearance on a landing mission. Turning heaters on earlier than originally planed would, according to the team, serve to make certain that things were sufficiently warm so as to make frozen fuel an unlikely occurrence. JPL Center Director Ed Stone then spoke, reiterating earlier concerns i.e. that "we made a serious error. MOC was on the wrong trajectory and our checks and balances did not detect this." He stated that JPL is working to assure mission success for MPL and that "we are doubling and redoubling our efforts" so as to examine everything in preparation for the 3 December landing. Stone then addressed the recommendations and corrective action being taken at JPL. He described changes to management as involving appointment of senior managers and clarification of all lines of programmatic communication. With regards to the Navigation team, he described how team members are being added and that near-simultaneous tracking of the MPL using a dedicated antenna will be performed. This process will track both the Mars Global Surveyor and Mars Polar Lander spacecraft so as to provide an independent calculation of the MPL with respect to Mars. He added that a preparation for a TCM 5 burn was in place and that it would be used. Stone spoke of improving coordination with systems engineering. He repeated that the heaters will be turned on within MPL early enough so as to assure that the propulsion system is warm enough to negate fuel freezing. He added that peer review will be used to examine plans and procedures, that a Red team that will be used to ask tough question of what the project is doing, and that an increased emphasis will be placed on mission assurance so as to be certain that all of the checks and balances are in place and working. He closed by noting that there are "hard lessons to be learned" from the MOC incident. The session was then opened to questions from reporters. A question regarding the "Cheaper-Better-Faster" approach and whether this experience should serve to undermine NASA's confidence in it brought a clear response from Ed Stone. He noted that it was "important to note that the problems happened in only one area". He added that the transition from a time wherein larger teams with inherent checks and balances to one where smaller teams relay upon a certain amount of external checks and balances as having some effect. Another question on this topic brought a response from Ed Weiler who said that it was clear that there were places where established procedures were not followed, and that this should have been caught, but that this incident should not be taken as an indictment of the overall cheaper-better-faster approach. A question was then asked of the panel as to whether this whole incident was not indicative of a larger, more fundamental flaw, and that someone should either be reassigned or fired. Weiler shot back with "I can't speculate on people's psychology". He added "do we fire the people? Who do I replace them with? These are the best people who do these things in the world". Weiler then encouraged people to speak out if they saw problems saying that "they can go all the way up the chain -- they can even call me -- they won't get fired."
Editor's note: Ed Weiler can be reached at eweiler@mail.hq.nasa.gov, +1.202.358.1409 (voice) , or +1.202.358.3096 (fax).Another question made note of the fact that historically speaking, that NASA's problems (Apollo 1 fire, etc) often had an individual or individuals who waved "a red flag" trying to alert management to impending problems. A reporter asked if this had happened with MCO. Stephenson replied that the Navigation team had known for quite some time that there was some problem, but that he guessed that they did not think it serious enough to raise it as a major issue. He added that the Navigation Team was more concerned with fine tuning the orbit and that they were not acting as if there was a looming problem ahead. A question from a reporter at KSC asked whether Lockheed Martin had indeed not done work according to NASA's specifications (i.e. metric units) brought a simple response from Lockheed Martin representative Noel Hinners i.e. that it was "overlooked." A follow-up question asked if the specification was there but no requirement existed to check it. Hinners again responded that the specification was supposed to have been checked but that Lockheed Martin did not check it. Another question for Hinners concerned the possible effect of downsizing on the MCO. Hinners replied that he did not think it had an effect and that Lockheed Martin always tries to capture lessons learned from senior personnel so that newer employees can get the full benefit of their experience. A question from JSC caused a minor stir. Jim Oberg asked if the "gag order" was going to be lifted on people involved with the project and if they'd be able to talk openly about what happened. Ed Stone replied rather sternly that the team had a landing to prepare for in three weeks and that this was their main focus right now. Stone did not directly address the "gag order" issue. A follow-up question from NASA HQ addressed the issue again. Once again Stone dodged the issue and stated that a "report will be issued in February and that it will probably be released". He did not directly address the issue of employees having the right to speak out.
Editor's note: members of the MCO and MPL teams from JPL, NASA, and Lockheed Martin are hereby invited to pass on their experiences to NASA Watch at nasawatch@reston.com. Let us know if your name can be used or if you'd prefer that it not be used.Another question touched on Lockheed Martin's award fee for this project and how it would be affected by the loss of the spacecraft. Neither Ed Weiler or Ed Stone knew that total possible award fee for both the MCO and MPL projects could be. But said they'd get back to the reporter who asked the question. Although prodded several times, Ed Stone refused to speculate on what penalty (if any) might be forthcoming for Lockheed Martin other than to refer to an established governmental process whereby such fees are determined -and that this would happen at some point after the MPL landing.
|