REG NASA-LLIS-1182--1990 Lessons Learned - Atlantis Fuel Cell Mishap Investigation Board Report - April 4 1990.pdf

1、Lessons Learned Entry: 1182Lesson Info:a71 Lesson Number: 1182a71 Lesson Date: 1990-04-04a71 Submitting Organization: KSCa71 Submitted by: George W.S. Abbey/ Eric RaynorSubject: Atlantis Fuel Cell Mishap Investigation Board Report - April 4, 1990 Description of Driving Event: On April 4, 1990, at ap

2、proximately 9:05 a.m. EDT, one of the three fuel cells (#3) installed in the Orbiter Atlantis, OV-104, was damaged while an attempt was being made to vent the fuel cell prior to its removal and replacement. Atlantis returned to the Kennedy Space Center (KSC) on March 3, 1990, after successfully comp

3、leting the STS-36 mission. The vehicle was undergoing processing in the Orbiter Processing Facility (OPF) in preparation for the STS-38 mission in July of this year. Testing and processing was being accomplished by the Shuttle Processing Contractor (SPC) at KSC. The Fuel Cell Single Cell Voltage Tes

4、t was accomplished on March 30 and 31, and the analysis of the test results indicated there were two degraded internal cells. A decision to remove and replace the fuel cell was subsequently made on April 2 by the Orbiter Project Manager.The accident occurred while attempting to vent the fuel cell wi

5、th the Orbiter hydrogen (H 2) purge vent port capped. This allowed the H2 pressure to exceed the oxygen (02) pressure in the fuel cell, 2 side of the fuel cell. The Potassium Hydroxide (KOH) was found at the 02 purge port of the fuel cell indicating the ninety-six internal cells, the regulator, and

6、the accumulator would have to be replaced due to the corrosive qualities of KOH. No one was injured and damage was limited to fuel cell #3.Lesson(s) Learned: On Saturday, March 31, the mid-body mechanical supervisor and the lead mechanical technician both had the day off. Supervision was delegated t

7、o the lead electrical technician for that shift. He was not familiar with the work to be accomplished or with the capabilities and experience levels of the individuals assigned from the mechanical group. Two individuals were assigned to a task they had not previously performed or observed. The quali

8、ty inspector was a check and balance that might have caught the error had he been assigned based on his experience with the task or his knowledge of the system. Quality inspectors are not given inspection work based upon their knowledge or previous Provided by IHSNot for ResaleNo reproduction or net

9、working permitted without license from IHS-,-,-inspection assignments. They respond to a call-board that lists tasks calling for an inspector; consequently, an inspector appears in a somewhat indiscriminate fashion to perform the inspection task.On second shift, Saturday, March 31, three people unfa

10、miliar with the task came together on platform 4 west to perform and verify Operations and Maintenance Instruction (OMI) V1093 Post Operations Instruction #3.The lead mechanical technician from first shift stayed over to take the assigned technician to platform 4 west to review the task. He failed t

11、o mention that the purge vent port should not be capped after removal of the flex hose. There was no precautionary note in the OMI addressing the need to keep the purge vent port clear and open and no placard on the Orbiter. One of the significant shop practices stressed to all technicians working a

12、round the Orbiter is to cap disconnected lines or openings to avoid contamination. Personnel are therefore naturally conditioned to cap lines and openings.The fuel cell had to be vented to ambient pressure prior to its removal. The first attempt to vent the fuel cell by a fuel cell system engineer o

13、ccurred on Wednesday, April 4, at approximately 2 a.m. EDT. The fuel cell was not damaged at this point because the 02 reactant valve did not close. The fact that there was no H2 flow was observed but no Interim Problem Report (IPR) was initiated. A disconnected ground wire from the 02 reactant valv

14、e prevented the valve from closing and consequently protected the fuel cell from any damage.A walkdown was not accomplished by engineering. Had such a procedure been implemented, the damage might have been avoided as the fuel cell system engineer might have been alerted to the disconnected ground wi

15、re and the capped purge vent port.One thread that runs through the series of events leading up to the mishap is the lack of system training for technicians and quality inspectors. Technicians and quality inspectors are trained and certified in their basic skills, i.e, lockwiring, torquing, etc.; how

16、ever, they are given no Orbiter systems training. Another factor that contributed to this mishap is the lack of communication between engineering, technicians, and quality. All OMIs are reviewed and approved in detail by several engineering organizations. Some portions of these approved OMIs are the

17、n handed over to shop personnel (technicians) to be performed without requiring additional engineering involvement. This same situation exists with the quality inspector who must approve the technicians work. The personnel are highly motivated and want very much to do a good job. It is essential tha

18、t they be given the knowledge they need to do that job.The importance of keeping the purge vent port clear and open was not communicated by anyone in the process. The capping of the Orbiter H2 purge vent port during the performance of OMI V1093 Post Operations Instruction #3 was not an isolated inci

19、dent. On March 19, the same port was capped on the Orbiter Columbia, OV-102, during performance of the same OMI. In this case, an experienced Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-technician checking the work caught the error and had the ca

20、p removed. The supervisor was notified but no subsequent action was taken. Had this near-miss been properly communicated to all technicians, engineering, and quality, perhaps the mishap two weeks later could have been avoided. The significance of effective communications cannot be overemphasized in

21、the very complex world of Orbiter processing and test operations. Communications and identification of precautions in the OMI and a good working relationship between the technicians, the engineers, and quality personnel is essential to successful operations.The geography of KSC facilities does not l

22、end itself to ease of overview or development of team unity. The Launch Control Center (LCC) control rooms, from which systems engineers conduct tests, are a considerable distance from the OPF where technicians perform work on the vehicle. The remoteness of the two facilities impedes communications

23、between engineers and technicians. As the program plans new control rooms and equipment, these new rooms could be located in the OPF between the two bays. Many benefits would be derived from the control rooms being in such a location. Having the engineers in close proximity to technicians and the ve

24、hicle would improve the overall efficiency of orbiter processing as well as communications among all parties. A key factor in achieving a higher flight rate is reducing OPF flow time. This change could substantially enhance KSCs capability to achieve shorter processing flows. The ability of engineer

25、s to perform walkdowns prior to implementing test operations would be greatly facilitated by their proximity and involvement with the work.Operations and Maintenance Instructions are obviously the key documents in ensuring work is done properly at KSC. A review to ensure they are correct and adequat

26、e is essential. Following the Challenger accident, direction was given to the design centers to review and concur in all OMIs that affected Criticality 1 or 1R equipment as well as changes to those OMIs. A commitment was also made to the Congress that this would be implemented. In the case of the or

27、biter, this does not now occur. The Johnson Space Center did review and concur in all OMIs after the Challenger accident and, until the summer of 1989, the JSC Resident Office at KSC concurred in all changes.The apparent lack of discipline in utilizing the paper system resulted in a communication br

28、eakdown and confusion that may have contributed to the mishap. There should be a coordinated and documented plan of action between work shifts for accomplishing processing and testing. First and second shifts utilized two different work authorization documents (WADs) to perform the same task. An IPR

29、 was not prepared when anomalous conditions were observed. Constraints were not identified when performing electrical work, i.e., removal of ground wires. A deviation was not logged in a timely fashion.The technicians, engineers, and inspectors at KSC are a highly motivated and dedicated team. Proce

30、ssing and testing of Orbiters is a very complex and demanding task. These personnel are indeed on the firing line three shifts a day, seven days a week. Problems and difficulties are always present. With so much depending upon their ability to perform their work, it behooves us to provide them with

31、all the skills and talents they require. Training, knowledge, and communications are all essential to the task. The cost for repair of this fuel cell is significant, but considering Space Shuttle Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Operat

32、ions will continue for as long as 20 more years, it could be an inexpensive lesson if we can learn from this accident and truly apply the lessons learned.Recommendation(s): PRIMARY CAUSE - THE HYDROGEN (H2) PURGE VENT PORT ON ORBITER OV-104 WAS CAPPED.Finding 1: The H2 purge vent port was capped alt

33、hough there were no steps in OMI V1093 Post Operations Instruction #3 to cap the vent.Recommendation: An alert should be widely circulated to all technicians and quality inspectors emphasizing the criticality of rigorously following procedures. The alert should include the necessity to obtain a devi

34、ation to the work document to perform any work (including usual shop practices that alter configuration of the vehicle or ground support equipment GSE not specifically identified in the OMI.Finding 2: None of the personnel involved (two mechanical technicians and a quality inspector) were aware of t

35、he criticality of the H2 purge vent port remaining clear and open at all times.Recommendation: Precautionary notes should be included in OMI V1093 relative to not capping critical vents that must remain clear and open. All OMIs should be reviewed to identify such vents and appropriate precautionary

36、notes incorporated. Placards should be utilized on the vehicle to identify critical vents that must remain clear and open.Finding 3: System training and certification relative thereto are not required for technicians or quality inspectors working on the Orbiter.Recommendation: System training and kn

37、owledge should be made part of certification requirements for all technicians and quality inspectors and should be one of the primary considerations for pay increases and promotions. System training should be made available during off-shift hours. A systematic review of all training and certificatio

38、n requirements should be accomplished across the contractor organization. Currency and proficiency training should be a major consideration.Finding 4: The two mechanical technicians and the quality inspector assigned to perform OMI V1093 Post Operations Instruction #3 had no previous on-the-job trai

39、ning or experience performing this task.Recommendation: Technicians and quality inspectors should not be assigned to a task they have not previously performed without the supervision of a system engineer, supervisor, or another experienced technician who has performed the task.Provided by IHSNot for

40、 ResaleNo reproduction or networking permitted without license from IHS-,-,-Finding 5: With the exception of the part number, the nomenclature on the “bulkhead“ fitting did not agree with the nomenclature in the OMI.Recommendation: Orbiter and GSE displayed nomenclature should be reviewed to ensure

41、it is consistent with the OMIs.Finding 6: Operation and maintenance instructions are reviewed and approved by several organizations/contractors. This review and approval process involves system engineering, quality engineering, and safety. After engineering approval, portions of the OMIs are handed

42、over to the technicians to perform without any requirement for further engineering involvement or direction.Recommendation: Station system engineering personnel in the OPF to witness and direct the conduct of their respective OMIs.CONTRIBUTING CAUSE: FAILURE TO INVESTIGATE THE LACK OF H2 FLOW AND TO

43、 GENERATE AN INTERIM PROBLEM REPORT (IPR).Finding 7: The second shift fuel cell system engineer did not prepare an IPR for either of the anomalies, i.e., the failure of the 02 reactant valve to close on command and the lack of H2 flow with the H2 reactant valve closed and the purge valve open.Recomm

44、endation: Re-emphasize to engineers that IPRs must be prepared and documented real time for all anomalous conditions.OBSERVATIONS1: The midbody mechanical supervisor and lead mechanical technician were off work the night of the mishap. The midbody lead electrical technician was supervising the mecha

45、nical technicians. He was not familiar with the capabilities of the personnel or the task being performed.Recommendation: In the absence of a discipline supervisor and lead technician, supervision should be delegated to the next senior technician within that discipline on the shift. The delegation a

46、nd chain of command should be clearly understood.2: On March 19, during OMI V1093 Post Operations Instruction #3 on Orbiter Columbia, OV-102 (OPF High Bay 1), the H2 purge vent port was also capped. An experienced technician sent to review the work found the purge vent port capped and directed the c

47、ap be removed. The incident was reported to the supervisor but there was no shop-wide awareness. The inadvertent capping was neither discussed at “tailgate“ meetings nor were engineering and quality notified.Recommendation: Any improper performance of an OMI step should be documented and generate a

48、review of that OMI by the shop supervisor, engineering, and quality to determine if the document Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-needs to be revised. When improper work practices are discovered an alert should be generated and circula

49、ted within the technician, engineering, and quality organizations. These alerts should be discussed at weekly “tailgate“ meetings. Engineers should attend technician “tailgate“ meetings to discuss system issues or problems related to processing which are of interest to both groups.3: A walkdown of the area was not performed prior to fuel cell venting.Recommendation: A walkdown should be performed by engineers in the OPF prior to