1、Lessons Learned Entry: 1580Lesson Info:a71 Lesson Number: 1580a71 Lesson Date: 2005-01-18a71 Submitting Organization: GSFCa71 Submitted by: David Coolidgea71 Authored by: Karen HaltermanSubject: This document contains lessons learned from the NOAA-N Prime mishap which occurred at Lockheed Martin Spa
2、ce Systems Company in Sunnyvale, CA on September 6, 2003. Abstract: The fully assembled NOAA N-Prime satellite was significantly damaged after sliding off an improperly configured cart while being rotated from a vertical position to a horizontal orientation for an instrument shimming operation at th
3、e prime contractors facility on September 6, 2003. This document presents several lessons learned and recommendations for proactive measures to prevent a recurrence of this unfortunate accident. They may be considered applicable to any government/contractor collaborative relationship established to
4、build, test, and launch space flight hardware. The lessons learned relate to the Integration and Test (I the planned operational scenario vs. the actual execution; and the planning activities, including scheduling, crew assembly and test documentation preparation. The second approach was to utilize
5、the Human Factors Analysis and Classification System (HFACS) 2000 to provide a comprehensive framework for identifying and analyzing human error. Evidence from a number of sources, including witness interviews, test and handling procedures, and project documents, were used to develop the accident sc
6、enarios and populate the HFACS model. The Board found that the direct cause of the mishap was a failure of the contractor operations team to follow procedures to properly configure the turn over cart prior to placing NOAA-N Prime on the cart. The necessary 24 bolts to secure the adapter plate were n
7、ot in place and the team relied on paperwork rather than through visual and mechanical verification as required by the procedures. The Board also discovered that the entire operation was flawed and it exposed issues that were systemic in nature. The I providing training for supervisors in monitoring
8、 employees and correcting poor process discipline; establishing effective process guidelines for regulating I staffing product assurance and safety personnel according to requirements; establishing an effective safety program and promoting safety awareness to all levels of the organization, establis
9、hing an effective I using video monitoring as an aid to supervision and performance monitoring; and establishing a training program to disseminate lessons learned from this and other mishaps and near misses. The NASA corrective actions described in the NOAA-N Prime Corrective Action Plan and Impleme
10、ntation Report include: providing a full time, dedicated government civil servant in-plant representative; establishing clear roles and responsibilities for the government in-plant representatives; providing sufficient resources for the Defense Contract Management Agency (DCMA) product assurance fun
11、ctions; establishing oversight guidelines for I implementing oversight guidelines for I establishing an effective safety oversight program; implementing a thorough I and NASA conducting periodic independent reviews of the GSFC POES Project, which manages the Lockheed Martin spacecraft contract. In a
12、ddition, DCMA evaluated the effectiveness of their oversight process and more DCMA resources have been applied to the NOAA contract. Lesson(s) Learned: The lessons learned in the NOAA-N Prime mishap are listed below. They are not profound; they are all obvious. The accident occurred because a long d
13、uration program that had experienced several years of high performance success grew complacent, both at the government and at the contractor. The rules were relaxed, shortcuts were taken, product assurance grew sloppy, and proper oversight was not applied. The recommended actions are equally straigh
14、tforward. Obey the rules, follow the procedures proactively, train the team, always consider systems safety, and provide effective independent oversight. In order to emphasize specific points, ten lessons are identified. The lessons are not independent, stand alone items. All are interrelated facets
15、 of the overall NOAA-N Prime mishap lesson which is to perform I no one saw it coming. The government and the contractor were overconfident and complacent based on past successes. The lesson learned is that successful performance in the past does not predict success in the future, especially on long
16、 duration projects. Even the most extraordinary effort in the past does not guarantee that future efforts will be sustained at the same high quality level. Proactive measures must be taken to maintain a highly functional I the test procedure was scissored out of existing procedures without a clear t
17、op level road map. The Responsible Test Engineer (RTE) did not have the full I the proper I and hazardous operations need special care. The product assurance representatives must understand their roles as witnesses and they must be trained to not buy off procedure steps unless they have personally w
18、itnessed them.Everyone needs to be concerned about the safety of people and high value flight hardware. If any member of the team has a question or a doubt, it should be investigated, not dismissed. Each member of the team should be empowered to halt an operation. Organizational practices must be es
19、tablished to reinforce the role and responsibility of contractor and government inspectors as independent verification agents. Long duration contracts and/or contracts for multiple satellites need extra attention. While there are advantages to being familiar with the spacecraft and the I safety engi
20、neering should not be a part time function drawn from a centralized corporate safety engineering pool. Safety engineers should participate in satellite design, manufacturing and I they should be replaced with specific actions having measurable results where possible. I&T team members should be able
21、to ask questions at any time and every I&T team member should know that he is authorized to stop an operation if he feels there is a problem. Government presence required by a procedure can only be waived by approval from the Project Offices System Assurance Manager. Waiving mandatory government I&T
22、 presence should be rarely, if ever, permitted. The role of the government must be clear. Which government I&T responsibilities are delegated by NASA to DCMA should be known to everyone in the contractors and governments organizations. NASA responsibilities on the I&T floor should be defined so that
23、 the contractor understands each type of oversight. The contractor must track, examine, categorize and trend the nature and closure of the contractor identified non-conformance reports or the DCMA generated Corrective Action Requests. The government must also monitor and track the closure of actions
24、, deficiencies, and recommendations resulting from outside audits and reviews. All results should be shared with the contractor. Corrective measures to improve I&T should be jointly developed with the contractor and jointly monitored for effectiveness. Provided by IHSNot for ResaleNo reproduction or
25、 networking permitted without license from IHS-,-,-There must be consequences if procedure execution discipline is not followed. I&T employees need to be told by their supervisors what is expected of them. Poor performers should not have access to high value flight hardware. The supervisors must lea
26、d by example and follow all the rules themselves. 7. Develop clear documentation, minimize use of redlines Lessons Learned: The operation being run on the day of the NOAA-N Prime mishap was a unique, one-time activity. It consisted of removing an instrument, installing a shim, and reinstalling the i
27、nstrument. Individual segments of the operation, such as the instrument installation, were documented in released and previously run I&T test procedures. But, instead of developing a new procedure for the shimming operation, portions of existing procedures were cobbled together with red-lines. This
28、made it difficult to follow the flow as it wove in and out of various procedures In addition, portions of the existing procedures were red-lined. All of this out of sequence activity may have confused the I&T crew and contributed to the mishap. Red-lines are useful tools that enable the continuation
29、 of an I&T operation when minor unexpected problem arise. But, the use of red-lines must be minimized and carefully controlled. Red-lines must be clearly understood and appropriate to the operation before they are executed. If a procedure is red-lined during an I&T operation because of an unforeseen
30、 circumstance, there must be concurrence by the independent product assurance personnel and government quality witnesses. If any member of the I&T team questions the red-line, then the operation should be stopped and the issue resolved with the cognizant engineering personnel. Red-lines must be docu
31、mented as either one-time or permanent for future procedure use. It is up to the contractors product assurance organization and government oversight to ensure that red-lines are not used on the floor repeatedly. If a red-line is determined to be a permanent change, then the procedure should be forma
32、lly updated through the established configuration control review and approval process. The contractor and the government should audit the use of red-lines to assure that they are used appropriately. If excessive use of red-lines are found, corrective measures should be taken by the contractor. 8. Pe
33、rform advanced I&T planning Lessons Learned: Sufficient time must be allocated to prepare for each I&T operation according to approved configuration management requirements. The NOAA-N Prime instrument shimming operation was inserted into the I&T schedule on a Thursday with the intent to execute it
34、two days later on Saturday. There was a rush to develop the authorizing paperwork. The preparation of the I&T procedure for an operation that had never been performed in this sequence was hurried. It was difficult to assemble an I&T team on short notice, most technicians who were approached on Frida
35、y about working on Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Saturday declined. No I&T operation has such urgency that planning can be bypassed. All new I&T operations must be carefully planned and communicated to all participants. It should no
36、t be a race against time to prepare the paperwork and search out an I&T team. The focus of a new I&T operation should be on accurately preparing it. System safety considerations must be made. All configuration management steps must be followed to develop new procedures. A full review should be made
37、of the procedure by all appropriate groups (system engineering, I&T, quality, safety, government, etc.) prior to the release of a new procedure. There are times when a complete procedure cannot be developed in advance, such as while troubleshooting an intermittent problem where the follow on steps v
38、ary based on the test results. Even in these cases, there should be time to plan the basic approach and assemble the correct I&T team. The government should be aware of the amount of time it takes to prepare I&T documentation and should not allow the contractor to work with the flight hardware witho
39、ut the required paperwork. The contractors product assurance organization should not allow an operation to proceed if it has not fully matured. The NASA in-plant representatives or DCMA should stop any operation that does not appear to be properly planned. 9. Provide configuration control of ground
40、support equipment that interfaces to flight hardware Lessons Learned: Two different spacecraft programs were housed in the same high bay, clean room complex when the NOAA-N Prime accident occurred. Each had multiple satellites in production. The programs used similar mechanical ground equipment incl
41、uding two functionally identical satellite turn over carts. There were two turn over carts that could be used by either program when configured correctly. This shared equipment was not maintained under configuration control. The prevailing philosophy at the time of the mishap was that the configurat
42、ion of the mechanical ground equipment should be checked prior to use in an I&T operation. The accident occurred because the procedure to assure that the turn over cart was in the proper configuration was not followed. The root cause of the accident was the bolts that attached the adapter plate to t
43、he turn over cart were missing, so the satellite fell off the cart as it was being tilted from vertical to horizontal. Had the turn over cart been under configuration control, the bolts should not have been removed without the proper authority and coordination with the I&T team that was using the tu
44、rn over cart. All ground equipment, mechanical and electrical, that interfaces to flight hardware should be under configuration control. Electrical ground equipment, especially software systems, must be configured items maintained in a known, reproducible state. Data base values, limits, coefficient
45、s, flight software load images, ground software, calibration status, etc. must all be controlled so that Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-spacecraft test results can be evaluated and compared from test to test. Similarly, mechanical gr
46、ound equipment must be controlled in a known state. Proof testing data, drawings and schematics, failure modes effects analysis, operational hazard analysis, etc. must be available so that the equipment can be used with confidence. Configuration control should also account for the whereabouts of mec
47、hanical equipment so that it can be easily located when needed in a shared use environment. Note that configuration management of mechanical ground equipment does not relieve the I&T crew of the responsibility of verifying its configuration before use. 10. An effective government in-plant office is
48、needed Lessons Learned: This section applies to a government in-plant resident office in the I&T phase of spacecraft development. The Defense Contract Management Administration usually has an office in the plant of major aerospace contractors. The NASA project also typically maintains a small in-pla
49、nt group of engineers who have product assurance and engineering functions. The NASA project generally delegates specific spacecraft I&T product assurance functions to DCMA. The roles of the in-plant office and DCMA must be clearly established and communicated to the entire I&T team. The contractor typically receives a copy of NASAs Letter of Delegation of responsibilities to DCMA, but the contents may not always be provided to the I&T team. It is confusing to the contractor when it is uncle
