REG NASA-LLIS-6396-2012 Lessons Learned - An Electrical Outage Revealed Emergency Systems Not Functioning as Expected.pdf

1、Public Lessons Learned Entry: 6396 Lesson Info: Lesson Number: 6396 Lesson Date: 2011-2-1 Submitting Organization: JPL Submitted by: David Oberhettinger Subject: An Electrical Outage Revealed Emergency Systems Not Functioning as Expected Abstract: JPL experienced a total electrical power loss in Jun

2、e 2010 due to an operator error at a public utility substation. Deficiencies were noted in the emergency response and notifications, and in the operation of emergency generators and emergency lighting. Process changes are needed to address the significant potential impacts of these deficiencies on p

3、ersonnel, flight assets, and JPL activities. Description of Driving Event: On June 25, 2010, at 7:38 pm, the NASA/Caltech Jet Propulsion Laboratory (JPL) experienced a total electrical power loss due to an operator error at a public utility substation that de-energized the 66kv operating bus (Refere

4、nce (1). The supplied power for the entire JPL Oak Grove campus was unavailable for approximately 3 minutes. The Lab-wide power outage was categorized as a Class D mishap because it impacted the safety of both personnel and flight assets, including: An employee was stranded in an elevator for approx

5、imately 1 hour, 55 minutes, until JPL Protective Services forcibly opened the elevator doors with emergency extraction equipment. This was a new building, and the JPL Fire Department did not have a key to this elevator due to the requirements of a maintenance and response contract with the elevator

6、vendor. In several buildings, emergency lights, emergency generators, and/or uninterruptable power supplies (UPSs) did not function as designed or planned. The affected buildings included mission-critical facilities currently engaged in testing spaceflight hardware. The failures were related to: o U

7、nreliable generators, o Insufficient generator capacity, o A defective relay and an improperly programmed switch for automatic switchover to generator power, o A tripped UPS circuit breaker, o Emergency lighting fixtures that had been removing during remodeling, o Inoperable emergency lighting inver

8、ters, o Emergency lighting inadequate to illuminate the egress path, and o Insufficient capacity of an auxiliary air conditioning system. The procedures for timely incident reporting to JPL executive management were inadequate. Because the personnel list used for notifications by the Critical Hardwa

9、re and Proximate Hazard System (CHAPHS) did not include JPL senior management, the JPL Deputy Director was not notified of the incident until the next morning. The incident caused no personnel injury or illness, and damage was limited to the estimated $16,000 cost for repair to the elevator (Figure

10、1). Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Figure 1. Forcible opening damaged the elevator Reference (1) identifies the root causes of the incident as a lack of robust preventive maintenance for many emergency systems, poor configuration man

11、agement of emergency systems, and a failure to activate emergency notification. References: 1. “JPL Investigation of the Laboratory Power Outage,“ September 9, 2010. 2. NFPA 101: Life Safety Code, National Fire Protection Association. 3. “Lab Wide Power Outage,“ JPL Corrective Action Notice No. 1609

12、 June 29, 2010. Lesson(s) Learned: There is an expectation that facility emergency systems, such as backup generators and emergency lighting, will operate as designed and planned when they are unexpectedly needed. However, because these systems are rarely or never tested under authentic conditions

13、because of the risks such tests pose to sensitive equipment (e.g., computers that require constant air conditioning), the emergency systems may not operate as expected. Recommendation(s): The initial response to the JPL-wide power outage needs to be augmented with process changes that address the si

14、gnificant potential impacts on personnel, flight assets, and JPL activities: 1. For critical facilities with dedicated backup power generators, conduct periodic testing of the automatic switchover from facility power. Periodically assess whether backup generators, UPS systems, and air conditioning n

15、eed to be upgraded. 2. Place exit signs and emergency lighting inverters and batteries on a monthly and annual inspection and test schedule. The inspection of the exit signs and emergency lighting should assess compliance with Reference Provided by IHSNot for ResaleNo reproduction or networking perm

16、itted without license from IHS-,-,-(2). 3. Revise the JPL Multi-Hazard Emergency Response Plan (MERP) to cover generic unscheduled power outages. Update the Everbridge notification tool to include timely notification of JPL senior management of all incidents where there is a potential impact to pers

17、onnel safety or Lab-wide operations. At least once per month, update JPL Protective Services on changes to the CHAPHS user list to ensure that the CHAPHS point-of-contact information used by Everbridge is current. 4. Assure that the JPL Fire Department is trained and equipped to perform emergency ex

18、tractions on all JPL elevators. Assure that facility contracts do not inhibit emergency response. Evidence of Recurrence Control Effectiveness: JPL has opened a corrective action notice (Reference (3) to address the issues raised herein. Documents Related to Lesson: N/A Mission Directorate(s): N/A A

19、dditional Key Phrase(s): Safety and Mission Assurance.Advanced planning of safety systems Safety and Mission Assurance.Maintenance Additional Categories.Emergency Preparedness Additional Categories.Facilities Additional Categories.Fire Protection Additional Categories.Safety & Mission Assurance Additional Info: Project: N/A Approval Info: Approval Date: 2012-02-09 Approval Name: mbell Approval Organization: HQ Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-