REG NASA-LLIS-2216-2009 Lessons Learned - Ban Nonessential Infrastructure Support Services That Could Disrupt Critical Mission Operations.pdf

1、Lessons Learned Entry: 2216Lesson Info:a71 Lesson Number: 2216a71 Lesson Date: 2009-08-18a71 Submitting Organization: JPLa71 Submitted by: David Oberhettingera71 POC Name: Joseph Guinn; Ronald D. Sharpa71 POC Email: Joseph.R.Guinnjpl.nasa.gov; Ronald.D.Sharp-104583jpl.nasa.gova71 POC Phone: 818-354-

2、0425 (Guinn); 818-393-7919 (Sharp)Subject: Ban Nonessential Infrastructure Support Services That Could Disrupt Critical Mission Operations Abstract: Contractors arrived at the Phoenix Navigation Mission Support Area during a mission-critical Mars Phoenix project event, and they continued to perform

3、disruptive repairs to a badge reader despite a declared facility freeze. Establish and maintain institutional procedures for coordination of nonessential infrastructure services with scheduled configuration control freezes instituted at the request of flight projects during critical mission operatio

4、ns.Description of Driving Event: Eight days before the May 2008 arrival of Mars Phoenix Lander at the north polar region of Mars, the Phoenix Project at the NASA/Caltech Jet Propulsion Laboratory (JPL) was preparing a final trajectory correction maneuver (TCM-5) to guide the spacecraft on its naviga

5、ted course. A final TCM is a mission-critical event in which an error, or non-performance of the maneuver, can result in mission loss (Reference (1). Due to the criticality of this spacecraft command sequence and the limited time available to prepare it, the Phoenix Mission Manager imposed a Level 2

6、 facility freeze for the Phoenix Navigation Mission Support Area (MSA) on the 6th floor of Building 264. A Level 1 configuration control freeze is requested by a flight project, and it restricts changes to the configuration of multiple JPL facilities during a critical event. In contrast, the restric

7、tions associated with a Level 2 freeze are limited to a specific facility or service. The project notifies a facility manager of the Level 2 freeze, and a printed tag is placed at the facility entrance. Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,

8、-Access to the MSA is restricted by an electronic badge reader that operates the door lock. An upgrade to the badge reader system in Building 264 was scheduled for May 2008, which happened to be on a Saturday during the Level 2 freeze for TCM-5 sequence development (Reference (2). Contractors arrive

9、d during the freeze, and started work on the reader even though the Phoenix Mission Manager explained that there was a critical event in progress. The upgrade work involved removal of ceiling tiles directly above the MSA and work in the crawl space without regard for the computer equipment directly

10、below. No drop cloths were placed over the sensitive computer equipment being used to prepare the TCM, and the contractor intrusion disrupted the work of the navigation team. A phone call to JPL Security halted the badge reader work for the day, with the badge readers restored to the previous config

11、uration. The JPL practice is for infrastructure organizations such as JPL computer and network services to consult the Interplanetary Network Directorate (IND) Critical Activities Schedule and defer planned activities in accordance with any scheduled freezes (Reference (3). IND also holds a weekly p

12、lanning meeting to discuss scheduled freezes and other concurrent activities. However, it is the JPL facilities and infrastructure services that have the responsibility to consult these resources to ensure that their plans do not conflict with critical project activities. References: 1. “Inadequate

13、MCO Contingency Maneuver Planning,“ NASA Lesson Learned No. 0916, NASA Engineering Network, May 4, 2000. http:/www.nasa.gov/offices/oce/llis/imported_content/lesson_0916.html2. “Badge Readers Being Changed On The 6th Floor During Level 2 Freeze,“ JPL Incident Surprise Anomaly (ISA) No. Z92597, May 1

14、7, 2008.3. IND Critical Activities Schedule, http:/dsnprocess.jpl.nasa.gov/icas/Lesson(s) Learned: Although flight projects are careful to assess risks to mission operations such as unplanned facility power interruptions, the impact on flight operations of scheduled facility services like visitor to

15、urs, janitorial service, and facility and equipment maintenance may not be as obvious. Had JPL Security known that a facility freeze was in effect, critical Phoenix project activities in preparation for the Mars landing would not have been disrupted by this infrastructure service.Recommendation(s):

16、Establish and implement institutional procedures for coordination of nonessential infrastructure services with scheduled configuration control freezes that have been instituted at the request of flight projects during critical mission operations.Evidence of Recurrence Control Effectiveness: Provided

17、 by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-JPL has referenced this lesson learned as additional rationale and guidance supporting Paragraph 6.3.2.3 (“Engineering Practices: Mission Operations - Process and Procedure“) in the Jet Propulsion Laboratory st

18、andard “Flight Project Practices, Rev. 7,“ JPL DocID 58032, September 30, 2008.Documents Related to Lesson: N/AMission Directorate(s): a71 Space OperationsAdditional Key Phrase(s): a71 0.a71 0.a71 0.a71 0a71 1.Facilitiesa71 1.Maintenancea71 1.Configuration Change Controla71 1.Logistics and maintenan

19、cea71 1.Ground support systemsa71 0a71 0.a71 1.Flight OperationsAdditional Info: a71 Project: Mars PhoenixApproval Info: a71 Approval Date: 2010-01-15a71 Approval Name: mbella71 Approval Organization: HQProvided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-