REG NASA-LLIS-1781-2007 Lessons Learned - Verify the Proper Performance of Critical Backups.pdf

1、Lessons Learned Entry: 1781Lesson Info:a71 Lesson Number: 1781a71 Lesson Date: 2007-04-3a71 Submitting Organization: JPLa71 Submitted by: David Oberhettingera71 POC Name: Gary A. Hamilton, Stephen D Walla71 POC Email: Gary.A.Hamiltonjpl.nasa.gov, Stephen.D.Walljpl.nasa.gova71 POC Phone: 818.354.4405

2、, 818.354.7424Subject: Verify the Proper Performance of Critical Backups Abstract: When a 2004 backup tape for a Cassini instrument testbed was needed in late 2005 to restore important files, it was discovered that a coding error in the backup script had been causing the backup files to be overwritt

3、en and the backup data to be lost. Do not assume that critical data backups are being properly performed, but assure that a computer system administrator familiar with both the script and the backup hardware verifies the actual performance of the backups.Description of Driving Event: In support of t

4、he Cassini spacecrafts mission to map the surface of Titan with its RADAR imaging instrument, the RADAR instrument testbed was used in 2005 to test operational sequences. System backups of these simulations on the testbed were performed periodically using magnetic tape. The testbed Instrument Ground

5、 System Engineer (IGSE) is responsible for the maintenance and upgrade of the testbed hardware for their instrument. The testbed employs a Remote Terminal Interface Unit (RTIU) to connect the RADAR engineering model (EM) to the ground support equipment (GSE) software, sending commands to the EM and

6、returning telemetry. Following a November 2005 failure of the RADAR RTIU hard drive, the RADAR IGSE attempted to restore Instrument Expanded Block (IEB) files to the RTIU from a backup copy. (IEB update files are uplinked to the Cassini spacecraft to support science sequences.) This file restoration

7、 from the backup tape failed, and it was discovered that this backup tape created in 2004 was missing three of the four data partitions (Reference (1). Examining the script that runs Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-the backups, a codi

8、ng error was found that had caused the partitions to be overwritten and the backup data lost. Because of this failure, the sequence IEBs were uplinked to Cassini without the benefit of a testbed run. The data backup failure caused only a minor loss of instrument simulation fidelity, but this minor l

9、oss might have had a major impact in the event of a Cassini spacecraft or instrument flight anomaly. Failures of other more critical NASA ground system or flight system data backups (using the same or different recording media) may occur, and they may escape detection unless a computer failure promp

10、ts a need to retrieve the backups. References: 1. “Cassini RADAR Testbed Has Dead Disk with No Valid Backup Tape,“ Incident/Surprise Anomaly (ISA) Report No. Z87842, Jet Propulsion Laboratory, December 1, 2005.Lesson(s) Learned: Just that backup tapes are seen to move, or backup scripts are run for

11、a lengthy period of time, should not be construed as verifying that data backups are properly being performed.Recommendation(s): Assure that the actual performance of critical backups is verified by a computer system administrator familiar with both the script and the backup hardware.Evidence of Rec

12、urrence Control Effectiveness: JPL will reference this lesson learned as additional rationale and guidance supporting Paragraph 4.4.4.4 (?Information System Design: Commanding and Sequencing? Assured Commanding?) in the Jet Propulsion Laboratory standard ?Design, Verification/Validation & Ops Princi

13、ples for Flight Systems (Design Principles), Rev. 3,? JPL DocID 43913, December 11, 2006.Documents Related to Lesson: N/AMission Directorate(s): a71 Aeronautics Researcha71 Exploration Systemsa71 Sciencea71 Space OperationsProvided by IHSNot for ResaleNo reproduction or networking permitted without

14、license from IHS-,-,-Additional Key Phrase(s): N/A Additional Info: a71 Project: Cassinia71 Year of Occurrence: 2005Approval Info: a71 Approval Date: 2007-07-13a71 Approval Name: ghendersona71 Approval Organization: HQProvided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-