1、Lessons Learned Entry: 2016Lesson Info:a71 Lesson Number: 2016a71 Lesson Date: 2008-11-20a71 Submitting Organization: GRCa71 Submitted by: Don Hildermana71 POC Name: Don Hildermana71 POC Email: Don.R.Hildermannasa.gova71 POC Phone: 216-433-3538Subject: Record Management and Quality Assurance Lesson
2、Learn Abstract: Between February 1998 and March 2005, NASA conducted a study called National Aeronautics Operational Monitoring System, also known as NAOMS. NAOMS is a survey-based data collection system designed to collect aviation safety data from general aviation and air carrier pilots. NAOMS was
3、 designed to quantify aviation safety events and other safety-relevant phenomena. Because incidents posing a threat to aviation safety are relatively rare events, useful analysis requires the acquisition of a large, statistically meaningful database that is representative of the system. These data c
4、an be used to identify system-wide trends and establish performance measures with emphasis on tracking the effects of safety interventions (new procedures, technologies, and training) on national aviation safety levels. The NAOMS project selected a contractor to develop a survey methodology to acqui
5、re such data. It was recognized that the type of data, which was sought, information on incidents, infractions, and mistakes potentially affecting aviation safety, carries with it the potential for significant risk for the person reporting such information. Prior to interviewing pilots for the NAOMS
6、 survey, NASA promised that, those pilots who were willing to participate in the survey could be assured that their identity would be protected. The survey-based project team offered anonymity to those taking part in the survey, and one of the contractors responsibilities was to anonymize the datase
7、t. Data collection using the methodology began in April 2001 and ended in December 2004, during which time the project team conducted approximately 24,000 surveys of commercial airline pilots and approximately 5,000 general aviation pilots. Research conducted in the NAOMS project, including the surv
8、ey methodology, has not been peer-reviewed and should not be viewed or Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-considered as having been validated. Description of Driving Event: Because there was intense public and Congressional interest in t
9、he NAOMS data, NASA promised the Congress initial release of data (called phase 1) by the end of 2007. In a memorandum dated November 19, 2007, the NASA administrator directed the Chief of the Office of Safety and Mission Assurance to lead an effort to release as much NAOMS survey information as pos
10、sible by the end of 2007. The task was executed soon thereafter with the initial posting of NAOMS information on the NASA website on December 31, 2007. Phase 1 was completed on February 6, 2008, with a more comprehensive web posting of NAOMS information. Some people and organizations outside of NASA
11、 have said that the necessary data redaction for Phase 1 was overly conservative, that more data could have been released. After the Phase 1 postings, NASA initiated another review of all the NAOMS survey responses in order to allow for further release of information. A project team was selected to
12、implement the recommendation from the NASA Information Release Advisory Panel (2008) relating to an additional release of NAOMS Phase 1 information by the end of the year (2008). This effort is referred to as Phase 2 of the NAOMS information release. The objective of the NAOMS Phase 2 information re
13、lease was to redact the raw NAOMS survey responses using a redaction release strategy that: Releases the maximum amount of survey information, but - Does not release commercial confidential information -Meets NASAs obligation for survey response participant confidentiality as defined by the NAOMS In
14、formation Release Advisory Panel (2008) - Minimizes the threat to participant anonymity NASA intended to protect pilot participant identity and never planned to publish NAOMS raw data. NASA had to engage in an optimization process by releasing as much of the NAOMS data as possible, yet still maintai
15、n the anonymity and confidentiality of the pilots who were the participants in the survey (Phase 2 data). In retrospect, the NAOMS data release project required extensive efforts to release phase 2 data. These included, planning, redaction, quality assurance, risk analysis, and validation efforts. T
16、he cost of NAOMS phase 2 was approximately $1.1M. NAOMS Phase 2 Release Executive Summary dated September 30, 2008 states that NAOMS survey responses and the methodology used to acquire them have not been peer-reviewed. In addition, no product of the NAOMS project, including the survey methodology,
17、the survey responses, and analysis of the responses, should be viewed or considered as having been validated. Listed below are examples of inconsistencies from the survey responses: Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Example #1: Some “Fo
18、kker“ aircraft identified as “Poker“ aircraft Example #2: Boeing 737-300 aircraft identified by the following descriptions: B 737-300 B/737-300 B-737-300 B 737 300 Bowing 737 300 Many more combinations of example #2 exist Example #3: There are two subsets with the title Weather Diversion Example #4:
19、 Some free text truncated after 252 characters. Example #5: There are 25,763 records in the .mdb file and 25,105 records in the SAS files. Example #6: Over 400 AC survey responses misfiled in the GA surveys. Example #7: Total number of records, as compared to Phase 1, do not match Example #8: Joint
20、Implementation Measurement Data Analysis Team missing free text responses Example #9: Not all Phase 1 data was supplied for Phase II Lesson(s) Learned: The Lesson Learned from NAOMS is to ensure all future survey processes and procedures including data redaction and strategy are properly implemented
21、, maintained and controlled per NASA Quality Assurance Program NPD 8730.5 (http:/nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPD&c=8730&s=5). Survey response inconsistencies cast doubt on the validity of the dataset. Accuracy and completeness cannot be retroactively peer-reviewed into the dataset. At this
22、 point, the Phase 1 survey product exists in its final state.Recommendation(s): Projects should perform record management and project quality assurance per NPD 8730.5. In retrospect, the NAOMS data release project required extensive efforts to release phase 2 data. These included, planning, redactio
23、n, quality assurance, risk analysis, and validation efforts. Project Plans controlled by Project Managers, should identify the deliverables of the project.Evidence of Recurrence Control Effectiveness: N/ADocuments Related to Lesson: NASA Quality Assurance Program NPD 8730.5Mission Directorate(s): Pr
24、ovided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-a71 Aeronautics ResearchAdditional Key Phrase(s): a71 Program Management.Acquisition / procurement strategy and planninga71 Program Management.Business processesa71 Program Management.Education and public
25、 engagementa71 Program Management.Organizational Planninga71 Program Management.Role of civil service technical staff versus contractor staffa71 Safety and Mission Assurance.Qualitya71 Additional Categories.Information Technology/SystemsAdditional Info: a71 Project: National Aeronautics Operational Monitoring SystemApproval Info: a71 Approval Date: 2009-03-24a71 Approval Name: mbella71 Approval Organization: HQProvided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
