REG NASA-LLIS-2017-2009 Lessons Learned Probabilistic Risk Analysis for the NAOMS Phase 2 Data Release.pdf

1、Lessons Learned Entry: 2017Lesson Info:a71 Lesson Number: 2017a71 Lesson Date: 2009-1-22a71 Submitting Organization: GRCa71 Submitted by: Edward Zampinoa71 POC Name: Edward Zampinoa71 POC Email: Edward.J.Zampinoa71 POC Phone: (216)433-2042Subject: Probabilistic Risk Analysis for the NAOMS Phase 2 Da

2、ta Release Abstract: The original NAOMS survey data (called Phase 1) was posted on a publicly accessible website. Congress asked NASA to revise the redaction of the NAOMS survey data so that more information could be released. At the same time, NASA had to revise the NAOMS data presentation (called

3、Phase 2 data) so as to protect (as much as possible) the identity of pilots who were the participants in the survey. One particular step which NASA took in the NAOMS data release project was to perform a risk analysis. The purpose of this risk analysis was to provide a comparison of the probability

4、of identity disclosure between the NAOMS phase 1 and phase 2 data presentation. The NAOMS risk analysis was a very schedule driven task to solve a unique mathematical research problem within several months. Unique, conceptual problems were involved in the development of the mathematical model. This

5、was especially true because there were no standard predecessors for our particular problem. The probabilities for each of the enabling events in the success-tree had a substantial amount of uncertainty ( ie. wide ranges). This is due to the fact that any of the input variables for models to estimate

6、 those probabilities would have large uncertainties. The schedule commitment by NASA to solve a problem should not be made without understanding the nature of the problem and the full resources needed (including time).Description of Driving Event: A. The pressure to demonstrate progress on approxima

7、tely 2-week intervals lead to: 1.Adoption (Premature commitment) to a Bayesian model for Probability of matching records before Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-understanding the entire scope of the problem. 2.Setting aside the Success

8、-Tree development for the Global scenario and re-visiting it after the decision to post had been made. (Finally, there was time to think about it.) 3.Setting aside the preliminary White paper on the Bayesian analysis methodology, resulted in not locking-down critical definitions and understanding of

9、 the Bayesian (MCMC) methodology. Some explanation of the methodology was changing right up to the time of our presentation to the National Academy of Sciences. 4. Calculating probabilities before calculating and examining the protection metrics. 5.Lack of time to derive the relationship between dis

10、tance-Metrics and probability of defeating redaction. B. Consultation with the National Academy of Science should have been the first step. It was nearly the last step. This consultation would have provided a better understanding of the problem and provided recommendations on methodology early on in

11、 the project. C. Pressure to demonstrate progress (early on in the project) to management caused crude results to be presented due to the fact that the unique problem was not understood. More time was needed to mature the process. D.Throughout the project any lower risk values were suspect to the ri

12、sk team. Certain members of the team fell toward a higher degree of belief for the worst- case: the belief that if something can happen, it has happened or will happen. Lesson(s) Learned: Problems such as the NAOMS risk analysis simply require the necessary time to mature the methodology and arrive

13、at a solution. The time is required by the situation itself and is independent of schedule. We realized that premature or crude results were forced in order to meet a schedule. NASA can take the time necessary to provide a higher precision result. (To do the job correctly) The NAOMS risk analysis is

14、 an example of the dominance of schedule compliance over technical requirements. Another lesson learned was that the NAOMS risk team should have challenged the project management to a far greater degree about the schedule issue. We learned an important insight in the course of the project. The proba

15、bilities for each of the enabling events in the success-tree had a substantial amount of uncertainty. ( ie. wide ranges) This is due to the fact that any of the input variables for models to estimate those probabilities would have large uncertainties. In retrospect, our use of advanced statistical m

16、ethods and computational techniques only provided answers which, in the end, are nothing more than rough approximations. They could have been obtained by simple expert opinion.Recommendation(s): The schedule commitment by NASA to solve a problem should not be made without understanding the nature of

17、 the problem and the full resources needed (including time). NASA needs to assign mathematical research problems (like the NAOMS risk analysis) to a mathematical research group Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-instead of immediately dr

18、awing resources away from the reliability discipline simply because statistics and probability are involved.Evidence of Recurrence Control Effectiveness: N/ADocuments Related to Lesson: N/AMission Directorate(s): a71 Sciencea71 Exploration Systemsa71 Aeronautics ResearchAdditional Key Phrase(s): a71

19、 Program Management.Program planning, development, and managementa71 Program Management.Risk managementa71 Additional Categories.Research & Developmenta71 Additional Categories.Risk Management/AssessmentAdditional Info: a71 Project: National Aviation Operational Monitoring System (NAOMS)Approval Info: a71 Approval Date: 2009-02-20a71 Approval Name: mbella71 Approval Organization: HQProvided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-