1、 NASA HANDBOOK Estimation and Evaluation of Measurement Decision Risk NASA Measurement Quality Assurance Handbook ANNEX 4 Measurement System Identification: Metric July 2010 National Aeronautics and Space Administration Washington DC 20546 NASA-HDBK-8739.19-4Approved: 2010-07-13 Baseline APPROVED FO
2、R PUBLIC RELEASE DISTRIBUTION UNLIMITED Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ii This page intentionally left blank. Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-iii DOCUMENT HISTORY LOG
3、Status Document Revision Approval Date Description Baseline 2009-07-13 Initial Release (JWL4)This document is subject to reviews per Office of Management and Budget Circular A-119, Federal Participation in the Development and Use of Voluntary Standards (02/10/1998) and NPD 8070.6, Technical Standard
4、s (Paragraph 1.k). Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-iv This page intentionally left blank. Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Provided by IHSNot for ResaleNo reproduction o
5、r networking permitted without license from IHS-,-,-vi This page intentionally left blank. Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-vii TABLE OF CONTENTS FOREWORD V TABLE OF CONTENTS VII LIST OF FIGURES XIV LIST OF TABLES XV PREFACE XVI ACKNOW
6、LEDGEMENTS XVII PURPOSE AND SCOPE XVII EXECUTIVE SUMMARY XVIII CHAPTER 1: INTRODUCTION 1 1.1 WHY COMPUTE RISKS? 1 1.1.1 Measurement Quality Metrics 1 1.1.2 Economics 2 1.2 FACTORS AFFECTING RISKS 2 1.3 READERS GUIDE 3 1.4 TERMS AND DEFINITIONS 6 1.5 ACRONYMS 16 CHAPTER 2: UNCERTAINTY ANALYSIS OVERVI
7、EW 18 2.1 UNCERTAINTY ANALYSIS AND RISK MANAGEMENT 18 2.1.1 Measurement Decision Risk 18 2.1.2 Parts Conformance 18 2.1.3 Statistical Tolerancing 19 2.2 UNCERTAINTY ANALYSIS FUNDAMENTALS 19 2.2.1 Preliminaries 19 2.2.2 The Basic Error Model 19 2.2.3 Measurement Error Sources 20 2.2.4 Error and Uncer
8、tainty 21 2.2.5 Procedures for Obtaining an Uncertainty Estimate for an Error Source 22 2.2.6 Degrees of Freedom for Combined Estimates 23 2.2.7 Expanded Uncertainty 24 2.3 MULTIVARIATE UNCERTAINTY ANALYSIS 24 2.3.1 Error Modeling 25 2.3.2 Computing System Uncertainty 25 Provided by IHSNot for Resal
9、eNo reproduction or networking permitted without license from IHS-,-,-viii 2.3.3 Process Uncertainties 26 2.3.4 Cross-Correlations 26 2.4 UNCERTAINTY ANALYSIS SCENARIOS 26 2.5 INTERPRETING AND APPLYING EQUIPMENT SPECIFICATIONS 27 2.6 UNCERTAINTY ANALYSIS EXAMPLES 27 CHAPTER 3: MEASUREMENT DECISION R
10、ISK ANALYSIS BASICS 28 3.1 PRELIMINARIES 28 3.1.1 Definition of Probability 28 3.1.2 Joint Probability 29 3.1.3 Conditional Probability 30 3.2 FALSE ACCEPT RISK 31 3.2.1 Unconditional False Accept Risk 31 3.2.2 Conditional False Accept Risk 32 3.2.3 UFAR and CFAR 32 3.3 FALSE REJECT RISK 32 3.4 RISK
11、 ANALYSIS ALTERNATIVES 33 3.4.1 Process-Level Analysis 33 3.4.2 Bench-Level Analysis 33 3.5 THE 4:1 TUR ALTERNATIVE 34 3.5.1 The Z540.3 Definition 34 3.5.2 A Critique of the 4:1 Requirement 35 3.6 RECOMMENDATIONS 35 CHAPTER 4: COMPUTING RISK 37 4.1 CALIBRATION SCENARIOS 37 4.1.1 Risk Variables 37 4.
12、1.2 Probability Relations 38 4.1.3 Calibration Scenario Results 39 4.2 THE CLASSICAL METHOD 40 4.2.1 Measurement Decision Risk Recap 40 4.2.2 Estimating Risk 41 4.3 THE BAYESIAN METHOD 46 4.3.1 Risk Analysis for a Measured Variable 46 4.3.2 A priori Knowledge 46 4.3.3 Post-Test Knowledge 47 4.3.4 Bi
13、as Estimates 47 4.3.5 UUT Attribute In-Tolerance Probability 49 4.3.6 MTE Attribute In-Tolerance Probability 49 4.3.7 Bayesian False Accept Risk 49 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ix 4.4 THE CONFIDENCE LEVEL METHOD 50 4.4.1 Confidence
14、 Level Estimation 50 4.4.2 Applying Confidence Level Estimates 51 4.4.3 UUT Attribute Adjustment 51 CHAPTER 5: COMPENSATING MEASURES 52 5.1 INCREASING UUT IN-TOLERANCE PROBABILITY 52 5.2 REDUCING MEASUREMENT UNCERTAINTY 53 5.2.1 Pareto Analysis 53 5.2.2 Multiple Independent Measurements 54 5.2.3 Seq
15、uential Testing 55 5.3 USING GUARDBANDS 56 5.3.1 Guardband Multipliers 56 5.3.2 Test Guardband Limits 57 5.3.3 Reporting Guardband Limits 60 5.4 BAYESIAN GUARDBANDS 61 5.5 CONFIDENCE LEVEL GUARDBANDS 63 5.6 MINIMIZING COSTS 64 5.6.1 A Simplified Model 64 REFERENCES 67 APPENDIX A - MEASUREMENT UNCERT
16、AINTY ANALYSIS 70 A.1 APPENDIX A NOMENCLATURE 70 A.2 UNCERTAINTY ANALYSIS FUNDAMENTALS 71 A.2.1 Preliminaries 71 A.2.2 Error Distributions 72 A.2.3 Error and Uncertainty 78 A.2.4 Combining Uncertainties 80 A.3 ESTIMATING UNCERTAINTY 82 A.3.1 The Nature of Uncertainty Estimates 82 A.3.2 Computing Met
17、hods 83 A.3.3 Categories of Estimates 84 A.4 MULTIVARIATE UNCERTAINTY ANALYSIS 89 A.4.1 Error Modeling 89 A.4.2 Computing System Uncertainty 89 A.5 EXPANDED UNCERTAINTY 91 A.6 TEST AND CALIBRATION SCENARIOS 91 A.6.1 Basic Notation 92 A.6.2 Measurement Uncertainties 93 A.6.3 Measurement Error Sources
18、 93 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-x A.6.4 Calibration Error and Measurement Error 95 A.6.5 UUT Attribute Bias 95 A.6.6 MTE Bias 96 A.7 CALIBRATION SCENARIOS 97 A.7.1 Scenario 1: The MTE Measures the UUT Attribute Value 97 A.7.2 Scen
19、ario 2: The UUT Measures the MTE Attribute Value 100 A.7.3 Scenario 3: MTE and UUT Output Comparison (Comparator Scenario) 101 A.7.4 Scenario 4: MTE and UUT Measure a Common Attribute 104 A.8 UNCERTAINTY ANALYSIS SUMMARY 105 Uncertainty Analysis Examples 106 APPENDIX A REFERENCES 106 APPENDIX B - TE
20、ST AND CALIBRATION QUALITY METRICS 108 B.1 INTRODUCTION 108 B.2 APPENDIX B NOMENCLATURE 108 B.3 DISCUSSION 109 B.3.1 Unconditional False Accept Risk 109 B.3.2 Conditional False Accept Risk 109 B.3.3 False Reject Risk 111 B.3.4 Other Metrics 112 B.4 CONTROLLING RISKS WITH GUARDBANDS 113 B.4.1 Guardba
21、nd Risk Relations 113 B.4.2 Establishing Risk-Based Guardbands 114 B.5 COMPUTING PROBABILITIES 115 B.5.1 The Basic Set of Integrals 115 B.6 MEASUREMENT QUALITY METRICS ESTIMATION PROCEDURE 117 B.7 CONCLUSION 118 APPENDIX B REFERENCES 118 APPENDIX C: INTRODUCTION TO BAYESIAN MEASUREMENT DECISION RISK
22、 ANALYSIS 119 C.1 APPLICATION TO MEASUREMENT DECISION RISK ANALYSIS 121 C.1.1 Conditional False Accept Risk 121 C.1.2 Unconditional False Accept Risk 122 C.1.3 False Reject Risk 122 C.1.4 Conditional False Reject Risk 123 C.2 APPLICATION OF BAYES THEOREM WITH MEASURED VALUES 123 C.2.1 CFAR Revisited
23、 125 C.2.2 CFAR After Adjustment 127 C.2.3 Bench-Level Implementation of the Method 128 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-xi APPENDIX C REFERENCES 128 APPENDIX D: DERIVATION OF KEY BAYESIAN EXPRESSIONS 129 D.1 INTRODUCTION 129 D.2 COMPU
24、TATION OF IN-TOLERANCE PROBABILITIES 129 D.2.1 UUT In-Tolerance Probability 129 D.2.2 MTE In-Tolerance Probability 130 D.3 COMPUTATION OF VARIANCES 131 D.3.1 Variance in Instrument Bias 131 D.3.2 Treatment of Multiple Measurements 132 D.4. EXAMPLE 133 D.5 DERIVATION OF EQ. (D-3) 136 D.6 ESTIMATION O
25、F BIASES 138 D.7 BIAS CONFIDENCE LIMITS 139 APPENDIX D REFERENCES 143 APPENDIX E: TRUE VS. REPORTED PROBABILITIES 144 E.1 APPENDIX E NOMENCLATURE 144 E.2 PROBABILITY RELATIONS 144 E.2.1 False Accept Risk 144 E.2.2 False Reject Risk 144 E.3 TRUE VS. REPORTED 145 E.4 COMPUTING ROBSAND RTRUE145 E.4.1 N
26、ormally Distributed UUT Attribute Biases 146 E.4.2 Uniformly Distributed Attribute Biases 148 E.4.3 Triangularly Distributed Attribute Biases 149 E.4.4 Quadratically Distributed Attribute Biases 150 E.4.5 Cosine Distributed Attribute Biases 152 E.4.6 U-Distributed Attribute Biases 153 E.4.7 Lognorma
27、lly Distributed Attribute Biases 154 E.5 EXAMPLES 157 APPENDIX E REFERENCES 157 APPENDIX F: USEFUL NUMERICAL ALGORITHMS 159 F.1 BISECTION ROUTINE 159 F.1.1 Function Root 159 F.1.2 Function Bracket 160 F.2 GAUSS QUADRATURE INTEGRATION 161 F.2.1 Subroutine GaussQuadrature 161 Provided by IHSNot for Re
28、saleNo reproduction or networking permitted without license from IHS-,-,-xii F.2.2 Subroutine GaussLegendre 162 APPENDIX F REFERENCES 163 APPENDIX G: CALIBRATION FEEDBACK ANALYSIS 164 G.1 APPENDIX G NOMENCLATURE 164 G.2 ESTIMATING RISK FROM A MEASURED UUT VALUE 165 G.3 EXAMPLE 166 G.4 CASES WITH UNK
29、NOWN X 167 G.5 ESTIMATING MTE BIAS AND BIAS UNCERTAINTY AT T = 0 168 G.6 ESTIMATING MTE BIAS UNCERTAINTY AT T = T 169 G.6.1 Fundamental Postulate 169 G.6.2 Estimating Uncertainty Growth 169 G.7 ESTIMATING TEST PROCESS UNCERTAINTY 170 G.7.1 UUT Bias Uncertainty 170 G.7.2 Test Process Uncertainty 170
30、APPENDIX G REFERENCES 172 APPENDIX H: RISK-BASED END OF PERIOD RELIABILITY TARGETS 173 H.1 BACKGROUND 173 H.1.1 General Methodology 173 H.1.2 Application 173 H.2 APPENDIX H NOTATION 173 H.3 ASSUMPTIONS 174 H.3.1 Normal Distributions with Zero Population Bias 174 H.3.2 No Guardbands 174 H.3.3 False A
31、ccept Risk Definition 174 H.3.4 Single-Parameter Reliability Functions 174 H.3.5 Zero Measurement Process Uncertainty 175 H.3.6 Use of “True” Reliabilities 175 H.4 FALSE ACCEPT RISK COMPUTATION 175 H.5 RELIABILITY DEPENDENCE 175 H.6 VARIANCE IN THE FALSE ACCEPT RISK 177 H.7 VARIANCE IN THE RELIABILI
32、TY FUNCTIONS 178 H.8 FALSE ACCEPT RISK CONFIDENCE LIMIT 179 H.8.1 Developing the Confidence Limit 179 H.8.2 Implementing the Solution 180 H.9 A NOTE OF CAUTION 180 H.10 AN ALTERNATIVE METHOD 181 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-xiii H.
33、10.1 Developing a Binomial Confidence Limit for Rx182 APPENDIX H REFERENCES 182 APPENDIX I: SET THEORY NOTATION FOR RISK ANALYSIS 184 I.1 BASIC NOTATION 184 I.2 ADDITIONAL NOTATION 184 APPENDIX J: POST-TEST RISK ANALYSIS 187 J.1 STRESS RESPONSE 187 J.1.1 Shipping and Handling 187 J.1.2 Usage Environ
34、ment 188 J.2 UNCERTAINTY GROWTH 189 J.2.1 Attributes Data Analysis 190 J.2.2 Variables Data Analysis 193 J.3 IMPLEMENTATION 197 APPENDIX K: DERIVATION OF THE DEGREES OF FREEDOM EQUATION 199 K.1 TYPE A DEGREES OF FREEDOM 199 K.1.1 Random Error 199 K.2 TYPE B DEGREES OF FREEDOM 199 K.2.1 Methodology 2
35、00 K.2.2 Comparison with Eq. G3 of the GUM 202 K.2.3 Estimating uLand up203 K.2.4 Degrees of Freedom for Combined Estimates 204 APPENDIX K REFERENCES 207 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-xiv List of Figures Figure 1. The Test and Calib
36、ration Support Hierarchy. xix Figure 2-1. The Variance Addition Rule for Measurement Errors 22 Figure 4-1. UUT/MTE Hierarchy 44 Figure 5-1. Risk vs. UUT Attribute In-Tolerance Probability. 52 Figure 5-2. Risk vs. Measurement Process Standard Uncertainty. 53 Figure 5-3. Pareto Chart. 54 Figure 5-4. S
37、equential Testing involving n Test Steps. 56 Figure 5-5. Test Guardband Limits. 58 Figure 5-6. Reporting Guardband Limits. 61 Figure A-1. The Normal Error Distribution. 74 Figure A-2. The Lognormal Distribution. 74 Figure A-3. The Uniform Error Distribution. 75 Figure A-4. The Triangular Error Distr
38、ibution. 75 Figure A-5. The Quadratic Error Distribution. 76 Figure A-6. The Cosine Error Distribution 77 Figure A-7. The U Distribution 77 Figure A-8. Scenario 1 97 Figure A-9. Scenario 2 100 Figure A-10. Scenario 3 102 Figure A-11. Scenario 4 104 Figure D-1. Proficiency Audit Example 134 Figure E-
39、1. The Normal Distribution. 147 Figure E-2. The Uniform Distribution 148 Figure E-3. The Triangular Distribution 149 Figure E-4. The Quadratic Distribution 151 Figure E-5. The Cosine Distribution 152 Figure E-6. The U Distribution 153 Figure E-7. The Right-Handed Lognormal Distribution 154 Figure E-
40、8. The Left-Handed Lognormal Distribution 155 Figure G-1. The Calibration Feedback Loop. 164 Figure G-2. Measurement Reliability vs. Time 166 Figure J-1. Probability Density Function for UUT Attribute Bias 190 Figure J-2. Measurement Reliability vs. Time 191 Figure J-3. First Degree Regression Fit t
41、o the Data of Table J-2 196 Figure J-4. Second Degree Regression Fit to the Data of Table J-2 197 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-xv List of Tables Table 4-1. Risk Variables Nomenclature. 38 Table 4-2. Risk Computation Nomenclature. 3
42、9 Table 4-3. Correspondence between the Risk Analysis Nomenclature and the Probability Functions of Chapter 3. 39 Table 4-4. Measurement Result Definitions for the Calibration Scenarios. 40 Table 4-5. Risk Analysis Probability Density Functions 41 Table 5-1. Uncertainty k-Factors for a 1% False Acce
43、pt Risk. 58 Table 5-2. Risks Associated with a k-Factor of 2. 59 Table 5-3. Bayesian Guardband Limits. 62 Table 5-4. Confidence Level Guardband Limits. 64 Table 5-5. Variables Used in the Simplified Cost Model. 65 Table A-1. Appendix A Nomenclature. 70 Table A-2. Basic Notation. 92 Table B-1. Measur
44、ement Quality Metrics Variables. 108 Table C-1. Bayes Theorem Results for Different Values of p(s). 121 Table D-1. Proficiency Audit Results Arranged for Bayesian Analysis. 134 Table E-1. Variables Used to Estimate True vs. Reported Percent In-Tolerance. 144 Table E-2. Selection Rules for UUT Attrib
45、ute Bias Distributions. 146 Table E-3. Parameters of the Lognormal Distribution. 156 Table E-4. True vs. Reported % In-Tolerance for a Reported 95% In-Tolerance. 157 Table E-5. Reported vs. True % In-Tolerance for a True 95% In-Tolerance. 157 Table G-1. Variables Used in Calibration Feedback Analysi
46、s. 164 Table H-1. Variables Used in Estimating Risk-Based Reliability Targets. 173 Table J-1. Example Service History Data 195 Table J-2. Conditioned Variables Data Sample. 195 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-xvi Preface Measurement d
47、ata are used to make decisions that impact all areas of technology. Whether measurements support research, design, production, or maintenance, ensuring that the data supports the decision is crucial. The quality of measurement data affects the consequences that follow measurement-based decisions. Ne
48、gative consequences from measurement results can range from wasted resources to loss of mission or life. Historically, to ensure measurement data supported decisions, selection and calibration of MTE was the emphasis for measurement quality assurance. With ever increasing technology requirements, the emphasis needs shift to understanding and controlling measurement decision risk in all areas of technology. AS9100C defines risk as, “An undesirable situat
