1、A Framework for Early Reliability Assessment,Bojan Cukic, Erdogan Gunel, Harshinder Singh, Lan Guo, Dejan Desovski West Virginia UniversityCarol Smidts, Ming Li University of Maryland,(WVU UI: Integrating Formal Methods and Testing in a Quantitative Software Reliability Assessment Framework 2003),2,
2、Overview,Introduction and Motivation. Software Reliability Corroboration Approach. Case Studies. Applying Dempster Shafer Inference to NASA datasets. Summary and Further Work.,3,Introduction,Quantification of the effects of V&V activities is always desirable. Is software reliability quantification p
3、ractical for safety/mission critical systems? Time and cost considerations may limit the appeal. Reliability growth applicable only to integration testing, the tail end of V&V. Estimation of operational usage profiles is rare.,4,Is SRE Impractical for NASA IV&V?,Most IV&V techniques are qualitative
4、in nature. Mature software reliability estimation methods based exclusively on testing. Can IV&V techniques be utilized for reliability? Requirements readings, inspections, problem reports and tracking, unit level tests,Req Design Code Test (Verification & Validation)Unit Integration Acceptance,Life
5、 cycle long IV&V Implementation,Traditional Software Reliability Assessment Techniques,5,Contribution,Develop software reliability assessment methods that build on: Stable and mature development environments. Lifecycle long IV&V activities. Utilize all relevant available information Static (SIAT), d
6、ynamic, requirements problems, severities. Qualitative (formal and informal) IV&V methods. Strengthening the case for IV&V across NASA enterprise. Accurate, stable reliability measurement and tracking. Available throughout the development lifecycle.,6,Assessment vs. Corroboration,Current thinking So
7、ftware reliability “tested into” the product through the integration and acceptance testing. Our thinking Why “waste” the results of all the qualitative IV&V activities. Testing should corroborate that the life-cycle long IV&V techniques are giving the “usual” results, that the project follows usual
8、 quality patterns.,7,Approach,Software quality Measures (SQM),Reliability Prediction Systems (RPS),RPS Combination Techniques,SW Reliability Corroboration Testing,SQM1,SQM3,SQM2,SQM4,SQM6,SQM5,SQMi,SQMj,RPS1,RPS2,RPSk,RPSm,. . .,RPS Combination (Experience, Learning, Dempster-Schafer),BHT software r
9、eliability corroboration,Null Hypothesis, H0 Alternative Hypothesis, Ha,Software Development Lifecycle,Trustworthy Software Reliability Measure,8,Software Quality Measures (roots),The following ones used in experiments. Lines of code Defect density No defect that remain unresolved after testing, div
10、ided by the LOC. Test coverage LOCtested / LOCtotal. Requirements traceability RT= #_requirements_implemented/#_original_requirements. Function points . . . In principle, any measures available could/should be taken into account. Defining appropriate Reliability Prediction Systems (RPS).,9,Reliabili
11、ty Prediction Systems,An RPS is a complete set of measures from which software reliability can be predicted. The bridge between an RPS and software reliability is a MODEL. Therefore, select (and collect) those measures that have the highest relevance to reliability. Relevance to reliability ranked f
12、rom expert opinions Smidts 2002.,10,RPS for Test Coverage,11,Approach,Software quality Measures (SQM),Reliability Prediction Systems (RPS),RPS Combination Techniques,SW Reliability Corroboration Testing,SQM1,SQM3,SQM2,SQM4,SQM6,SQM5,SQMi,SQMj,RPS1,RPS2,RPSk,RPSm,. . .,RPS Combination (Experience, Le
13、arning, Dempster-Schafer),BHT software reliability corroboration,Null Hypothesis, H0 Alternative Hypothesis, Ha,Software Development Lifecycle,Software Reliability Measure,12,Reliability “worthiness” of different RPS,32 measures ranked by five experts,13,Combining RPS,Weighted sums used in initial e
14、xperiments. RPS results weighted by the expert opinion index. Removing inherent dependencies/correlations. Dempster-Shafer (D-S) belief networks approach developed. Network automatically built from datasets by the Induction Algorithm. Existence of suitable NASA datasets? Pursuing leads with several
15、CMM level 5 companies.,14,Approach,Software quality Measures (SQM),Reliability Prediction Systems (RPS),RPS Combination Techniques,SW Reliability Corroboration Testing,SQM1,SQM3,SQM2,SQM4,SQM6,SQM5,SQMi,SQMj,RPS1,RPS2,RPSk,RPSm,. . .,RPS Combination (Experience, Learning, Dempster-Schafer),BHT softw
16、are reliability corroboration,Null Hypothesis, H0 Alternative Hypothesis, Ha,Software Development Lifecycle,Software Reliability Prediction,15,Bayesian Inference,Allows for the inclusion of imprecise (subjective) probability of failure. Subjective estimate reflects beliefs. Hypothesis on the event o
17、ccurrence probability is combined with new evidence, which may change the degree of belief.,16,Bayesian Hypothesis Testing (BHT),Hypothesized reliability H0 comes as a result of RPS combination. Based on the level of (in)experience, the degree of belief assigned: P(H0). Corroboration testing now loo
18、ks for the evidence in favor of the hypothesized reliability.Ho : q qo alternative hypothesis.,17,The number of corroboration tests according to BHT theory,18,Controlled Experiments,Two independently developed versions of PACS (smart card based access control). Controlled requirements document (NSA
19、specs).,19,RPS Experimentation,RPS predictions of system failure rates:,Predicted Failure Rate: 0.084 Actual Failure Rate: 0.09,20,Reliability Corroboration,Accurate predictors appear adequate Low levels of trust in the prediction accuracy. No experience in repeatability at this point in time.,21,“R
20、esearch Side Products”,Significant amount of time spent studying and developing Dempster-Shafer inference networks. “No hope” of demonstrating this work within the scope of integrating RPS results. Availability of suitable datasets. But, some datasets are available. So, use them for D-S demo! Predic
21、ting fault-prone modules in two NASA projects (KC2, JM1) KC2 contains over 3,000 modules, 520 modules of research interest 106 modules have errors, ranging from 1 to 13 414 modules are error free JM1 contains 10,883 modules 2,105 modules have errors, ranging from 1 to 26 8,778 modules are error free
22、 Each dataset contains 21 software metrics, mainly McCabe and Halstead,22,How D-S Networks Work,Combining distinct sources of evidence by the D-S scheme. Building D-S networks by prediction logic. Nodes connected by implication rules. Each implication rule assigned a specific weight. Updating belief
23、 for the corresponding nodes Propagating the updated belief to the neighboring nodes, and throughout the entire network. D-S network can be tuned for a various range of verification requirements.,23,D-S Networks vs. Logistic Regression,KC2,JM1,24,D-S Networks vs. ROCKY,KC2,JM1,25,D-S Networks vs. Se
24、e5,KC2,JM1,26,D-S Networks vs. WEKA,KC2 dataset,27,D-S Networks vs. WEKA,JM1,28,Status and Perspectives,Software reliability corroboration allows: Inclusion of IV&V quality measures and activities into the reliability assessment. A significant reduction in the number of (corroboration) tests. Software reliability of safety/mission critical systems can be assessed with a reasonable effort. Research directions. Further experimentation (data sets, measures, repeatability). Defining RPS based on the “formality” of the IV&V methods.,
