Building Models from Your Software Data.ppt

1、,Building Models from Your Software Data,Brad Clark, Ph.D. Software Metrics, Inc. 16th International Forum on COCOMO and Software Cost Modeling Los Angeles, CA October 23-26, 2001,Tutorial: Building Models,2,Agenda,1:00 - 2:30 PM Tutorial 2:30 - 3:00 PM Break 3:00 - 4:30 PM Tutorial conclusionMiscel

2、laneous Bathrooms Telephones Tutorial Format: Collaborative participation One person talks at a time Keep discussions to the point No attribution End-of-course evaluation,Tutorial: Building Models,3,Directions,Gerontology Auditorium,Parking Structure A,Electrical Engineering Building (Hughes),Comput

3、er Science Building (Salvatori),CSE - 3rd Floor,School of Engineering,McClintock Ave.,West 37th Place,Gate #1,Tutorial: Building Models,4,Tutorial Outline,Purpose A software engineering modeling example Model building steps Mean-based model exercise Regression based model exercise Summary,Tutorial:

4、Building Models,5,The need for models,Models are useful for forecasting, performance analysis, and decision-making WBS is narrowly addressed with current estimation models Strength of cause and effect relationships Impact of decision making: Personnel turnover Establish data requirements (model para

5、meters) Explain assignable causes of variation and their degree of influence Used to validate data Poor data definitions and collection consistency Poor processes that produce the data,Tutorial: Building Models,6,WBS Help,Software Cost Estimation Models,Tutorial: Building Models,7,Decision Impact An

6、alysis,Do we give the team an incentive to stay or do we look for new hirers?,Estimated PM = 2.94 * KSLOC * PCON,PCON Effect on PM3% / yr 0.81 0.0%6% / yr 0.90 +11.0% 12% / yr 1.00 +23.5% 24% / yr 1.12 +38.0% 48% / yr 1.29 +59.0%,Estimated Person Months for a 100 KSLOC project with 3% PCON = 238 PMS

7、ame project with 12% PCON = 294 PM (23.5% increase)If the burdened labor rate is $10,000/PM, the cost increase is $235,000/PM,Why not give everyone a financial incentive to stay?,PM: Person Months PCON: Personnel Continuity,Tutorial: Building Models,8,Data validation -1,Check for internal consistenc

8、y Be suspicious of “perfect” data Understand reason for outliers Check data relationships Effort and size Effort and schedule Size and defects Effort and defects,Tutorial: Building Models,9,Data validation -2,What looks suspicious here?,Tutorial: Building Models,10,Tutorial Objectives,Share data ana

9、lysis experiences with real data (COCOMO as a thinking aid) Show how models created from data are based on the average (or mean) of the data and its spread or variation Show how model performance improves with the removal of assignable causes of variation Raise awareness on the many sources of varia

10、tion in software engineering data,Tutorial: Building Models,11,What Will We Do?,Using supplied data, we will build simple models Mean or Median One variable regression models Stratifying data Two sets of data The first set will be used to learn a technique The second set will be used to practice the

11、 technique Intent is to show how to create small models by example,Tutorial: Building Models,12,What You Will Walk Away With,A new skill: using Excel to look at data Data summaries Graphing data Simple regression models An understanding of what is behind numbers produced by models, a.k.a. understand

12、ing variation An intelligent consumer of data (which you can practice during this conferences presentations) A responsible data reporter Understanding model parameters and their impact on explaining variation,Tutorial: Building Models,13,About the Instructor and SMI,Brad Clark Former Navy Pilot Work

13、ed in civil service for 10 years Attended USC Graduate School: 1992 - 1997 Development of the COCOMO II model Process Maturity Effects on Effort Started consulting in 1998 in using measurement to manage software projects Software Metrics, Inc. (SMI) Very small, private consulting company located in

14、Haymarket, Va. Started in 1983 by John and Betsy Bailey Focus: Using software measurement to manage software projects: estimation, feasibility analysis, performance,Tutorial: Building Models,14,About You,What is your name? Where do you work? Do you have any experience with statistics or empirical mo

15、deling?,Tutorial: Building Models,15,Tutorial Outline,Purpose A software engineering modeling example Model building steps Mean-based model exercise Regression based model exercise Summary,Tutorial: Building Models,16,What is a model?,A model is a representation of the essential structure of some ob

16、ject or event in the real world. Physical (airplane, building, bridge) Symbolic (language, computer program, mathematical equation) Two major characteristics of models Models are necessarily incomplete Models may be changed or manipulated with relative ease No model includes every aspect of the real

17、 world Building models necessarily involves simplifying assumptions It is critical that the assumptions made when constructing models be understood and be reasonable.,Source: Introductory Statistics Concepts, Models, and Applications by David Stockburger,Tutorial: Building Models,17,Using Data to Es

18、timate,Effort Consumption = 11.9 Person Hours / Function Point,Estimated Function EffortEffort = Points * Consumption880.6 = 74 * 11.93,665.2 = 308 * 11.95,057.5 = 425 * 11.9,ActualEffort165 14,0803,602,Tutorial: Building Models,18,First Model: Sample Mean (est. X),Tutorial: Building Models,19,Neces

19、sary and Sufficient Information,What additional information do we want to know about the stated relationship to make it more accurate?,Effort Consumption = 11.9 Person Hours / Function Point,Tutorial: Building Models,20,Data Analysis: PHr/FP,0,1,2,3,4,5,6,5,10,15,20,25,30,35,40,45,50,PHr/FP,Frequenc

20、y,0,80% CI,90% CI,Confidence Interval (CI),11.9,20.3,3.6,95% CI,18.2,5.7,22.3,1.6,Tutorial: Building Models,21,Reducing the Confidence Interval,Some assignable causes of variation among project data points Noisy data (size and effort) Complexity of the software (effort) Amount of required testing (e

21、ffort) Building components for reuse (effort) Changes in requirements (size) Required reliability and safety features (size) Interoperability (effort and size) Development / maintenance team experience (effort) Turnover of key people (effort),Tutorial: Building Models,22,Measurement Specifications,S

22、taff Turnover Specification ExampleTypical Data ItemsNumber of personnelNumber of personnel gained (per period)Number of personnel lost (per period)Typical AttributesExperience factorOrganizationTypical Aggregation StructureActivityTypically Collected for EachProjectCount Actuals Based OnFinancial r

23、eporting criteriaOrganization restructuring or new organizational chart,Source: Practical Software Measurement Objective Information for Decision Makers by McGarry et. al.,Tutorial: Building Models,23,Models Depend on Solid Data,Models are created from data Models are only as good as the data used t

24、o create them life-cycle phase overtime to get work done experience tools complexity reuse Data used to create models must be well specified,Tutorial: Building Models,24,Accounting for Requirements Volatility,Assignable cause of variation: Adjust the size with the effects of requirements volatility

25、(REVL) Adj FP = FP * (1 + REVL%),90% CI,11.9,4.2,8.0,Tutorial: Building Models,25,Impact of Personnel Continuity on Effort This factor captures the turmoil caused by the project losing key, lead personnel. The loss of key personnel leads to extra effort in new people coming to work for the project a

26、nd having to spend time coming up to speed on what has to be done. The rating scale is in terms of the projects personnel turnover normalized to a year.,Source: Software Cost Estimation with COCOMO II by Barry Boehm et. al.,Variation: Staff Turnover,Tutorial: Building Models,26,Accounting for Staff

27、Turnover,Assignable cause of variation: Adjust the effort with the effects of Personnel Continuity (PCON) Adj_PHr = PHr / PCON,90% CI,10.6,5.2,7.9,Tutorial: Building Models,27,COCOMO Suite,Attempts to identify and quantify assignable causes of variation (drivers) Model Purpose COCOMO Custom cost and

28、 schedule estimation COCOTS COTS Based Systems cost estimation COQUALMO Defect introduction and removal CORADMO Rapid application development cost and schedule estimation COPSEMO Staged schedule & effort model COPROMO Productivity improvement model COSYSMO System engineering cost and schedule est.,T

29、utorial: Building Models,28,My favorite!,Statistical Regression fits a line through points minimizing the least square error between the points and the line The regression analysis yields a line with a slope, M, and intercept, A: Y = A + MX + e The goodness of fit is given by a statistic called R2.

30、The closer to 1.0, the better the fit.,Second Model: Linear Regression Analysis,Tutorial: Building Models,29,Starting Point: Scatter Plot,Source: Albrecht and Gaffney, “Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation,“ IEEE Transactions on So

31、ftware Engineering, Vol SE-9, No 6, Nov 1983.,Y = 11.3 + 0.0651 X + e,Tutorial: Building Models,30,Regression Analysis Example,Tutorial: Building Models,31,Model Accuracy,PRED(L) = X Means that the model estimates within L% of the actual values X% of the time In other words, how often does the model

32、 predict within the desired circle?,Models are necessarily incomplete and are not 100% accurate,Tutorial: Building Models,32,Model Evaluation,Which model would you choose?,Tutorial: Building Models,33,Summary -1: Two Models,Mean X,1-Variable Regression,Tutorial: Building Models,34,Summary -2,Definit

33、ion of a model Data specifications Normal versus T distribution Model characteristics Model Usage Model boundaries Confidence interval Model accuracy Assignable causes of variation Large model examples,Tutorial: Building Models,35,Tutorial Outline,Purpose A software engineering modeling example Mode

34、l building steps Mean-based model exercise Regression based model exercise Summary,Tutorial: Building Models,36,Modeling Steps: 1. Decide what relationship you would like to investigate,What do you want to know Estimation of Requirements Volatility Establishing thresholds for performance monitoring

35、Working overtimes effect on personnel turnover Estimation of the number of defects to be found before Final Acceptance Test,People,Code Size,Requirements,Defects,Build Duration,Rework,Documentation,Test Cases,Design Units,Cost,Reqts Evolution,Process Maturity,Change Requests,Function Points,Tutorial

36、: Building Models,37,2. Identify assignable causes of variation (drivers),Use your experience and intuition Possible sources of variation: Customer participation - Development team experience Application domain exp. - Complexity of application Development flexibility - Design constraints Requirement

37、s volatility - Adaptation of existing code Programming lang. exp. - Use of modern methodologies Compression of schedule - Use of software tools Code inspections - Management capability Team size - Application size Personnel turnover - Architecture & Risk resolution,Tutorial: Building Models,38,3. Co

38、llect data,Specify data to be collected based on: assignable causes of variation what is available Select 10 projects to go back and collect extra data Based on project applicability Use measurement specifications as a checklist for each data How much project data is enough?,Tutorial: Building Model

39、s,39,4. Normalize data and check for consistency,In some cases it may be appropriate to normalize: normalize data to a “per unit” measure size defects calendar days, weeks, months effort hours, effort days, effort months normalize about the mean of the data to get percentage increase or decrease fro

40、m the mean Plot data Check that known relationships exist Detect outliers and investigate Scatter plots are very useful,Tutorial: Building Models,40,5. Build model and evaluate,Models should be: Simple Explainable Analyzable Most important: They should make sense! Models make explicit data relations

41、hips Show strength and direction of relationships While a relationship exists - it may not be valid or make sense The relationships you want to use in modeling are ones that show valid “cause and effect”,Tutorial: Building Models,41,6. Add or remove drivers,Drivers are data attributes that explain (

42、or drive) variation. The more drivers used in a model - the more data that must be collected. While a driver may make sense to use in explaining variation, the data may not support this conclusion Collect more data, the current dataset may be biased and not represent a true sample There may be drive

43、rs that are correlated, this could cover the effects of the weak performing driver Warning: Correlation effects between drivers,Tutorial: Building Models,42,7. Repeat steps 3 to 7,If the model does not have an acceptable accuracy, then: collect more data analyze it for its influence on variation add

44、 and remove cost drivers evaluate model,Tutorial: Building Models,43,8. Pilot model,Document model and create a tool for its use Model must be piloted to test its reasonableness, understandability, and accuracy. Collect actual values of model inputs (including assignable causes of variation) Model s

45、hould be used with its confidence interval Feedback should be incorporated into model and tool,Tutorial: Building Models,44,Tutorial: Building Models,45,Tutorial Outline,Purpose A software engineering modeling example Model building steps Mean-based model exercise Regression based model exercise Sum

46、mary,Tutorial: Building Models,46,Exercise 1: Growth Model,Modeling step #1: What do we want to investigate? We are going to develop a growth model based on real data in a report from NASA Software Engineering Laboratory* A growth model increases size based on “other information” It will be used in

47、estimating cost and schedule for future software projects When will the model be used? What information will be available at the time? What will be the scope of the model? What will be included or excluded in the estimate?,* Cost and Schedule Estimation Study Report, SEL-93-001,Tutorial: Building Mo

48、dels,47,Exercise 1: Assignable Causes of Variation,Modeling step #2: What are possible causes (that can be controlled) of growth? ?,Tutorial: Building Models,48,Exercise 1: Survey the Data,Modeling step #3: Collect data Using Microsoft Excel, open the file with the NASA SEL data. Select the data def

49、initions worksheet Project type Programming Language Duration Effort for management and technical Estimated SLOC size Actual SLOC size New SLOC size Growth (derived) Reuse (derived),Tutorial: Building Models,49,Exercise 1: Plot the Data,Modeling step #4: Normalize data and check for consistency Copy Data worksheet and name it “Scatter Plots” Create a scatter plot of the following data elements against Growth% Project Type Programming Language Duration Effort Estimated SLOC New SLOC Reuse%,