Automating Analysis and Test.ppt

1、(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 1,Automating Analysis and Test,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 2,Learning objectives,Understand the main purposes of automating software analysis and testing Identify activities that can be fully or partially automated Understand cost a

2、nd benefit trade-offs in automation Separate publicity from important features in descriptions of commercial A&T tools,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 3,Three Potential Roles of Automation,Necessary for introducing a task example: coverage tools enable measuring structural coverage o

3、f test suites Useful to reduce cost example: capture and replay tools reduce the costs of reexecuting test suites Useful to increase (human) productivity example: software inspection is a manual activity, but tools to organize and present information and manage communication increase the productivit

4、y of people,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 4,Approaching Automation,Prioritize automation steps based on variations in impact, maturity, cost, scope of the technology fit and impact on the organization and process Three (non-orthogonal) dimensions for automation value and current co

5、st of the activity extent to which the activity requires or is made less expensive by automation cost of obtaining or constructing tool support,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 5,Automation Costs Vary Enormously,Some tools are so simple to develop that they are justifiable even if the

6、ir benefits are modest example: generate test cases from finite state machine models Some tools that would be enormously valuable are simply impossible example: identify exactly which parts of a program can never be executed (a provably undecidable problem),(c) 2007 Mauro Pezz & Michal Young,Ch 23,

7、slide 6,Costs May Depend on Scope,Sometimes a general-purpose tool is only marginally more difficult to produce than a tool specialized for one project example: general capture and replay for Windows applications vs capture and replay for a specific Windows application Investment in the general-purp

8、ose tool, whether to build it or to buy it, can be amortized across projects In other cases, simple, project-specific tools may be more cost effective Tool construction is often a good investment in a large project example: simulators to permit independent subsystem testing,(c) 2007 Mauro Pezz & Mic

9、hal Young,Ch 23, slide 7,Focusing Where Automation Pays,Simple repetitive tasks are often straightforward to automate humans are slow and make errors in repetitive tasks But .judgment and creative problem solving remain outside the domain of automation Example: Humans are Very good at identifying re

10、levant execution scenarios that correspond to test case specifications Very inefficient at generating large volumes of test cases or identifying erroneous results within a large set of outputs from regression tests Automating the repetitive portions of the task reduces costs, and improves accuracy a

11、s well,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 8,Planning: The Strategy Level,Prescribes tools for key elements of the quality process Can include detailed process and tool prescriptions Recommends different tools contingent on aspects of a project (application domain, development languages,

12、 size, overall quality,.) Often included in the A&T strategy: tools for Organizing test design and execution Generating quality documents Collecting metrics Managing regression test suites Less often included: tools for Generating test cases Dynamic analysis,(c) 2007 Mauro Pezz & Michal Young,Ch 23,

13、 slide 9,Planning: The Project Level,The A&T Plan Indicates Tools inherited from the strategy Additional tools selected for that project For new or customized tools, the A&T plan must include Costs (including training) Implied activities Potential risks The plan positions tools within the developmen

14、t process and the analysis and test methodology Avoid waste of cost and effort from lack of contextualization of the tools Example: tools for measuring code coverage simple and inexpensive (if not properly contextualized) an annoyance, producing data not put to productive use,(c) 2007 Mauro Pezz & M

15、ichal Young,Ch 23, slide 10,Process Support: Planning & Monitoring,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 11,Automation in Process Management,Managing a process involves . planning a set of activities with appropriate cost and quality trade-offs monitoring progress to identify risks as earl

16、y as possible avoiding delays by adjusting the plan as needed . and requires . human creativity and insight for which no tool can substitute Tools can support process management and improve decision making by organizing and monitoring activities and results facilitating group interaction managing qu

17、ality documents tracking costs,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 12,Classic Planning Tools,Facilitate task scheduling, resource allocation, and cost estimation by arranging tasks according to resource and time constraints Can be specialized to A&T management with features for deriving

18、relations among tasks, launching tasks, and monitoring completion of activities Examples: tools to recognize delivery of a given artifact schedule execution of a corresponding test suite notify test designer of test results record the actual execution time of the activity signal schedule deviations

19、to the quality manager Most useful when integrated in the analysis and test environment,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 13,Version and Configuration Control Tools,Analysis and testing involve complex relations among a large number of artifacts Version and configuration management too

20、ls relate versions of software artifacts trigger consistency checks and other activities support analysis and testing activities like they control assembly and compilation of related modules example: trigger execution of the appropriate test suites for each software modification Improve efficiency i

21、n well-organized processes not a substitute for organization,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 14,Monitoring,Integrated quality tracking improves efficiency in a well-structured process, does not by itself bring order out of chaos Progress must be monitored in terms of schedule (actual

22、 effort and completion times vs plan) level of quality Quality of the final product cannot be directly measured before its completion but we can derive useful indications example: orthogonal defect classification see chapter 20,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 15,Quality Tacking,Essen

23、tial function: recognize deviations from expectation as early as possible to reduce consequences Proxy measures must be computed early must be interpreted in a way that avoids misleading conclusions or distorted incentives Example: lines of code useful as a simple proxy for productivity must be care

24、fully interpreted to avoid creating both an incentive for verbosity and a disincentive for effective reuse Example: number of faults detected useful to detect deviations from the norm one should be as concerned about the causes of abnormally low numbers as high Collection, summary, and presentation

25、of data can be automated Design and interpretation cannot be automated,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 16,Managing People,People may work in different groups in different companies distributed across time zones and continents A large proportion of a software engineers time is devoted

26、 to communication We need to facilitate effective communication limit disruptions and distractions of unmanaged communication,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 17,Managing Communication,Simple general-purpose tools (e-mail, chat, forum, .) balance synchronous with asynchronous communic

27、ation examples When excessive interruptions slow progress, we may replace synchronous with asynchronous communication Conversely, when communication is splintered into many small exchanges punctuated by waits for reply, we may replace asynchronous with synchronous communication Communication is most

28、 effective when all parties have immediate access to relevant information Task-specific tools can improve on general-purpose support Example: tools for distributed software inspections Extend chat interfaces or forum with Managed presentation of the artifact to be inspected Appropriate portions of c

29、hecklists and automated analysis results,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 18,Measurement,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 19,Metrics,Measuring progress . Metrics are proxy measures (rough guides) based on what we can measure Anything that is correlated with the real mea

30、sure of interest under typical conditions Usually require calibration to local conditions,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 20,Static Metrics: Size,Static metrics measure some software properties, often to estimate other properties (i.e., as proxies for things we cant measure) Size is

31、the most basic property strongly correlated with schedule and cost several possible variations, depending on white space, comments, programming style Course measures include counts of modules or interfaces functions, methods, formal parameters, etc Many more complex measures . but lines of code is a

32、bout as good (or bad) as complex measures for judging effort,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 21,Measuring Complexity,Intuitive rationale: If we could measure how complicated a program or its parts were, we could . Focus test & analysis on most error-prone parts of a system Make bette

33、r plans and schedules Consider redesign of excessively complex subsystems But we cant measure true (logical) complexity directly. Control flow complexity is a proxy.,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 22,Cyclomatic complexity,Among attempts to measure complexity, only cyclomatic complex

34、ity is still commonly collected cyclomatic complexity V(g) = number of independent paths through the control flow graph = e - n + 2 (edges - nodes + 2),(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 23,Cyclomatic metrics and complexity,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 24,Interpreting

35、 Cyclomatic Complexity,V(g) 20 high cyclomatic complexity complex programs V(g) 50 very high cyclomatic complexity programs very difficult or impossible to thoroughly test Cyclomatic vs logical complexity sign of complex control flow structure does not capture other aspects of logical complexity tha

36、t can lead to difficulty in testing,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 25,Metrics & Quality Standards,Quality standards May be prescribed (e.g., by contract) May be adopted voluntarily as guidance A quality standard like ISO/IEC 9126 requires measurement of user-perceived quality but do

37、esnt say how to measure it To implement . We must find objective indicators (metrics) for each required quality,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 26,ISO/IEC 9126 Metrics (level 1),Broad qualities require refinement and mapping to objectively measurable properties,(c) 2007 Mauro Pezz &

38、Michal Young,Ch 23, slide 27,Automating Program Analysis, Test Case Generation, and Test Execution,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 28,Test Case Generation and Execution,Automation is important because It is large fraction of overall test and analysis costs can become a scheduling bot

39、tleneck near product delivery deadlines Designing a test suite involves human creativity Instantiating and executing test cases is a repetitive and tedious task can be largely automated to reduce costs accelerate the test cycle,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 29,Automated Testing - S

40、tages,Push the creative work as far forward as possible E.g., designing functional test suites is part of the specification process At each level, from systems requirements through architectural interfaces and detailed module interfaces Construct scaffolding with the product Automate instantiation a

41、nd execution So they are not a bottleneck So they can be repeated many times,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 30,Static Analysis and Proof,Effective for Quick and cheap checks of simple properties Example: simple data flow analyses can identify anomalous patterns Expensive checks nece

42、ssary for critical properties Example: finite state verification tool to find synchronization faults,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 31,Design for Verification,Decompose Verification Problems Design: enforce design rules to accommodate analysis example: encapsulate safety-critical pr

43、operties into a safety kernel Verification: focus on encapsulated or simplified property example: prove safety properties of the (small) kernel check (cheaply, automatically) that all safety-related actions are mediated by the kernel,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 32,Undecidability

44、and Automated Analysis,Some tools report false alarms in addition to real violations of the properties they check example: data flow analyzers Some tools avoid false alarms but may also fail to detect all violations example: bug finders Some tools are heavyweight with respect to requirement for skil

45、led human interaction and guidance to provide strong assurance of important general properties examples Finite state verification systems (model checkers) can verify conformance between a model of a system and a specified property require construction of the model and careful statement of the proper

46、ty Theorem provers execute with interactive guidance requires specialists with a strong mathematical background to formulate the problem and the property interactively select proof strategies,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 33,Complex analysis tools,Verifiers based on theorem proving

47、 verify a wide class of properties require extensive human interaction and guidance Finite state verification tools restricted focus execute completely automatically almost always require several rounds of revision to properly formalize a model and property to be checked,(c) 2007 Mauro Pezz & Michal

48、 Young,Ch 23, slide 34,Simple analysis tools,Restricted to checking a fixed set of simple properties do not require any additional effort for specificationType checkers typically applied to properties that are syntactic = enforce a simple well-formedness rule violations are easy to diagnose and repa

49、ir Often rules are stricter than one would like Data flow analyzers sensitive to program control and data flow often used to identify anomalies rather than simple, unambiguous faults Checkers of domain specific properties Web site link checkers ,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 35,Cog

50、nitive Aids,Supporting creative, human processes,(c) 2007 Mauro Pezz & Michal Young,Ch 23, slide 36,Cognitive Aids: Problems to Address,Nonlocality Information that requires a shift of attention Example: following a reference in one file or page to a definition on another creates opportunities for human error Information clutter Information obscured by a mass of distracting irrelevant detail,