1、9/11/13,T. Dawson, TASC,Assurance Cases in Planning and Execution of NASA IV&V Projects,1,History of Evidence-Based Assurance at IV&V,2,Evidence-Based Assurance, that is, providing mission and safety assurance based on documented, objective evidence, is a component of the NASA IV&V Program Mission S
2、tatement and Strategic Plan The NASA IV&V Mission Statement reads, in part, “To provide our customers assurance that their safety and mission-critical software will operate reliably and safely and to advance the systems and software engineering disciplines.” The NASA IV&V Vision Outcome 1.2 of that
3、Plan reads: “We produce results that are empirically-derived and clearly indicate the reliability and safety of operating the system” - “epirically-derived” means, in part, based on objective, documented evidence For years NASA IV&V managers have struggled with determining the best ways to infuse Ev
4、idence-Based Assurance principles into the IV&V culture, and with implementing appropriate techniques and tools Evidence-based assurance* (working definition): providing assurance, through a structured argument based on evidence, that some mission need will be met Assurance Cases provide one approac
5、h to meeting these needs that is currently gaining momentum within the Program Evidence-Based Assurance is the need. The approach taken to fill this need is the use of Assurance Case methodologies,Evidence-Based Assurance at NASA IV&V,* Sometimes contrasted with process-based assurance,Since the NAS
6、A IV&V program was founded in 1993, there have been a very large number of activity types used in performing IV&V Many of these activities depend on subject matter expertise to perform the analysis. IV&V has subject matter expertise in a number of subject areas, including: software and its many aspe
7、cts hardware and its many aspects mission types various systems domains, e.g. GN&C and propulsion systems The level of documentation from these analyses has varied from project to project Human-rated mission typically produce more detailed documentation For example, the IV&V report to support the re
8、turn to flight decision following the Columbia disaster was over 1500 pages long, with detailed technical discussion of the analysis approaches used along with supporting detail,4,Evidence at NASA IV&V,Many IV&V efforts have been well documented This includes not only human-rated systems The fact re
9、mains that the level of documentation generated has been inconsistent from project to project From time to time, the analysis has consisted of the subject matter experts simply applying their expertise to the system under evaluation and providing conclusions, with the only documentation resulting fr
10、om this process being the conclusions themselves There would be no documentation of the approach taken, the evaluation criteria, or any other aspect of the analysis that supports the conclusions This does not meet Program needs, in that the results are not repeatable or reviewable It is not our asse
11、rtion that subject matter expertise is unnecessary or can be replaced by process only that mere existence of the expertise without documentation is insufficient,5,Evidence at NASA IV&V (cont.),Lack of documentation is not the only possible shortcoming of evidence-based assurance Even if the process
12、is fully documented, that documentation does not constitute evidence in an evidence-based assurance sense unless it supports a structured argument to make a given assurance claim This means documentation is necessary but not sufficient for evidence-based assurance In recent years there has been incr
13、eased emphasis on documentation to ensure better consistency across all projects Less emphasis has been placed on performing evidence-based assurance in any structured sense, e.g. using assurance cases Summarizing, IV&V activities sometimes (not universally) have had the following limitations: Activ
14、ities not being documented sufficiently for reproduction or review Activities not planned and executed in a structured, evidence-based assurance manner,6,Evidence at NASA IV&V (cont.),Assurance Cases,7,8,Assurance Case Basics,Assurance Cases are a type of structured argument that has a large body of
15、 literature in academics and industry Assurance cases provide not only the concepts and vernacular, but also a body of methodologies that are of use The fundamental Assurance Case structure involves using collected evidence to support an argument that proves a claim Evidence must be both objective a
16、nd documented in order to support the resulting argument(s),Assurance Case,Evidence,Argument,Claim,The full assurance case standard used here is IEEE 15026-2-2011, Systems and Software Engineering Systems and Software Assurance Part 2: Assurance Case, IEEE, NY, 11 Oct 2011 This standard is the IEEE
17、adoption of ISO/IEC 15026-2:2011 In addition to evidence, arguments, and claims, this standard includes the additional concepts of assumptions and justifications Initially we will concentrate on the simplified structure shown above, followed by an exploration of these additional concepts below,9,IEE
18、E Assurance Case Standard,Within IV&V, claims directly correspond to assurance goals For a given project goal to provide an assurance statement, that statement is a claim in the assurance case sense Its arguments must be supported by sufficient evidence Evidence is identified and collected during IV
19、&V activities IV&V activities build the argument However, the assurance case to be made is not whatever happens to be supported by the evidence collected by the activities that happen to be performed The activities are defined as necessary to collect the planned evidence The planned evidence is that
20、 evidence needed to support the intended claim Only by considering the goals (i.e. intended claims) can the appropriate IV&V activities be selected,10,Assurance Cases in IV&V,IV&V Activity,Identify/ Collect the EvidenceBuild the Argument,11,Intended Claims Support IV&V Planning,During planning, we w
21、alk through the assurance case backwards In the execution process, evidence supports arguments which support claims In the planning process, we Start with the intended claims Determine the necessary arguments Determine the necessary evidence Then plan the activities necessary to collect that evidenc
22、e.,Evidence,Argument,Intended Claims,Intended Assurance Case,IV&V Planning Process,Determine the IV&V Activities Necessary to Support the Intended Assurance Case,12,Integrated Assurance Case-Based IV&V Planning & Execution,Evidence,Argument,Intended Claims,Intended Assurance Case,IV&V Planning Proce
23、ss,IV&V Activity,Determine the IV&V Activities Necessary to Support the Intended Assurance Case,Identify/ Collect the EvidenceBuild the Argument,Conclusion: application of assurance case methodologies can and should provide a means of closing the project planning gap,The proposed planning steps are
24、therefore: Select the project goals Develop the list of claims that support to the selected goals Develop the list of arguments that support the intended claims Determine the needed evidence Define the necessary IV&V activities Provide execution details and direction to analysts It is important to n
25、ote that steps 1, 5 and 6 are already performed by IV&V projects Steps 2, 3 and 4 are the fundamental point of this approach, intended to provide input to the planner on how to perform steps 5 and 6,13,Assurance Case Process Summary for IV&V,Process Example,14,15,Simple Example: End-to-End Process,R
26、equirement: For module M, output q shall always be greater than or equal to output r for all input sets Note: module M is stateless Given: a developer-provided input/output table for Module M,Evidence: table of outputs for all inputs,Argument: by inspection of exhaustive set of cases, we confirm tha
27、t q r in all cases,Intended Claim: For module M, output q is always greater than or equal to output r for all input sets,Resulting Assurance Case,Input/output Table,Explanation of approach and results (make the argument),Make Claim,Intended Assurance Case,IV&V Planning Process,IV&V Activity,We deter
28、mine that we must:1. Obtain table covering all cases2. Examine all cases for value of q w.r.t r3. Document assurance case(Scheduling, assigning task, etc. are all important but not germane),Execute #1, #2 and #3 from planning process,16,Simple Example: Alternate Argument,Requirement: For module M, o
29、utput q shall always be greater than or equal to output r for all input sets Note: module M is stateless Given: No I/O table is available, but an executable model is available,Evidence: Executable model,Argument: by inspection of exhaustive set of cases, we confirm that q r in all cases,Intended Cla
30、im: For module M, output q is always greater than or equal to output r for all input sets,Resulting Assurance Case,Executable Model,Explanation of approach and results (make the argument),Make Claim,Intended Assurance Case,IV&V Planning Process,IV&V Activity,We determine that we must:1. Obtain execu
31、table model2. Generate table covering all cases3. Examine all cases for value of q w.r.t r4. Document assurance case,Execute #1 through #4 from planning process,Do we really go through this process for every requirement? Not necessarily we wont build 5,000 assurance cases for a requirement set with
32、5,000 requirements There may be individual requirements that merit this There is generally a one-to-one relationship between activities and assurance cases In picking an example, a simple example was necessary to illustrate the process This thought process could be used in the requirements analysis,
33、 i.e. in the analyst notes wherever those are currently captured (“Verified by examination of exhaustive I/O table that q r in all cases”),17,Comments on the Example,Approach A: Examine exhaustive, developer-provided I/O table (proof by inspection) Approach B: Generate I/O table from develop-provide
34、d executable model and continue with argument of Approach A Approach C: Generate model from design or requirements then continue with argument from Approach B Approach D: Prove directly (e.g. mathematically) from the design or requirements that claim is always true Approach E: Exhaustively exercise
35、the code in a test environment etc.Claim must not overstate, i.e. it must take into account the evidence Evidence from the requirements or design does not support a claim about the code,18,Simple Example: Alternative Arguments,Claim elaboration Iterative planning process,Real-World Considerations,19
36、,The project goals selected in step 1 may not lend themselves directly to claim development since Project goals are often high level Useful claims need to be relatively low-level in order to be directly relatable to IV&V activities If the initial project goals are too high-level, a necessary step is
37、 to decompose the claim into sub-claims The sub-claims then have their own associated arguments and evidence, or potentially further sub-claims IV&V planning is then performed for each lowest-level claim This introduces the concept of a claim being supported by something other than a single argument
38、, specifically that of a claim being supported by one or more sub-claims Claims can also be supported by assumptions (unsubstantiated claims) in addition to sub-claims and arguments.,Assurance Case Elaboration,21,Claim Elaboration,Assurance Case,Evidence,Argument,Sub-Claim,Assurance Case,Evidence,Ar
39、gument,Sub-Claim,Claim,Assurance Case Network,* A sub-claim is a claim. A sub-claim is just a claim that supports another claim,*,*,There may also be unintended results during evidence collection, which include: Conflicting evidence Incomplete evidence Inability to collect planned evidence The appro
40、priate claim (based on actual vs. intended evidence) may emerge to be different from the originally-intended claim These considerations are handled through planning process iteration, allowing mid-course corrections or revisions to IV&V plans,22,Planning Process Iteration,23,Iterative IV&V Planning,
41、Intended Assurance Case,Resulting Assurance Case,IV&V Planning Process,IV&V Activity,Evidence,1,2,3,1,2,3,4,4,Intermediate activity results , intermediate or final evidence , and the resulting assurance case that can be supported can all feed back into the IV&V planning process in order to allow adj
42、ustments to the IV&V plans.,Iterative IV&V planning can feed back to the intended assurance case if necessary.,Claim Considerations for IV&V,24,Two considerations have provided the biggest stumbling blocks: Claim structure starting point (first-level decomposition) When to stop There are numerous st
43、arting approaches to creating a claim structure the structure can be based on: IV&V project goals IV&V Three Questions Will the systems software do what it is supposed to do? Will the systems software not do what it is not supposed to do? Will the systems software respond as expected under adverse c
44、onditions? System architectural decomposition (GN&C, power, C&DH, ) System-level behaviors (attain proper orbit, collect intended science, ) How far? At what point does the assurance case approach become self-serving and not help attain IV&V goals? Is the solution a null set due to cost-effectivenes
45、s?,Conclusions and Observations from Initial IV&V Implementation of Assurance Cases,26,One consideration: Assurance Case Approaches Need Not be Mutually-Exclusive,GN&C Requirements Evidence,Argument,Claim: EDL will perform correctly,Claim: Requirements support subsequent phases,Claim: GN&C s/w will
46、perform as needed,Claim: The system will perform as needed,Argument,Argument,Argument,Other Evidence,Other Evidence,Notation and Tools,27,The IEEE standard does not specify notation, graphical or otherwise One standard in use at NASA IV&V is Goal Structuring Notation, or GSN GSN has been in use sinc
47、e 1997 The current standard initially in draft in May 2010 GSN introduces a graphical standard for representing assurance cases, and is supported by a variety of tools Unfortunately, GSN does not directly support the IEEE standard The IEEE standard has elements of claims, argument, evidence, justifi
48、cations and assumptions GSN has elements of goals, solutions, strategies, assumptions, contexts, and justifications GSN was not directly created to support assurance cases, although it can be (and often is) applied to them GSN has the broader scope of any structured argument, of which assurance case
49、s are one type. GSN defines Graphical representations of each of its elements Two types of linkages between elements, SupportedBy and InContextOf The total network of elements and linkages is known as the goal structure (what we have called the Assurance Case Network previously),28,Goal Structuring
50、Notation,Claim-argument-evidence (CAE) notation was created by Adelard and supported by their COTS ASCE tool ASCE stands for Assurance and Safety Case Environment CAE more closely follows the IEEE terminology, but does not include justifications or assumptions In the IEEE standard, an assumption is
51、just a special case of a claim, so a CAE claim can fill that need CAE also has the element other, which can attach general text to any element, which can fill the need for IEEE justifications The fifth and final CAE element is caption, which is used to provide annotation over the graph CAE also introduces linkages of various types between elements. As a tool (vs. a standard), ASCE provides functionality in addition to the graphical network representation, including reports, exporting, and others. ASCE was introduced here in the context of CAE, but ASCE supports both GSN and CAE,
