1、Introduction to Software Testing (2nd edition) Chapter 3 Test Automation,Paul Ammann & Jeff Offutthttp:/www.cs.gmu.edu/offutt/softwaretest/,Updated February 2016,What is Test Automation?,Reduces cost Reduces human error Reduces variance in test quality from different individuals Significantly reduce
2、s the cost of regression testing,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,2,The use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting funct
3、ions,Software Testability (3.1),Plainly speaking how hard it is to find faults in the software Testability is dominated by two practical problems How to provide the test values to the software How to observe the results of test execution,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & O
4、ffutt,3,The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met,Observability and Controllability,ObservabilitySoftware that affects hardware devices, databases, or remote files have low o
5、bservability ControllabilityEasy to control software with inputs from keyboards Inputs from hardware sensors or distributed software is harderData abstraction reduces controllability and observability,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,4,How easy it is to observe the
6、 behavior of a program in terms of its outputs, effects on the environment and other hardware and software components,How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors,Components of a Test Case (3.2),A test case is a multipart artifact with a d
7、efinite structureTest case valuesExpected results A test oracle uses expected results to decide whether a test passed or failed,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,5,The result that will be produced by the test if the software behaves as expected,The input values need
8、ed to complete an execution of the software under test,Affecting Controllability and Observability,Prefix valuesPostfix valuesVerification Values : Values needed to see the results of the test case values Exit Values : Values or commands needed to terminate the program or otherwise return it to a st
9、able state,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,6,Any inputs that need to be sent to the software after the test case values are sent,Inputs necessary to put the software into the appropriate state to receive the test case values,Putting Tests Together,Test caseTest se
10、tExecutable test script,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,7,The test case values, prefix values, postfix values, and expected results necessary for a complete execution and evaluation of the software under test,A set of test cases,A test case that is prepared in a f
11、orm to be executed automatically on the test software and produce a report,Test Automation Framework (3.3),Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,8,A set of assumptions, concepts, and tools that support test automation,Introduction to Software Testing, Edition 2 (Ch 3),
12、Ammann & Offutt,9,What is JUnit?,Open source Java testing framework used to write and run repeatable automated tests JUnit is open source (junit.org) A structure for writing test drivers JUnit features include: Assertions for testing expected results Test features for sharing common test data Test s
13、uites for easily organizing and running tests Graphical and textual test runners JUnit is widely used in industry JUnit can be used as stand alone Java programs (from the command line) or within an IDE such as Eclipse,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,10,JUnit Tests
14、,JUnit can be used to test an entire object part of an object a method or some interacting methods interaction between several objects It is primarily intended for unit and integration testing, not system testing Each test is embedded into one test method A test class contains one or more test metho
15、ds Test classes include : A collection of test methods Methods to set up the state before and update the state after each test and before and after all tests Get started at junit.org,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,11,Writing Tests for JUnit,Need to use the method
16、s of the junit.framework.assert class javadoc gives a complete description of its capabilities Each test method checks a condition (assertion) and reports to the test runner whether the test failed or succeeded The test runner uses the result to report to the user (in command line mode) or update th
17、e display (in an IDE) All of the methods return void A few representative methods of junit.framework.assert assertTrue (boolean) assertTrue (String, boolean) fail (String),Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,12,JUnit Test Fixtures,A test fixture is the state of the te
18、st Objects and variables that are used by more than one test Initializations (prefix values) Reset values (postfix values) Different tests can use the objects without sharing the state Objects used in test fixtures should be declared as instance variables They should be initialized in a Before metho
19、d Can be deallocated or reset in an After method,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,13,Simple JUnit Example,public class Calc static public int add (int a, int b)return a + b; ,import org.junit.Test; import static org.junit.Assert.*; public class CalcTest Test public
20、 void testAdd()assertTrue (“Calc sum incorrect”, 5 = Calc.add (2, 3); ,Test values,Expected output,Printed if assert fails,Note: JUnit 4 syntax,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,14,Testing the Min Class,import java.util.*;public class Min /* Returns the mininum elem
21、ent in a list* param list Comparable list of elements to search* return the minimum element in the list* throws NullPointerException if list is null or* if any list elements are null* throws ClassCastException if list elements are not mutually comparable* throws IllegalArgumentException if list is e
22、mpty*/ ,public static T min (List list)if (list.size() = 0)throw new IllegalArgumentException (“Min.min“);Iterator itr = list.iterator();T result = itr.next();if (result = null) throw new NullPointerException (“Min.min“);while (itr.hasNext() / throws NPE, CCE as neededT comp = itr.next();if (pareTo
23、(result) 0)result = comp; return result; ,MinTest Class,Standard imports for all JUnit classes :,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,15,import static org.junit.Assert.*; import org.junit.*; import java.util.*;,Test fixture and pre-test setup method (prefix) :,Post tes
24、t teardown method (postfix) :,private List list; / Test fixture/ Set up - Called before every test method. Beforepublic void setUp() list = new ArrayList();,/ Tear down - Called after every test method. After public void tearDown() list = null; / redundant in this example ,Introduction to Software T
25、esting, Edition 2 (Ch 3), Ammann & Offutt,16,Min Test Cases: NullPointerException,Test public void testForNullList() list = null;try Min.min (list); catch (NullPointerException e) return;fail (“NullPointerException expected”); ,Test (expected = NullPointerException.class) public void testForNullElem
26、ent() list.add (null);list.add (“cat“);Min.min (list); ,This NullPointerException test uses the fail assertion,This NullPointerException test decorates the Test annotation with the class of the exception,This NullPointerException test catches an easily overlooked special case,Test (expected = NullPo
27、interException.class) public void testForSoloNullElement() list.add (null);Min.min (list); ,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,17,More Exception Test Cases for Min,Test (expected = ClassCastException.class) SuppressWarnings (“unchecked“) public void testMutuallyIncom
28、parable() List list = new ArrayList();list.add (“cat“);list.add (“dog“);list.add (1);Min.min (list); ,Test (expected = IllegalArgumentException.class) public void testEmptyList() Min.min (list); ,Note that Java generics dont prevent clients from using raw types!,Special case: Testing for the empty l
29、ist,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,18,Remaining Test Cases for Min,Finally! A couple of “Happy Path” tests,Test public void testSingleElement() list.add (“cat“);Object obj = Min.min (list);assertTrue (“Single Element List“, obj.equals (“cat“); Testpublic void tes
30、tDoubleElement() list.add (“dog“);list.add (“cat“);Object obj = Min.min (list);assertTrue (“Double Element List“, obj.equals (“cat“);,Summary: Seven Tests for Min,Five tests with exceptions null list null element with multiple elements null single element incomparable types empty elements Two withou
31、t exceptions single element two elements,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,19,Data-Driven Tests,Problem : Testing a function multiple times with similar values How to avoid test code bloat? Simple example : Adding two numbers Adding a given pair of numbers is just l
32、ike adding any other pair You really only want to write one test Data-driven unit tests call a constructor for each collection of test values Same tests are then run on each set of data values Collection of data values defined by method tagged with Parameters annotation,Introduction to Software Test
33、ing, Edition 2 (Ch 3), Ammann & Offutt,20,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,21,Example JUnit Data-Driven Unit Test,import org.junit.*; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; import org.junit.runners.Parameterized.Parameters; import
34、static org.junit.Assert.*; import java.util.*;RunWith (Parameterized.class) public class DataDrivenCalcTest public int a, b, sum;public DataDrivenCalcTest (int v1, int v2, int expected) this.a = v1; this.b = v2; this.sum = expected; Parameters public static Collection parameters() return Arrays.asLi
35、st (new Object 1, 1, 2, 2, 3, 5); Test public void additionTest() assertTrue (“Addition Test“, sum = Calc.add (a, b); ,Test 1 Test values: 1, 1 Expected: 2,Test 2 Test values: 2, 3 Expected: 5,Constructor is called for each triple of values,Test method,Tests with Parameters: JUnit Theories,Unit test
36、s can have actual parameters So far, weve only seen parameterless test methods Contract model: Assume, Act, Assert Assumptions (preconditions) limit values appropriately Action performs activity under scrutiny Assertions (postconditions) check result,Introduction to Software Testing, Edition 2 (Ch 3
37、), Ammann & Offutt,22,Theory public void removeThenAddDoesNotChangeSet (Set someSet, String str) / Parameters!assumeTrue (someSet != null) / AssumeassumeTrue (someSet.contains (str) ; / AssumeSet copy = new HashSet(someSet); / Actcopy.remove (str);copy.add (str);assertTrue (someSet.equals (copy); /
38、Assert ,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,23,Question: Where Do The Data Values Come From?,Answer: All combinations of values from DataPoints annotations where assume clause is true Four (of nine) combinations in this particular case Note: DataPoints format is an ar
39、ray,DataPointspublic static String animals = “ant“, “bat“, “cat“;DataPointspublic static Set animalSets = new HashSet (Arrays.asList (“ant“, “bat“),new HashSet (Arrays.asList (“bat“, “cat“, “dog“, “elk”),new HashSet (Arrays.asList (“Snap”, “Crackle”, “Pop“);,Nine combinations of animalSetsi.contains
40、 (animalsj) is false for five combinations,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,24,JUnit Theories Need BoilerPlate,import org.junit.*; import org.junit.runner.RunWith; import static org.junit.Assert.*; import static org.junit.Assume.*;import org.junit.experimental.theo
41、ries.DataPoint; import org.junit.experimental.theories.DataPoints; import org.junit.experimental.theories.Theories; import org.junit.experimental.theories.Theory;import java.util.*;RunWith (Theories.class) public class SetTheoryTest / See Earlier Slides ,Introduction to Software Testing, Edition 2 (
42、Ch 3), Ammann & Offutt,25,Running from a Command Line,This is all we need to run JUnit in an IDE (like Eclipse)We need a main() for command line execution ,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,26,AllTests,import org.junit.runner.RunWith; import org.junit.runners.Suite;
43、 import junit.framework.JUnit4TestAdapter;/ This section declares all of the test classes in the program. RunWith (Suite.class) Suite.SuiteClasses ( StackTest.class ) / Add test classes here.public class AllTests / Execution begins in main(). This test class executes a/ test runner that tells the te
44、ster if any fail.public static void main (String args)junit.textui.TestRunner.run (suite();/ The suite() method helps when using JUnit 3 Test Runners or Ant.public static junit.framework.Test suite()return new JUnit4TestAdapter (AllTests.class); ,Introduction to Software Testing, Edition 2 (Ch 3), A
45、mmann & Offutt,27,How to Run Tests,JUnit provides test drivers Character-based test driver runs from the command line GUI-based test driver-junit.swingui.TestRunner Allows programmer to specify the test class to run Creates a “Run” buttonIf a test fails, JUnit gives the location of the failure and a
46、ny exceptions that were thrown,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,28,JUnit Resources,Some JUnit tutorials http:/open.ncsu.edu/se/tutorials/junit/(Laurie Williams, Dright Ho, and Sarah Smith ) http:/www.laliluna.de/eclipse-junit-testing-tutorial.html(Sascha Wolski and
47、 Sebastian Hennebrueder) http:/ (Diaspar software) http:/ (Clarkware consulting) JUnit: Download, Documentation http:/www.junit.org/,Introduction to Software Testing, Edition 2 (Ch 3), Ammann & Offutt,29,Summary,The only way to make testing efficient as well as effective is to automate as much as possible Test frameworks provide very simple ways to automate our tests It is no “silver bullet” however it does not solve the hard problem of testing :,What test values to use ?,This is test design the purpose of test criteria,
