1、 ETSI EG 201 015 V2.1.1 (2012-02) Methods for Testing and Specification (MTS); Standards engineering process; A Handbook of validation methods ETSI Guide ETSI ETSI EG 201 015 V2.1.1 (2012-02)2Reference REG/MTS-00122 ValidHandB Keywords interoperability, methodology, procedure, testing, validation ET
2、SI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present document can be
3、downloaded from: http:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dispute, the refe
4、rence shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents
5、 is available at http:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced except as authorized by written permissi
6、on. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2012. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTMand the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members. 3GPPTM and LTE are Trade Mar
7、ks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. GSM and the GSM logo are Trade Marks registered and owned by the GSM Association. ETSI ETSI EG 201 015 V2.1.1 (2012-02)3Contents Intellectual Property Rights 5g3Foreword . 5g3Introduction 5g31 Scope 6g32 Re
8、ferences 6g32.1 Normative references . 6g32.2 Informative references 6g33 Definitions and abbreviations . 8g33.1 Definitions 8g33.2 Abbreviations . 8g34 Validation overview . 9g35 Reasons for validating standards 9g35.1 Problems commonly found by validating standards . 9g35.1.1 Editorial issues 10g3
9、5.1.2 Technical issues 10g35.2 Avoiding problems normally found during validation . 11g35.3 Using validation methods to detect problems . 12g36 Methods for validating standards . 12g36.1 Different types of standard . 12g36.2 Explicit validation 12g36.2.1 Review methods 12g36.2.1.1 Overview of review
10、ing methods . 12g36.2.1.2 Using reviews to validate standards 13g36.2.1.3 Basic review process . 13g36.2.1.3.1 Phase 1: Planning the review . 14g36.2.1.3.1.1 Assign a leader for the review process . 15g36.2.1.3.1.2 Select and invite the reviewers. 15g36.2.1.3.1.3 Schedule a date for the review . 15g
11、36.2.1.3.2 Phase 2: Preparing for the review 16g36.2.1.3.3 Phase 3: Reviewing the specification 16g36.2.1.3.3.1 Presentation of the overview from the document author(s) . 16g36.2.1.3.3.2 Inspection of the document 17g36.2.1.3.3.3 Preparation and agreement of any required change proposals . 17g36.2.1
12、.3.4 Phase 4: Processing change proposals . 17g36.2.1.3.4.1 Implement changes in a document revision . 17g36.2.1.3.4.2 Review actual changes . 17g36.2.2 Model-based validation methods 17g36.2.2.1 Modelling and simulation . 18g36.2.2.2 Model quality assessment . 19g36.2.3 Test suite validation 19g36.
13、2.4 Product-based validation (proof-by-use) 19g36.2.4.1 ETSI Plugtests events 19g36.2.4.1.1 Benefits of using a testing events for validating standards 19g36.2.4.1.2 Criteria for selecting a testing event for validation 20g36.2.4.1.3 Overall process 20g36.2.4.1.3.1 Responsibilities 20g36.2.4.1.3.2 F
14、actors to consider in making a Plugtests event successful . 21g36.2.4.2 Prototyping 23g36.3 Implicit validation 23g36.3.1 Requirements cataloguing . 23g36.3.2 Test specification development . 24g36.3.2.1 ICS and IFS . 24g3ETSI ETSI EG 201 015 V2.1.1 (2012-02)46.3.2.2 Test Suite Structure . 25g36.3.2
15、.3 Test Purposes 25g36.3.2.4 Test Suite 25g36.3.2.5 IXIT. 25g36.3.3 Conformance and interoperability testing . 25g36.3.4 Development of descriptive and guidance documentation . 26g36.3.5 Product development 26g36.4 Validation best practice 26g36.4.1 Identify the most appropriate validation level 26g
16、36.4.1.1 Defined validation levels . 26g36.4.1.2 Select the required method(s) 28g36.4.1.3 Plan validation into the standards development process . 28g36.4.1.4 Resolve problems detected during validation 29g3Annex A: Sample review forms. 30g3A.1 Introduction 30g3A.1.1 A form for recording reviewers
17、comments . 30g3Annex B: Test Suite Validation Guidelines . 31g3B.1 Introduction 31g3B.2 Validation levels . 31g3B.2.1 Level 1 validation (Basic) 31g3B.2.1.1 Description 31g3B.2.2 Level 2 validation (Strong) . 31g3B.2.2.1 Description 31g3B.2.3 Level 3 validation (Rigorous) . 32g3B.2.3.1 Description 3
18、2g3B.3 Completing the Terms of Reference. 32g3B.3.1 The extent of validation 32g3B.3.2 Responsibilities 33g3B.3.3 Costs associated with validation . 33g3B.4 The validation report 34g3History 36g3ETSI ETSI EG 201 015 V2.1.1 (2012-02)5Intellectual Property Rights IPRs essential or potentially essentia
19、l to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified
20、to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/ipr.etsi.org). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the
21、existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Foreword This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS). Introduct
22、ion ETSI produces technical standards for a wide range of communication applications. All these standards need to be thoroughly validated and many different methods are available for this purpose. The present document collects and presents those validation methods which are practical and have been s
23、uccessfully applied either within ETSI or in other standardization bodies. Each method has its advantages and disadvantages and they are probably most effective if used in combination and at all stages of the standards development process. A three level approach for the basic, strong and rigorous va
24、lidation of standards is presented to guide the selection and application of standard validation methods. ETSI ETSI EG 201 015 V2.1.1 (2012-02)61 Scope The present document identifies and describes a number of practical methods that can be used to validate all types of standards produced by ETSI. Th
25、e methods described are as follows: review methods; model-based methods; product-based methods; and testing. It also gives guidance on how and when each of these methods can be used most effectively within ETSIs standards engineering process. 2 References References are either specific (identified b
26、y date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies. Referenced documents which are not found to be publi
27、cly available in the expected location might be found at http:/docbox.etsi.org/Reference. NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 2.1 Normative references The following referenced documents are necessary
28、 for the application of the present document. Not applicable. 2.2 Informative references The following referenced documents are not necessary for the application of the present document but they assist the user with regard to a particular subject area. i.1 ETSI EG 201 383: “Methods for Testing and S
29、pecification (MTS); Use of SDL in ETSI deliverables; Guidelines for facilitating validation and the development of conformance tests“. i.2 ETSI EG 201 872: “Methods for Testing and Specification (MTS); Methodological approach to the use of object-orientation in the standards making process“. i.3 ETS
30、I EG 202 107: “Methods for Testing and Specification (MTS); Planning for validation and testing in the standards-making process“. i.4 ETSI EG 202 237: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Generic approach to interoperability testing“. i.5 ETSI EG 202 568: “M
31、ethods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Testing: Methodology and Framework“. i.6 ETSI ES 201 873-1: “Methods for Testing and Specification (MTS); The Testing and Test Control Notation version 3; Part 1: TTCN-3 Core Language“. ETSI ETSI EG 201 015 V2.1.1 (2012-02)
32、7i.7 ETSI ES 202 553: “Methods for Testing and Specification (MTS); TPLan: A notation for expressing Test Purposes“. i.8 ETSI TS 102 385-2: “Broadband Radio Access Networks (BRAN); HiperMAN;Conformance Testing for WiMAX/HiperMAN 1.2.1; Part 2: Test Suite Structure and Test Purposes (TSS Internet Pro
33、tocol Testing (IPT): IPv6 Security; Requirements Catalogue“. i.10 ETSI TS 102 593: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); IPv6 Security; Conformance Test Suite Structure and Test Purposes (TSS digital Private Mobile Radio (dPMR); Part 3: Requirements catalogue
34、“. i.12 ETSI TS 102 870-2: “Intelligent Transport Systems (ITS); Testing;Conformance test specifications for GeoNetworking Basic Transport Protocol (BTP); Part 2: Test Suite Structure and Test Purposes (TSS Overview of validation techniques for European Telecommunication Standards (ETSs) containing
35、SDL“. i.14 ETSI ETR 212: “Methods for Testing and Specification (MTS); Implementation Conformance Statement (ICS) proforma style guide“. i.15 ETSI ETS 300 406: “Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology“. i.16 ET
36、SI Directives, Version 27 (2010). i.17 ISO/IEC 9646-1: “Information technology; Open Systems Interconnection; Conformance testing methodology and framework; Part 1: General concepts“. i.18 ITU-T Recommendation Z.100: “Specification and Description Language (SDL)“. i.19 ITU-T Recommendation Z.120: “M
37、essage Sequence Chart (MSC)“. i.20 ITU-T Recommendation X.680: “Information technology - Abstract Syntax Notation One (ASN.1): Specification of basic notation“. i.21 OMG: “Unified Modelling Language (UML)“. i.22 OMG: “UML Testing Profile (UTP)“. i.23 IEEE Guide to the Software Engineering Body of Kn
38、owledge (SWEBOK). i.24 IEEE 1028-2008: “IEEE Standard for Software Reviews and Audits“. i.25 “The Elements of UML 2.0 Style“, Scott W. Ambler; Cambridge University Press; ISBN: 0-521-61678-6. i.26 “Seven process modeling guidelines (7PMG)“, J. Mendling, H. A. Reijers, and W. M. P. van der Aalst; Inf
39、ormation and Software Technology Volume 52, Issue 2, February 2010. i.27 “Design and Code inspections to reduce errors in program development“, M.E., Fagan; IBM Systems Journal Volume 15, Issue 3, 1976. i.28 “An approach to quality engineering of TTCN-3 test specifications“, Helmut Neukirchen, Benja
40、min Zeiss, Jens Grabowski; International Journal on Software Tools for Technology Transfer (STTT) Volume 10 Issue 4, 2008. i.29 “A Flexible Framework for Quality Assurance of Software Artefacts with Applications to Java, UML, and TTCN-3 Test Specifications“, Jens Nodler, Helmut Neukirchen, Jens Grab
41、owski; Proceedings of the 2nd International Conference on Software Testing, Verification and Validation (ICST), 2009; IEEE Computer Society, ISBN 978-0-7695-3601-9. ETSI ETSI EG 201 015 V2.1.1 (2012-02)8i.30 “On the Application of Software Metrics to UML Models“, Jacqueline A. McQuillan and James F.
42、 Power; Lecture Notes in Computer Science, 2007; Volume 4364, Models in Software Engineering; Springer. i.31 “Come, Lets Play: Scenario-Based Programming Using LSCs and the Play-Engine“, David Harel, Rami Marelly, Springer 2005. i.32 “A Visual Approach to Validating System Level Designs“, Jochen Klo
43、se, Thomas Kropf, Jrgen Ruf, ISSS02, October 2-4, 2002, Kyoto, Japan. 3 Definitions and abbreviations 3.1 Definitions For the purposes of the present document, the following terms and definitions apply: validation: the process, with associated methods, procedures and tools, by which an evaluation is
44、 made that a standard can be fully implemented, conforms to rules for standards, satisfies the purpose expressed in the record of requirements on which the standard is based and that an implementation that conforms to the standard will have the functionality expressed in the record of requirements o
45、n which the standard is based 3.2 Abbreviations For the purposes of the present document, the following abbreviations apply: 7PMG Seven Process Modelling Guidelines ASN.1 Abstract Syntax Notation number 1 i.20 ATS Abstract Test Suite CR Change RequestCTI ETSI Centre for Testing and Interoperability
46、HiperMAN High Performance Radio Metropolitan Access Network ICS Implementation Conformance Statement IFS Implementable Functions Statement IPv6 Internet Protocol version 6 ITS Intelligent Transport System IXIT Implementation eXtra Information for Testing LSC Life Sequence Charts MSC Message Sequence
47、 Chart i.19 MV Member Vote SDL Specification and Design Language i.18 STF Specialist Task Force SUT System Under Test SWEBOK Software Body of Engineering Knowledge T3Q TTCN-3 Quality evaluation software tool TB Technical Body TC Test Case TC/WG Technical Commitee/Working Group TD Test Descriptions T
48、oR Terms of Reference TP Test Purpose TPLan Notation for expressing Test Purposes i.7 TSS Test Suite Structure TTCN-3 Testing and Test Control Notation version 3 i.6 UML Unified Modelling Language i.21 UTP UML Testing Profile i.22 WG Working Group WiMax Worldwide Interoperability for Microwave ACCes
49、s ETSI ETSI EG 201 015 V2.1.1 (2012-02)94 Validation overview Any standard is a collection of requirements which, if implemented correctly, achieve certain objectives in a product. These requirements could specify physical characteristics such as length, voltage and accessibility or behaviour such as the interchange of messages and the responses to particular stimuli. The purpose of validation is to ensure as far as possible that the requirements expressed in the standard do, in fact, meet
