1、 ETSI EG 202 810 V1.1.1 (2010-03)ETSI Guide Methods for Testing and Specification (MTS);Automated Interoperability Testing;Methodology and Frameworkfloppy3ETSI ETSI EG 202 810 V1.1.1 (2010-03)2Reference DEG/MTS-00119 AutoIOP_Fwk Keywords interoperability, methodology, testing ETSI 650 Route des Luci
2、oles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present document can be downloaded from: http
3、:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dispute, the reference shall be the pr
4、inting on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at http
5、:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced except as authorized by written permission. The copyright and
6、 the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2010. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTM, TIPHONTM, the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members. 3GPPTM is a Trade Mar
7、k of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. LTE is a Trade Mark of ETSI currently being registered for the benefit of its Members and of the 3GPP Organizational Partners. GSM and the GSM logo are Trade Marks registered and owned by the GSM Association
8、. ETSI ETSI EG 202 810 V1.1.1 (2010-03)3Contents Intellectual Property Rights 4g3Foreword . 4g3Introduction 4g31 Scope 5g32 References 5g32.1 Normative references . 5g32.2 Informative references 5g33 Definitions and abbreviations . 6g33.1 Definitions 6g33.2 Abbreviations . 7g34 Basic concepts and gu
9、idelines 8g34.1 Interoperability of distributed systems . 8g34.2 Interoperability testing of distributed systems 8g34.3 Verdicts 9g34.4 Automation . 9g34.4.1 Limitations 10g34.4.2 Degree of automation 10g34.5 Test case development 10g34.5.1 Requirements on test description content . 11g34.5.2 Struct
10、ure, style and documentation of test cases . 11g34.5.3 Test reporting 12g34.6 Controllability of equipment with non standard interfaces 12g35 Automated interoperability testing of distributed systems . 13g35.1 A generic test framework for interoperability testing . 13g35.1.1 System under test 13g35.
11、1.2 Interconnecting network and application support nodes . 14g35.1.3 Means of interoperability testing 14g35.2 A development process for an interoperability test system 14g35.2.1 Interoperability test system design 15g35.2.1.1 Define test configuration . 15g35.2.1.2 Define message structure 15g35.2
12、.1.3 Specify test parameters . 16g35.2.2 Specify test cases 16g35.2.3 Implement codec and adaptation function 17g35.2.4 Validation . 18g3Annex A: TTCN-3 library for interoperability testing . 19g3Annex B: Interface versus entity based approach for realizing the interoperability testing framework . 2
13、0g3Annex C: Bibliography 22g3History 23g3ETSI ETSI EG 202 810 V1.1.1 (2010-03)4Intellectual Property Rights IPRs essential or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI membe
14、rs and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/webapp.ets
15、i.org/IPR/home.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, ess
16、ential to the present document. Foreword This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS). Introduction The encouragement of the European Commission for the adoption and promotion of generic test frameworks for the validation of standards
17、 based on multiple stacks including middleware provides evidence that successful testing as well as interoperability is key factors to enable the use of new technologies providing all benefits attached to them including competitiveness and innovation. However, technologies are becoming more complex,
18、 collaborative, and inter-dependant. Therefore, methodologies and approaches for ensuring interoperability need to be innovative and consider new evolving challenges such as the distribution of components and their remote access in an embedded environment. This guide adapts and presents a solid and
19、proven method to these new challenges. The current and future e-communication market can be described as a convergent multimedia market with an increasingly complex structure. Within the present competitive environment, the risk of non-interoperability is increasing due to a fast evolution of techno
20、logy and the use of non-open standards. The main purpose of standardization is to enable interoperability in a multi-vendor, multi-network, multi-service environment. The absence of interoperability should not be the reason why final services for which there is great demand do not come into being. E
21、TSI is very much aware of these developments and market demands. At ETSI, the inhibitors to interoperability that can be encountered during the standards development process are well known. A key part of this process is the development of test specifications for conformance and interoperability, and
22、 the provision of validation services based on many years of experience. On one hand, testing the conformance of every single entity (protocol layer) in a complex system seems often prohibitively expensive. On the other hand, pure interoperability testing does not ensure that entities strictly adher
23、e to standard specifications. For example, ETSI observed that 90 % of tests executed at an interoperability event showed that systems were interoperable. However, only 60 % of these executed tests showed reference points conforming to the standard used to link these systems together. In a production
24、 environment, this would undoubtedly lead to interoperability problems. Over the past few years, a combination of these two testing approaches has evolved. At ETSI, this approach is called interoperability testing with conformance checking in which key reference points are observed during the execut
25、ion of end-to-end interoperability tests to evaluate if the message flow at these points conforms to the requirement stated in standards. So far, this approach has shown very promising results but has mainly applied with manual interoperability testing. The focus of the present document is a generic
26、 methodology for automated interoperability testing with conformance checking of complex distributed systems. The proof of concept of this methodology has been done through a case study in the domain of IP Multimedia Subsystem (IMS). However, also other technologies such as IPv6, Robust Header Compr
27、ession (ROHC), Health Level 7 messaging, grid, cloud, WiMax, IPTV, WiMedia, Voice over IP (VoIP) for air traffic control, and smart cards have been analyzed and considered in the writing of the present document. ETSI ETSI EG 202 810 V1.1.1 (2010-03)51 Scope The present document describes a methodolo
28、gy for automated interoperability testing for the validation of standards, e.g. in the context of interoperability events and for the validation of products against standards. It extends and complements the best working practices presented in EG 202 237 i.1 which targets the certification of product
29、s based on interoperability testing. The present document also proposes a generic framework for interoperability testing of distributed systems and includes guidance on, for example, access to reference points, the establishment and maintenance of test synchronisation, and the assignment of test ver
30、dicts. 2 References References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For a specific reference, subsequent revisions do not apply. Non-specific reference may be made only to a complete document or a part thereof and only in th
31、e following cases: - if it is accepted that it will be possible to use all future changes of the referenced document for the purposes of the referring document; - for informative references. Referenced documents which are not found to be publicly available in the expected location might be found at
32、http:/docbox.etsi.org/Reference. NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 2.1 Normative references The following referenced documents are indispensable for the application of the present document. For dat
33、ed references, only the edition cited applies. For non-specific references, the latest edition of the referenced document (including any amendments) applies. Not applicable. 2.2 Informative references The following referenced documents are not essential to the use of the present document but they as
34、sist the user with regard to a particular subject area. For non-specific references, the latest version of the referenced document (including any amendments) applies. i.1 ETSI EG 202 237: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Generic approach to interoperabil
35、ity testing“. i.2 ETSI White Paper No.3 (Sophia-Antipolis, France, 2008): “Achieving Technical Interoperability - the ETSI Approach“, Hans van der Veer and Anthony Wiles. i.3 ETSI EG 202 568: “Methods for Testing and Specification (MTS); Internet Protocol Testing (IPT); Testing: Methodology and Fram
36、ework“. i.4 ISO/IEC 9646 (parts 1 to 7): “Information technology - Open Systems Interconnection - Conformance testing methodology and framework“. ETSI ETSI EG 202 810 V1.1.1 (2010-03)6i.5 ETSI ES 201 873-1: “Methods for Testing and Specification (MTS); The Testing and Test Control Notation version 3
37、; Part 1: TTCN-3 Core Language“. NOTE: Also published as ITU-T Recommendation series Z.140. i.6 ETSI ES 202 553: “Methods for Testing and Specification (MTS); TPLan: A notation for expressing Test Purposes“. i.7 ETSI ES 201 873-10: “Methods for Testing and Specification (MTS); The Testing and Test C
38、ontrol Notation version 3; Part 10: TTCN-3 Documentation Comment Specification“. NOTE: Also published as ITU-T Recommendation series Z.140. i.8 S. Schulz: “Test suite development with TTCN-3 libraries“, International Journal on Software Tools for Technology Transfer, 10(4), 327-36, Springer, 2008. i
39、.9 ETSI TR 102 788: “Methods for Testing and Specification (MTS); Automated Interoperability Testing; Specific Architectures“. 3 Definitions and abbreviations 3.1 Definitions For the purposes of the present document, the terms and definitions given in EG 202 237 i.1 and the following apply: abnormal
40、 test termination: describes the result of executing an abstract test case after it has been prematurely terminated by the test system abstract test case: complete and independent specification of the action required to achieve a specific test purpose, defined at the level of abstraction of a partic
41、ular abstract test method, starting and ending in a stable testing state (ISO/IEC 9646-1 i.4) abstract test suite: test suite composed of abstract test cases (ISO/IEC 9646-1 i.4) base specification: specification of a protocol, abstract syntax, encoding rules, or information object end-to-end testin
42、g: testing approach focused on verifying the correct behaviour of a set of interconnected systems by stimulating and observing the system functionalities from the end users point of view executable test case: realization of an abstract test case (ISO/IEC 9646-1 i.4) executable test suite: test suite
43、 composed of executable test cases (ISO/IEC 9646-1 i.4) fail verdict: test verdict given when the observed test outcome either demonstrates non-interoperability of equipment under test with respect to the end-to-end functionality on which the test case is focused, or demonstrates non-conformance wit
44、h respect to at least one of the conformance requirement(s) on which the test purpose(s) associated with the test case is (are) focused, or contains at least one invalid test event, with respect to the relevant specifications NOTE: The above definition extends the original definition for this term p
45、rovided in ISO/IEC 9646-1 i.4. inconclusive verdict: test verdict given when the observed test outcome is such that neither a pass nor a fail verdict can be given (ISO/IEC 9646-1 i.4) pass verdict: test verdict given when the observed test outcome gives evidence of all equipment under test interoper
46、ating for the end-to-end functionality on which the test case is focused, or conformance to the conformance requirement(s) on which the test purpose(s) of the test case is (are) focused, and when all the test events are valid with respect to the relevant specifications NOTE: The above definition ext
47、ends the original definition for this term provided in ISO/IEC 9646-1 i.4. test architecture: abstract description of logical entities as well as their interfaces and communication links involved in a test ETSI ETSI EG 202 810 V1.1.1 (2010-03)7test case: generic, abstract or executable test case (IS
48、O/IEC 9646-1 i.4) test configuration: concrete instance of a test architecture defined on the basis of test components, ports and their connection test description: informal description of a test case usually written in English prose test event: indivisible unit of test specification at the level of
49、 abstraction of the specification (e.g. sending or receiving a single message) (ISO/IEC 9646-1 i.4) test log: human readable record of information produced as a result of a test campaign which is sufficient to record the observed test outcomes and to verify the assignment of the test result (including test verdicts) test oracle: mechanism that determines a test verdict test outcome: sequence of test events, together with associated data and/or parameter values, which occurred during test execution of a specific parameterised execut
