ITU-T Q 3930-2012 Performance testing of distributed systems C Concepts and terminology (Study Group 11)《分布式系统(预发布)性能测试 概念和术语》.pdf

1、 International Telecommunication Union ITU-T Q.3930TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (08/2012) SERIES Q: SWITCHING AND SIGNALLING Signalling requirements and protocols for the NGN Testing for next generation networks Performance testing of distributed systems Concepts and terminology R

2、ecommendation ITU-T Q.3930 ITU-T Q-SERIES RECOMMENDATIONS SWITCHING AND SIGNALLING SIGNALLING IN THE INTERNATIONAL MANUAL SERVICE Q.1Q.3 INTERNATIONAL AUTOMATIC AND SEMI-AUTOMATIC WORKING Q.4Q.59 FUNCTIONS AND INFORMATION FLOWS FOR SERVICES IN THE ISDN Q.60Q.99 CLAUSES APPLICABLE TO ITU-T STANDARD S

3、YSTEMS Q.100Q.119 SPECIFICATIONS OF SIGNALLING SYSTEMS No. 4, 5, 6, R1 AND R2 Q.120Q.499 DIGITAL EXCHANGES Q.500Q.599 INTERWORKING OF SIGNALLING SYSTEMS Q.600Q.699 SPECIFICATIONS OF SIGNALLING SYSTEM No. 7 Q.700Q.799 Q3 INTERFACE Q.800Q.849 DIGITAL SUBSCRIBER SIGNALLING SYSTEM No. 1 Q.850Q.999 PUBLI

4、C LAND MOBILE NETWORK Q.1000Q.1099 INTERWORKING WITH SATELLITE MOBILE SYSTEMS Q.1100Q.1199 INTELLIGENT NETWORK Q.1200Q.1699 SIGNALLING REQUIREMENTS AND PROTOCOLS FOR IMT-2000 Q.1700Q.1799 SPECIFICATIONS OF SIGNALLING RELATED TO BEARER INDEPENDENT CALL CONTROL (BICC) Q.1900Q.1999 BROADBAND ISDN Q.200

5、0Q.2999 SIGNALLING REQUIREMENTS AND PROTOCOLS FOR THE NGN Q.3000Q.3999 General Q.3000Q.3029 Network signalling and control functional architecture Q.3030Q.3099 Network data organization within the NGN Q.3100Q.3129 Bearer control signalling Q.3130Q.3179 Signalling and control requirements and protoco

6、ls to support attachment in NGN environments Q.3200Q.3249 Resource control protocols Q.3300Q.3369 Service and session control protocols Q.3400Q.3499 Service and session control protocols supplementary services Q.3600Q.3649 NGN applications Q.3700Q.3849 Testing for next generation networks Q.3900Q.39

7、99For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T Q.3930 (08/2012) i Recommendation ITU-T Q.3930 Performance testing of distributed systems Concepts and terminology Summary Recommendation ITU-T Q.3930 describes terminology and concepts of performance tests with a g

8、eneralized view of performance characteristics as the starting point. What kind of characteristics are indicators of a products performance and what kind of measurement data is captured and processed to provide relevant figures on requested performance are in this view the core of performance testin

9、g. The Recommendation includes a set of terminology and descriptions of concepts in performance testing that could be accepted as a common base for descriptions of performance and performance tests. History Edition Recommendation Approval Study Group 1.0 ITU-T Q.3930 2012-08-13 11 Keywords Performan

10、ce testing. ii Rec. ITU-T Q.3930 (08/2012) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent

11、organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes t

12、he topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards a

13、re prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommenda

14、tion may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are us

15、ed to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intelle

16、ctual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received

17、 notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/.

18、ITU 2013 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T Q.3930 (08/2012) iii Table of Contents Page 1 Scope 1 1.1 Organization of the present Recommendation . 1 2 References. 1 3 Definitions 1 3.1 T

19、erm defined elsewhere . 1 3.2 Terms defined in this Recommendation . 1 4 Abbreviations and acronyms 1 5 Conventions 2 6 Performance characteristics 3 6.1 Classifying performance characteristics into categories 3 6.2 Powerfulness characteristics . 3 6.3 Reliability characteristics . 8 6.4 Efficiency

20、characteristics 10 7 Measured objects 13 7.1 Measured services 13 7.2 Measured components 13 7.3 Service concepts . 14 7.4 Service characteristics 15 7.5 Service interfaces 19 8 Performance measurement data objectives and attributes 19 8.1 Performance metric objectives . 19 8.2 Measurement data attr

21、ibute sets . 21 8.3 Processing attributes or metric types 21 8.4 Identification attributes or metric identifiers 22 8.5 Unit attributes or metric formats 23 8.6 Conditional attributes . 24 9 Abstract performance metrics . 24 9.1 Abstract performance metrics and performance categories . 24 9.2 Abstra

22、ct powerfulness metrics . 25 9.3 Abstract reliability metrics . 28 9.4 Abstract efficiency metrics . 36 10 Performance data processing 41 10.1 Steps in performance data processing 41 10.2 Time series of performance data 42 10.3 Collection and storage of raw performance data 42 10.4 Condensation and

23、normalization of raw performance data 42 10.5 Performance data computations . 42 iv Rec. ITU-T Q.3930 (08/2012) Page 10.6 Evaluation of performance data 43 10.7 Presentation of performance data . 43 11 General performance test concepts . 43 11.1 Performance tests 43 11.2 Performance tests and system

24、 life cycle phases . 43 11.3 Performance test objectives 45 11.4 Performance objectives and performance requirements . 46 11.5 Performance measurement conditions 46 11.6 Performance targets 46 11.7 Performance measurements standards 46 11.8 Some performance test characteristics . 47 12 Performance t

25、est environment 47 12.1 Test environment concepts . 47 12.2 System under test concepts . 48 12.3 Test system concepts 50 13 Performance test specifications 53 13.1 Elements of performance test specifications 53 13.2 Test objectives 54 13.3 Test conditions 54 13.4 Test configurations . 56 13.5 Test d

26、ata specifications . 57 13.6 Test evaluation specifications . 58 14 Workload concepts . 58 14.1 Workload set or traffic set 58 14.2 Workload content . 58 14.3 Workload volume . 59 14.4 Load concepts . 59 14.5 Workload time distribution . 59 Bibliography. 61 Rec. ITU-T Q.3930 (08/2012) v Introduction

27、 The background to the present Recommendation is that few standard specifications define any kind of performance or performance targets. A common explanation to this is that capacity issues are not a matter for standardization bodies, but for product vendors and product buyers to decide. This opinio

28、n excludes the need for definition and application of other performance characteristics such as reliability, stability, efficiency and many others. Another more fundamental reason for writing the present Recommendation is that there are no strict definitions of what kind of characteristics should be

29、 regarded as indicators of performance. In the absence of strict definitions in this area, one tends to talk about performance characteristics in terms of types of performance tests, such as robustness tests or availability tests. A consequence of this is that conformance testing as specified in ISO

30、/IEC 9646-1 does not cover performance tests explicitly. A starting point is therefore a set of terminology and descriptions of concepts in performance testing that could be accepted as a common base for discussions about performance and performance tests. Rec. ITU-T Q.3930 (08/2012) 1 Recommendatio

31、n ITU-T Q.3930 Performance testing of distributed systems Concepts and terminology 1 Scope The present Recommendation describes terminology and concepts of performance tests with a generalized view of performance characteristics as the starting point. What kind of characteristics are indicators of a

32、 products performance and what kind of measurement data is captured and processed to provide relevant figures on requested performance is in this view the kernel of performance testing. Methods for performance testing will consequently be guided by the requirements on expected output. A set of follo

33、wing Recommendations will describe strategies, methodologies and techniques of performance testing. 1.1 Organization of the present Recommendation The present Recommendation has two parts. Part one is about performance characteristics and performance metrics. It contains a general view of performanc

34、e characteristics followed by a general view of measured objects, or what is measured in performance tests. Part one also contains general objectives and attributes of performance data and performance metrics, i.e., a conceptual view of performance metrics that is followed by a catalogue of abstract

35、 performance metrics that covers the performance characteristics described earlier. At the end of part one is a conceptual view of performance data processing, i.e., the steps between captured performance data and presented results on performance. Part two is about performance testing concepts. It s

36、tarts with a general view of performance testing followed by a conceptual view of the test environment and finally a conceptual view of performance test specifications. 2 References None. 3 Definitions 3.1 Term defined elsewhere None. 3.2 Terms defined in this Recommendation The core text of this Re

37、commendation defines the terms used in performance testing. 4 Abbreviations and acronyms This Recommendation uses the following abbreviations and acronyms: API Application Programming Interface CPU Central Processing Unit CSCF Call Session Control Function 2 Rec. ITU-T Q.3930 (08/2012) DHCP Dynamic

38、Host Configuration Protocol DIMM Dual In-line Memory Module DoS Denial of Service GMT Greenwich Mean Time HSS Home Subscriber Server HTTP Hypertext Transfer Protocol HTTPS Hypertext Transfer Protocol Secure IMS IP Multimedia Subsystem IP Internet Protocol IPsec Internet Protocol Security IPTV Intern

39、et Protocol TV IPv4 Internet Protocol version 4 IPv6 Internet Protocol version 6 IUT Implementation Under Test KSR Kilo Service Request MSR Mega Service Request MTBF Mean Time Between Failures OSI Open Systems Interconnection SC Supporting Components S-CSCF Serving-Call Session Control Function SCTP

40、 Stream Control Transmission Protocol SIP Session Initiation Protocol SMS Short Messaging Service SOAP Simple Object Access Protocol SUT System Under Test TC Tested Components TCP Transmission Control Protocol TS Test System UDP User Datagram Protocol UE User Equipment 5 Conventions None. Rec. ITU-T

41、 Q.3930 (08/2012) 3 6 Performance characteristics 6.1 Classifying performance characteristics into categories An almost infinite number of performance characteristics can be applied on any computer system. However, measuring a complete set of performance characteristics of a system (if possible) is

42、not only impractical, costly and time consuming, it can also be argued if it will improve or confuse the understanding of the tested systems performance. Performance measurements are focused on obtaining figures about a selected set of performance attributes of a system. To simplify the selection of

43、 performance characteristics for a system it is convenient to group performance characteristics about similar or related aspects of performance into performance categories and select desired performance characteristics in each category. Examples of such categories can be powerfulness, reliability or

44、 efficiency characteristics of a system. These performance categories can be exemplified by a car where: A powerfulness characteristic is “top speed“. A reliability characteristic is “maintenance intervals“. An efficiency characteristic is “mileage or fuel consumption per 100 km“. 6.2 Powerfulness c

45、haracteristics The powerfulness category of performance characteristics contains indicators of speed and quantity of service production. The powerfulness category can be applied on all levels from system or application level down to low-level services of different components. The powerfulness catego

46、ry has subcategories for responsiveness, capacity and scalability. 6.2.1 Responsiveness characteristics Time definitions The time to handle a service request can be split into a large number of steps, where each step causes a time delay. The responsiveness of a service request is the sum of all thes

47、e time delays, the total time it takes to handle it. Every time delay falls into one of three time delay categories: 1. transmission time 2. queuing time 3. processing time Transmission time and transmission rate Transmission time is delays caused by transferring data related to the processing of a

48、service request. Transmission time can be measured on any level from sending a service request to the SUT down to transmission of data between CPU and memory. Transmission time in a network is the time to send a specific amount of data between two nodes over a network. The transmission time starts w

49、hen the first bit is transmitted by the sender and ends when the last bit is received by the receiver. The transmission time is a function of the amount of data being transmitted and the transmission rate or bandwidth of the used link. A network link with a bandwidth of 100 Mbit/s has a nominal transmission rate of 12.5 Mbyte/s. However, the transport and network layer protocols used to send the data between the end nodes in a packet-switched network will decrease the nominal transmission rate by 20-40%. The res