1、 ETSI TR 102 779 V1.1.1 (2009-03)Technical Report Speech and multimedia Transmission Quality (STQ);Multi-component KPIfloppy3ETSI ETSI TR 102 779 V1.1.1 (2009-03) 2Reference DTR/STQ-00128m Keywords 3G, GSM, network, QoS, service ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel
2、.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present document can be downloaded from: http:/www.etsi.org The present document may be made
3、available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept
4、on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at http:/portal.etsi.org/tb/status/status.asp If you fi
5、nd errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced except as authorized by written permission. The copyright and the foregoing restriction extend to reproductio
6、n in all media. European Telecommunications Standards Institute 2009. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTM, TIPHONTM, the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members. 3GPPTM is a Trade Mark of ETSI registered for the benefit of its Memb
7、ers and of the 3GPP Organizational Partners. LTE is a Trade Mark of ETSI currently being registered for the benefit of its Members and of the 3GPP Organizational Partners. GSM and the GSM logo are Trade Marks registered and owned by the GSM Association. ETSI ETSI TR 102 779 V1.1.1 (2009-03) 3Content
8、s Intellectual Property Rights 4g3Foreword . 4g3Introduction 4g31 Scope 5g32 References 5g32.1 Normative references . 5g32.2 Informative references 5g33 Abbreviations . 6g34 MCKPI general concept . 6g34.1 MCKPI framework . 6g34.2 MCKPI basics 7g35 MCKPI technical aspects . 7g35.1 MCKPI validity 7g35
9、.2 MCKPI methodologies . 9g35.3 Extending MCKPI to non-seamless tests . 9g36 Summary and conclusion . 9g3Annex A: Details on statistical validity 10g3History 11g3ETSI ETSI TR 102 779 V1.1.1 (2009-03) 4Intellectual Property Rights IPRs essential or potentially essential to the present document may ha
10、ve been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI stand
11、ards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/webapp.etsi.org/IPR/home.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of oth
12、er IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality (STQ). Introduction Q
13、oS testing, if done properly, requires considerable resources in time and personnel and is therefore costly. Consequently, efforts to increase productivity are an important part of technical evolution in this field. There are some constants defining in the framework in which any such evolution has t
14、o work. The first and most important one is the requirement of testing from the customers perspective. The second and equally important requirement is to provide statistical relevance, so to say the “quality of quality measurements“. Since, as operational experience shows, the dominant factor in cos
15、t of measurements is the time spent on these measurements, efforts focus on the question on how the time per data sample can be reduced, assuming that the number of data points should not be reduced to maintain statistical validity of results. The requirement of testing from the customers point of v
16、iew restricts the space in which solutions can be found. Test cases cannot be modified arbitrarily; activities have still to reflect typical customer behaviour. A possible solution is therefore multiple use of collected data. This is mainly based on the fact that todays popular data services (ftp, h
17、ttp, e-mail) all are internet-based which means that internet access is common to them. NOTE: The concept of MCKPI is of course not limited to internet-based services or re-use of QoS data for internet access. The present document will, for practical as well as for didactical reasons, restrict itsel
18、f on this issue. The basic idea of MCKPI is now to use data collected in one type of test for KPI of another test also. This is best explained with an example. Assume the following sequence of testing: Internet access (Attach, PDP context activation). ftp download. http web site download. and calcul
19、ating the TS 102 250-2 i.1 KPI suite using the Internet access results both for ftp and http KPI. From the above example it becomes clear that MCKPI is not only an operation in the mathematical domain of data processing; the advantage in terms of cost reduction for QoS measurements only materializes
20、 if the way tests are run is changed. Such a far-reaching change in methodology and data processing needs, of course, careful consideration. The present document attempts to contribute to this consideration by providing facts, insights and discussion material on statistical and methodological aspect
21、s on MCKPI. ETSI ETSI TR 102 779 V1.1.1 (2009-03) 51 Scope The present document discusses the concept of multi-component QoS KPI , their methods and validity rules. The term Multi-component KPI (MCKPI) describes a KPI generated not from a single, continuous test case, but combined from results of di
22、fferent tests. The motive to use MCKPI is to reduce cost and/or increase yield of measurement campaigns. Aspects treated in the present document include: MCKPI concept and types. Statistical validity of MCKPI. Practical examples. 2 References References are either specific (identified by date of pub
23、lication and/or edition number or version number) or non-specific. For a specific reference, subsequent revisions do not apply. Non-specific reference may be made only to a complete document or a part thereof and only in the following cases: - if it is accepted that it will be possible to use all fu
24、ture changes of the referenced document for the purposes of the referring document; - for informative references. Referenced documents which are not found to be publicly available in the expected location might be found at http:/docbox.etsi.org/Reference. NOTE: While any hyperlinks included in this
25、clause were valid at the time of publication ETSI cannot guarantee their long term validity. 2.1 Normative references The following referenced documents are indispensable for the application of the present document. For dated references, only the edition cited applies. For non-specific references, t
26、he latest edition of the referenced document (including any amendments) applies. Not applicable. 2.2 Informative references The following referenced documents are not essential to the use of the present document but they assist the user with regard to a particular subject area. For non-specific refe
27、rences, the latest version of the referenced document (including any amendments) applies. i.1 ETSI TS 102 250-2: “Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects for popular services in GSM and 3G networks; Part 2: Definition of Quality of Service parameters and their computat
28、ion“. ETSI ETSI TR 102 779 V1.1.1 (2009-03) 63 Abbreviations For the purposes of the present document, the following abbreviations apply: DNS Domain Name Server KPI Key Performance Indicator NOTE: Quality of Service parameter. MCKPI Multi-Component KPI PDP Packet Data Protocol PDPCA PDP Context Acqu
29、isition 4 MCKPI general concept 4.1 MCKPI framework Many services in mobile communications have very similar structures if viewed from an end-to-end, macroscopic event flow perspective. For example, usage of all internet-based packet data services starts with obtaining network access (Attach), follo
30、wed by principal packet data service access (PDP Context Activation) and in most cases also basic Internet access (e.g. DNS URL resolution). From an “measurement effort“ optimization point of view, the question arises if measurement results of such common elements obtained with one type of service c
31、an be re-used to provide results for another type of service, without actually measuring them. For example, consider the following sequence: Service access (which may consist itself of several sub-phases). ftp test. http test. and then using the service access results for generating both end-to-end
32、KPI for ftp and http. Of course, this will only work if the preceding steps of above sequence are successful. In that case, the yield of measurements would be much higher by skipping the service teardown phase after ftp, and the service access phase before http. From the testing methodology, it mean
33、s to run the service access sequence only when it is needed (i.e. after a breakdown of service access) and otherwise re-use existing connections. Obviously, there are some pre-requisites which have to be fulfilled to even think of such a method: The sequence for which the MCKPI are computed is seaml
34、essly described by component KPI. The pre-condition of having usable service access for dependent tests is properly validated (or it is implicitly given). Later in the present document, we will see that there are other, less obvious aspects of validity of MCKPI. For the purpose of this introductory
35、clause, we will however stay at this general level. As a general remark, a certain type of MCKPI “philosophy“ is contained even in the basic TS 102 250-2 i.1 framework of KPI by defining “service independent KPI“ in contrast to service-related KPI. Interpreting this as a methodological guideline, it
36、 allows to run “pure internet access“ tests and use these results in arbitrary way. ETSI ETSI TR 102 779 V1.1.1 (2009-03) 74.2 MCKPI basics There are two basic types of KPI and therefore also two types of MCKPI. 1) Success or failure rates A KPI of this type is expressed as the percentage of success
37、ful (positive KPI) or unsuccessful (negative KPI) tries with respect to the total number of tries. The following deals, for clarity, with success rates only. Failure rate computing is done accordingly. Assume that a full (real) test for a end-to-end KPI consists of n phases, and Ri is the success ra
38、te of the i-th phase. Then an overall success rate can be written as the product of individual success rates: R(e2e) = R0*R1*R(n-1) This becomes obvious if each success rate is written in its original form. Si = Si/Ti With S and T being the success count and the Tries count for this phase. If now T(
39、i) = S(i-1), the product above can be shortened to: R(e2e) = S(n-1)/T0 Which is exactly the formula for a conventional end-to-end success rate KPI. 2) Times Likewise, and not requiring mathematical formulae, the end to end time for a single instance of a service test is the time from start of the fi
40、rst phase to the end of the last phase, of course for the case that all phases have been passed successfully. 5 MCKPI technical aspects 5.1 MCKPI validity The basic paradigm of MCKPI can be expressed verbally in a way such as “if a common phase is successful in a full test sequence of service A, it
41、would also have been successful in a full test sequence of service B“. Or put in more concrete and practical form, since the internet service access does not “know“ if it is used for subsequent ftp or http, its result can be “assigned“ to both in an end to end test. More generally, the single phases
42、 have to be independent. The question now is, what means “independent“? Obviously, there is, strictly speaking, no such independence at all when taking into account that in tests of the type dealt with in QoS: the system under test is constantly changing, which means that doing the same test at time
43、s T1 and T2 cannot be expected to have the same result anyway: and worse in drive testing which is still a major part of overall testing, the testing system itself is not even at the same location at times T1 and T2. At this point we should mention that there is one very fundamental aspect common to
44、 conventional testing as well as to MCKPI-inspired ones. This is, that the same radio network condition will produce different KPI results depending on timing. To explain, imagine a network having a coverage “hole“. Now assume a simple telephony drive test, in two differently “timed“ versions. ETSI
45、ETSI TR 102 779 V1.1.1 (2009-03) 8In version 1, we try to set up the connection when we are inside the “coverage hole“. Result will of course be a Failed Call. Now change the timing slightly such that we have established the connection before we reach the “hole“. In this case, our result will be a D
46、ropped Call. In STQ MOBILE meeting #18, a contribution (18TD10) was submitted - triggered by the MCKPI discussion - which deals with similar aspects, namely the question of statistical equivalence of different types of testing. This contribution shows results of numerical simulations on different dy
47、namics of radio network quality over time or space (the full content is provided in annex A for convenience). The conclusion of 18TD10 is: To get the same results for both methods, at least one of the following three requirements needs to be satisfied: - The KPIs are totally independent from each ot
48、her (and thus also from the radio environment). - The radio environment is constant during the measurements. - The radio environment varies so fast during the measurements that consecutive actions can be assumed to have a totally independent radio environment. Obviously the first condition can never
49、 be true since the course of testing action is directly influenced by the outcome of previous steps (this is however also true for conventional tests). The second condition is of course only a “rhetorical“ one. The third condition, however, can be assumed fulfilled . Even if a single test result is neither “time“ nor “space“ independent in the strict meaning, it can be assumed that having enough samples will eliminate such effects for an overall KPI expressing the probability of a given result. This does not mean the argument can be
