ITU-T G 1011-2015 Reference guide to quality of experience assessment methodologies (Study Group 12)《经验评估方法质量的参考指南(研究组12)》.pdf

1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T G.1011 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (06/2015) SERIES G: TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS AND NETWORKS Multimedia Quality of Service and performance Generic and user-related aspects Referenc

2、e guide to quality of experience assessment methodologies Recommendation ITU-T G.1011 ITU-T G-SERIES RECOMMENDATIONS TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS AND NETWORKS INTERNATIONAL TELEPHONE CONNECTIONS AND CIRCUITS G.100G.199 GENERAL CHARACTERISTICS COMMON TO ALL ANALOGUE CARRIER-TRANSMI

3、SSION SYSTEMS G.200G.299 INDIVIDUAL CHARACTERISTICS OF INTERNATIONAL CARRIER TELEPHONE SYSTEMS ON METALLIC LINES G.300G.399 GENERAL CHARACTERISTICS OF INTERNATIONAL CARRIER TELEPHONE SYSTEMS ON RADIO-RELAY OR SATELLITE LINKS AND INTERCONNECTION WITH METALLIC LINES G.400G.449 COORDINATION OF RADIOTEL

4、EPHONY AND LINE TELEPHONY G.450G.499 TRANSMISSION MEDIA AND OPTICAL SYSTEMS CHARACTERISTICS G.600G.699 DIGITAL TERMINAL EQUIPMENTS G.700G.799 DIGITAL NETWORKS G.800G.899 DIGITAL SECTIONS AND DIGITAL LINE SYSTEM G.900G.999 MULTIMEDIA QUALITY OF SERVICE AND PERFORMANCE GENERIC AND USER-RELATED ASPECTS

5、 G.1000G.1999 TRANSMISSION MEDIA CHARACTERISTICS G.6000G.6999 DATA OVER TRANSPORT GENERIC ASPECTS G.7000G.7999 PACKET OVER TRANSPORT ASPECTS G.8000G.8999 ACCESS NETWORKS G.9000G.9999 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T G.1011 (06/2015) i Recommendation

6、ITU-T G.1011 Reference guide to quality of experience assessment methodologies Summary Recommendation ITU-T G.1011 provides a reference guide to existing standards for quality of experience (QoE) assessment methodologies. This Recommendation specifies QoE assessment approaches and classifications of

7、 different applications, and identifies a taxonomy of QoE assessment standards with different technical categorizations. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T G.1011 2010-06-29 12 11.1002/1000/10864 2.0 ITU-T G.1011 2013-05-14 12 11.1002/1000/11931 3.0 ITU-T G.1011

8、 2015-06-29 12 11.1002/1000/12507 Keywords Quality of experience, QoE. _ * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/ 11830-en. ii Rec. ITU-T G.1

9、011 (06/2015) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is respon

10、sible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the IT

11、U-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborativ

12、e basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mand

13、atory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. T

14、he use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU take

15、s no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual proper

16、ty, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2015 All rights reserved.

17、No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T G.1011 (06/2015) iii Table of Contents Page 1 Scope . 1 2 References . 1 3 Definitions 3 3.1 Terms defined elsewhere 3 3.2 Terms defined in this Recommendation . 3 4 Abbrev

18、iations and acronyms 4 5 Conventions 4 6 General approach to quality of experience assessment 4 6.1 Subjective assessment 6 6.2 Objective assessment 7 7 Target services 8 7.1 Audio 8 7.2 Video 8 7.3 Data . 8 8 Measurement approaches for different scenarios . 8 8.1 Media-layer models 9 8.2 Packet-lay

19、er models 9 8.3 Bitstream-layer models . 10 8.4 Hybrid models 10 8.5 Planning models . 10 9 Taxonomy . 10 Appendix I 13 Bibliography. 17 Rec. ITU-T G.1011 (06/2015) 1 Recommendation ITU-T G.1011 Reference guide to quality of experience assessment methodologies 1 Scope Quality of experience (QoE) is

20、a widely used term in the telecommunication industry. However, there is no single document available that describes methods for assessing QoE. This Recommendation provides a reference guide to existing standardized methods in ITU for QoE assessment. The overall approaches for the different assessmen

21、t methodologies typically used are described, and a taxonomy of QoE assessment standards is defined. For different applications, this Recommendation identifies suitable subjective assessment methodologies, and where applicable, objective assessment methods that can be used to estimate subjective opi

22、nion, in addition to giving guidance on their usage and limitations. The intended users of this Recommendation include telecommunication carriers, service providers and equipment manufacturers. 2 References The following ITU-T Recommendations and other references contain provisions which, through re

23、ference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to investigate the possibility of applying the mo

24、st recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T G.107 Reco

25、mmendation ITU-T G.107 (2015), The E-model: a computational model for use in transmission planning. ITU-T G.107.1 Recommendation ITU-T G.107.1 (2015), Wideband E-model. ITU-T G.1030 Recommendation ITU-T G.1030 (2014), Estimating end-to-end performance in IP networks for data applications. ITU-T G.10

26、70 Recommendation ITU-T G.1070 (2012), Opinion model for video-telephony applications. ITU-T J.140 Recommendation ITU-T J.140 (1998), Subjective picture quality assessment for digital cable television systems. ITU-T J.144 Recommendation ITU-T J.144 (2004), Objective perceptual video quality measurem

27、ent techniques for digital cable television in the presence of a full reference. ITU-T J.246 Recommendation ITU-T J.246 (2008), Perceptual visual quality measurement techniques for multimedia services over digital cable television networks in the presence of a reduced bandwidth reference. ITU-T J.24

28、7 Recommendation ITU-T J.247 (2008), Objective perceptual multimedia video quality measurement in the presence of a full reference. ITU-T J.249 Recommendation ITU-T J.249 (2010), Perceptual video quality measurement techniques for digital cable television in the presence of a reduced reference. 2 Re

29、c. ITU-T G.1011 (06/2015) ITU-T J.341 Recommendation ITU-T J.341 (2011), Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference. ITU-T J.342 Recommendation ITU-T J.342 (2011), Objective multimedia video quality measurement o

30、f HDTV for digital cable television in the presence of a reduced reference signal. ITU-T P.10 Amd.2 Recommendation ITU-T P.10/G.100 (2006) Amendment 2 (2008), New definitions for inclusion in Recommendation ITU-T P.10/G.100. ITU-T P.561 Recommendation ITU-T P.561 (2002), In-service non-intrusive mea

31、surement device Voice service measurements. ITU-T P.562 Recommendation ITU-T P.562 (2004), Analysis and interpretation of INMD voice-service measurements. ITU-T P.563 Recommendation ITU-T P.563 (2004), Single-ended method for objective speech quality assessment in narrow-band telephony applications.

32、 ITU-T P.564 Recommendation ITU-T P.564 (2007), Conformance testing for voice over IP transmission quality assessment models. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.805 Recommendation ITU-T P.805 (2007), Subjective evaluat

33、ion of conversational quality. ITU-T P.806 Recommendation ITU-T P.806 (2014), A subjective quality test methodology using multiple rating scales. ITU-T P.830 Recommendation ITU-T P.830 (1996), Subjective performance assessment of telephone-band and wideband digital codecs. ITU-T P.835 Recommendation

34、 ITU-T P.835 (2003), Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm. ITU-T P.862 Recommendation ITU-T P.862 (2001), Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow

35、-band telephone networks and speech codecs. ITU-T P.862.1 Recommendation ITU-T P.862.1 (2003), Mapping function for transforming P.862 raw result scores to MOS-LQO. ITU-T P.862.2 Recommendation ITU-T P.862.2 (2007), Wideband extension to Recommendation P.862 for the assessment of wideband telephone

36、networks and speech codecs. ITU-T P.863 Recommendation ITU-T P.863 (2014), Perceptual objective listening quality assessment. ITU-T P.910 Recommendation ITU-T P.910 (2008), Subjective video quality assessment methods for multimedia applications. ITU-T P.911 Recommendation ITU-T P.911 (1998), Subject

37、ive audiovisual quality assessment methods for multimedia applications. ITU-T P.920 Recommendation ITU-T P.920 (2000), Interactive test methods for audiovisual communications. Rec. ITU-T G.1011 (06/2015) 3 ITU-T P.1201.1 Recommendation ITU-T P.1201.1 (2012), Parametric non-intrusive assessment of au

38、diovisual media streaming quality Lower resolution application area. ITU-T P.1201.2 Recommendation ITU-T P.1201.2 (2012), Parametric non-intrusive assessment of audiovisual media streaming quality Higher resolution application area. ITU-T P.1202.1 Recommendation ITU-T P.1202.1 (2012), Parametric non

39、-intrusive bitstream assessment of video media streaming quality Lower resolution application area. ITU-T P.1202.2 Recommendation ITU-T P.1202.2 (2013), Parametric non-intrusive bitstream assessment of video media streaming quality Higher resolution application area. ITU-T P.1301 Recommendation ITU-

40、T P.1301 (2012), Subjective quality evaluation of audio and audiovisual multiparty telemeetings. ITU-T P.1302 Recommendation ITU-T P.1302 (2014), Subjective method for simulated conversation tests addressing speech and audiovisual call quality. ITU-T P.1311 Recommendation ITU-T P.1311 (2014), Method

41、 for determining the intelligibility of multiple concurrent talkers. ITU-T P.1501 Recommendation ITU-T P.1501 (2014), Subjective testing methodology for web browsing. ITU-R BS.1116-3 Recommendation ITU-R BS.1116-3 (2015), Methods for the subjective assessment of small impairments in audio systems. I

42、TU-R BS.1285 Recommendation ITU-R BS.1285 (1997), Pre-selection methods for the subjective assessment of small impairments in audio systems. ITU-R BS.1387-1 Recommendation ITU-R BS.1387-1 (2001), Method for objective measurements of perceived audio quality. ITU-R BS.1534-2 Recommendation ITU-R BS.15

43、34-2 (2014), Method for the subjective assessment of intermediate quality levels of coding systems. ITU-R BT.500-13 Recommendation ITU-R BT.500-13 (2012), Methodology for the subjective assessment of the quality of television pictures. 3 Definitions 3.1 Terms defined elsewhere This Recommendation us

44、es the following term defined elsewhere: 3.1.1 quality of experience (QoE) ITU-T P.10 Amd.2: The overall acceptability of an application or service, as perceived subjectively by the end-user. NOTE 1 Quality of experience includes the complete end-to-end system effects (client, terminal, network, ser

45、vices infrastructure, etc.). NOTE 2 Overall acceptability may be influenced by user expectations and context. 3.2 Terms defined in this Recommendation None. 4 Rec. ITU-T G.1011 (06/2015) 4 Abbreviations and acronyms This Recommendation uses the following abbreviations and acronyms: CIF Common Interm

46、ediate Format FR Full Reference HDTV High Definition Television HVGA Half Video Graphics Array IPTV Internet Protocol Television ISDN Integrated Services Digital Network MOS Mean Opinion Score MOSA,V,MM Mean Opinion Score for Audio, Video or Multimedia NB Narrowband NR No Reference PSTN Public Switc

47、hed Telephone Network QCIF Quarter Common Intermediate Format QoE Quality of Experience QVGA Quarter Video Graphics Array RR Reduced Reference SD Standard Definition SWB Super Wideband VGA Video Graphics Array VoIP Voice over Internet Protocol WB Wideband 5 Conventions This Recommendation uses the f

48、ollowing conventions: LAB: laboratory testing; MON: monitoring; PLN: planning. 6 General approach to quality of experience assessment In principle, assessment of quality of experience (QoE) must be performed using subjective tests with metrics, such as the mean opinion score (MOS). However, it is al

49、so possible and sometimes more convenient to estimate QoE based on objective testing and associated quality estimation models. Through different quality estimation models, it is possible to measure or calculate the objective parameters affecting QoE, in order to evaluate QoE. Subjective testing needs more resources and effort, because it requires human subjects. On the other hand, objective measurement and automatic calculation using app