1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T G.1011 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (07/2016) SERIES G: TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS AND NETWORKS Multimedia Quality of Service and performance Generic and user-related aspects Referenc
2、e guide to quality of experience assessment methodologies Recommendation ITU-T G.1011 ITU-T G-SERIES RECOMMENDATIONS TRANSMISSION SYSTEMS AND MEDIA, DIGITAL SYSTEMS AND NETWORKS INTERNATIONAL TELEPHONE CONNECTIONS AND CIRCUITS G.100G.199 GENERAL CHARACTERISTICS COMMON TO ALL ANALOGUE CARRIER-TRANSMI
3、SSION SYSTEMS G.200G.299 INDIVIDUAL CHARACTERISTICS OF INTERNATIONAL CARRIER TELEPHONE SYSTEMS ON METALLIC LINES G.300G.399 GENERAL CHARACTERISTICS OF INTERNATIONAL CARRIER TELEPHONE SYSTEMS ON RADIO-RELAY OR SATELLITE LINKS AND INTERCONNECTION WITH METALLIC LINES G.400G.449 COORDINATION OF RADIOTEL
4、EPHONY AND LINE TELEPHONY G.450G.499 TRANSMISSION MEDIA AND OPTICAL SYSTEMS CHARACTERISTICS G.600G.699 DIGITAL TERMINAL EQUIPMENTS G.700G.799 DIGITAL NETWORKS G.800G.899 DIGITAL SECTIONS AND DIGITAL LINE SYSTEM G.900G.999 MULTIMEDIA QUALITY OF SERVICE AND PERFORMANCE GENERIC AND USER-RELATED ASPECTS
5、 G.1000G.1999 TRANSMISSION MEDIA CHARACTERISTICS G.6000G.6999 DATA OVER TRANSPORT GENERIC ASPECTS G.7000G.7999 PACKET OVER TRANSPORT ASPECTS G.8000G.8999 ACCESS NETWORKS G.9000G.9999 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T G.1011 (07/2016) i Recommendation
6、ITU-T G.1011 Reference guide to quality of experience assessment methodologies Summary Recommendation ITU-T G.1011 provides a reference guide to existing standards for quality of experience (QoE) assessment methodologies. This Recommendation specifies QoE assessment approaches and classifications of
7、 different applications, and identifies a taxonomy of QoE assessment standards with different technical categorizations. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T G.1011 2010-06-29 12 11.1002/1000/10864 2.0 ITU-T G.1011 2013-05-14 12 11.1002/1000/11931 3.0 ITU-T G.1011
8、 2015-06-29 12 11.1002/1000/12507 4.0 ITU-T G.1011 2016-07-29 12 11.1002/1000/12966 Keywords QoE, Quality of experience. * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.i
9、tu.int/11.1002/1000/11830-en. ii Rec. ITU-T G.1011 (07/2016) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-
10、T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four ye
11、ars, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the nec
12、essary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. Howev
13、er, the Recommendation may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative
14、equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of
15、a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU
16、 had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu
17、.int/ITU-T/ipr/. ITU 2016 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T G.1011 (07/2016) iii Table of Contents Page 1 Scope . 1 2 References . 1 3 Definitions 4 3.1 Terms defined elsewhere 4 3.2 Te
18、rms defined in this Recommendation . 4 4 Abbreviations and acronyms 4 5 Conventions 5 6 Overview of QoE assessment methods . 5 7 Concept . 5 7.1 General approach to quality of experience assessment 5 7.2 Content dependency of QoE assessment 9 8 Target services 9 8.1 Audio 9 8.2 Video 9 8.3 Data . 10
19、 9 Measurement approaches for different scenarios . 10 9.1 Media-layer models 10 9.2 Packet-layer models 11 9.3 Bitstream-layer models . 11 9.4 Hybrid models 12 9.5 Planning models . 12 10 Taxonomy . 12 Appendix I 17 Appendix II 20 II.1 Metrics related to TCP based streaming video . 20 II.2 Metrics
20、related to UDP based streaming video 20 Bibliography. 21 Rec. ITU-T G.1011 (07/2016) 1 Recommendation ITU-T G.1011 Reference guide to quality of experience assessment methodologies 1 Scope Quality of experience (QoE) is a widely used term in the telecommunication industry. However, there is no singl
21、e document available that describes methods for assessing QoE. This Recommendation provides a reference guide to existing standardized methods in ITU for QoE assessment. The overall approaches for the different assessment methodologies typically used are described, and a taxonomy of QoE assessment s
22、tandards is defined. For different applications, this Recommendation identifies suitable subjective assessment methodologies, and where applicable, objective assessment methods that can be used to estimate subjective opinion, in addition to giving guidance on their usage and limitations. The intende
23、d users of this Recommendation include telecommunication carriers, service providers and equipment manufacturers. 2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time
24、of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A li
25、st of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T G.107 Recommendation ITU-T G.107 (2015), The E-model: a computational model for use in tra
26、nsmission planning. ITU-T G.107.1 Recommendation ITU-T G.107.1 (2015), Wideband E-model. ITU-T G.1030 Recommendation ITU-T G.1030 (2014), Estimating end-to-end performance in IP networks for data applications. ITU-T G.1070 Recommendation ITU-T G.1070 (2012), Opinion model for video-telephony applica
27、tions. ITU-T G.1071 Recommendation ITU-T G.1071 (2015), Opinion model for network planning of video and audio streaming applications. ITU-T J.140 Recommendation ITU-T J.140 (1998), Subjective picture quality assessment for digital cable television systems. ITU-T J.144 Recommendation ITU-T J.144 (200
28、4), Objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference. ITU-T J.246 Recommendation ITU-T J.246 (2008), Perceptual visual quality measurement techniques for multimedia services over digital cable television networks in the presen
29、ce of a reduced bandwidth reference. ITU-T J.247 Recommendation ITU-T J.247 (2008), Objective perceptual multimedia video quality measurement in the presence of a full reference. ITU-T J.249 Recommendation ITU-T J.249 (2010), Perceptual video quality measurement techniques for digital cable televisi
30、on in the presence of a reduced reference. 2 Rec. ITU-T G.1011 (07/2016) ITU-T J.341 Recommendation ITU-T J.341 (2016), Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference. ITU-T J.342 Recommendation ITU-T J.342 (2011), O
31、bjective multimedia video quality measurement of HDTV for digital cable television in the presence of a reduced reference signal. ITU-T J.343 Recommendation ITU-T J.343 (2014), Hybrid perceptual bitstream models for objective video quality measurements. ITU-T J.343.1 Recommendation ITU-T J.343.1 (20
32、14), Hybrid-NRe objective perceptual video quality measurement for HDTV and multimedia IP-based video services in the presence of encrypted bitstream data. ITU-T J.343.2 Recommendation ITU-T J.343.2 (2014), Hybrid-NR objective perceptual video quality measurement for HDTV and multimedia IP-based vid
33、eo services in the presence of non-encrypted bitstream data. ITU-T J.343.3 Recommendation ITU-T J.343.3 (2014), Hybrid-RRe objective perceptual video quality measurement for HDTV and multimedia IP-based video services in the presence of a reduced reference signal and encrypted bitstream data. ITU-T
34、J.343.4 Recommendation ITU-T J.343.4 (2014), Hybrid-RR objective perceptual video quality measurement for HDTV and multimedia IP-based video services in the presence of a reduced reference signal and non-encrypted bitstream data. ITU-T J.343.5 Recommendation ITU-T J.343.5 (2014), Hybrid-FRe objectiv
35、e perceptual video quality measurement for HDTV and multimedia IP-based video services in the presence of a full reference signal and encrypted bitstream data. ITU-T J.343.6 Recommendation ITU-T J.343.6 (2014), Hybrid-FR objective perceptual video quality measurement for HDTV and multimedia IP-based
36、 video services in the presence of a full reference signal and non-encrypted bitstream data. ITU-T P.561 Recommendation ITU-T P.561 (2002), In-service non-intrusive measurement device Voice service measurements. ITU-T P.562 Recommendation ITU-T P.562 (2004), Analysis and interpretation of INMD voice
37、-service measurements. ITU-T P.563 Recommendation ITU-T P.563 (2004), Single-ended method for objective speech quality assessment in narrow-band telephony applications. ITU-T P.564 Recommendation ITU-T P.564 (2007), Conformance testing for voice over IP transmission quality assessment models. ITU-T
38、P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.800.1 Recommendation ITU-T P.800.1 (2016), Mean opinion score (MOS) terminology. ITU-T P.800.2 Recommendation ITU-T P.800.2 (2016), Mean opinion score interpretation and reporting. ITU-T P.
39、805 Recommendation ITU-T P.805 (2007), Subjective evaluation of conversational quality. ITU-T P.806 Recommendation ITU-T P.806 (2014), A subjective quality test methodology using multiple rating scales. Rec. ITU-T G.1011 (07/2016) 3 ITU-T P.830 Recommendation ITU-T P.830 (1996), Subjective performan
40、ce assessment of telephone-band and wideband digital codecs. ITU-T P.835 Recommendation ITU-T P.835 (2003), Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm. ITU-T P.862 Recommendation ITU-T P.862 (2001), Perceptual evaluation of speech
41、 quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs. ITU-T P.862.1 Recommendation ITU-T P.862.1 (2003), Mapping function for transforming P.862 raw result scores to MOS-LQO. ITU-T P.862.2 Recommendation ITU-T P.862.2 (2007
42、), Wideband extension to Recommendation P.862 for the assessment of wideband telephone networks and speech codecs. ITU-T P.863 Recommendation ITU-T P.863 (2014), Perceptual objective listening quality assessment. ITU-T P.910 Recommendation ITU-T P.910 (2008), Subjective video quality assessment meth
43、ods for multimedia applications. ITU-T P.911 Recommendation ITU-T P.911 (1998), Subjective audiovisual quality assessment methods for multimedia applications. ITU-T P.920 Recommendation ITU-T P.920 (2000), Interactive test methods for audiovisual communications. ITU-T P.1201.1 Recommendation ITU-T P
44、.1201.1 (2012), Parametric non-intrusive assessment of audiovisual media streaming quality Lower resolution application area. ITU-T P.1201.2 Recommendation ITU-T P.1201.2 (2012), Parametric non-intrusive assessment of audiovisual media streaming quality Higher resolution application area. ITU-T P.12
45、02.1 Recommendation ITU-T P.1202.1 (2012), Parametric non-intrusive bitstream assessment of video media streaming quality Lower resolution application area. ITU-T P.1202.2 Recommendation ITU-T P.1202.2 (2013), Parametric non-intrusive bitstream assessment of video media streaming quality Higher reso
46、lution application area. ITU-T P.1301 Recommendation ITU-T P.1301 (2012), Subjective quality evaluation of audio and audiovisual multiparty telemeetings. ITU-T P.1302 Recommendation ITU-T P.1302 (2014), Subjective method for simulated conversation tests addressing speech and audiovisual call quality
47、. ITU-T P.1311 Recommendation ITU-T P.1311 (2014), Method for determining the intelligibility of multiple concurrent talkers. ITU-T P.1312 Recommendation ITU-T P.1312 (2016), Method for the measurement of the communication effectiveness of multiparty telemeetings using task performance. ITU-T P.1501
48、 Recommendation ITU-T P.1501 (2014), Subjective testing methodology for web browsing. 4 Rec. ITU-T G.1011 (07/2016) ITU-R BS.1116-3 Recommendation ITU-R BS.1116-3 (2015), Methods for the subjective assessment of small impairments in audio systems. ITU-R BS.1285 Recommendation ITU-R BS.1285 (1997), P
49、re-selection methods for the subjective assessment of small impairments in audio systems. ITU-R BS.1387-1 Recommendation ITU-R BS.1387-1 (2001), Method for objective measurements of perceived audio quality. ITU-R BS.1534-3 Recommendation ITU-R BS.1534-3 (2015), Method for the subjective assessment of intermediate quality level of audio systems. ITU-R BT.500-13 Recommendation ITU-R BT.500-13 (2012), Methodology for the subjective assessment o
