1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T P.1301 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (10/2017) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Telemeeting assessment Subjective quality evaluation of audio and audio
2、visual multiparty telemeetings Recommendation ITU-T P.1301 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal characterist
3、ics Series P.30 P.300 Reference systems Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 Methods for obje
4、ctive and subjective assessment of speech and video quality Series P.800 Audiovisual quality in multimedia services Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed
5、 media Series P.1200 Telemeeting assessment Series P.1300 Statistical analysis, evaluation and reporting guidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than speech and video Series P.1500 For further details, please refer
6、 to the list of ITU-T Recommendations. Rec. ITU-T P.1301 (10/2017) i Recommendation ITU-T P.1301 Subjective quality evaluation of audio and audiovisual multiparty telemeetings Summary Recommendation ITU-T P.1301 concerns subjective quality assessment of telemeeting systems that provide multiparty co
7、mmunication between distant locations, using audio-only, video-only, audiovisual, text-based or graphical means as communication modes. The term multiparty refers to more than two meeting participants who can be located at two or more locations. Evaluation of those systems can focus on audio-only, v
8、ideo-only or audiovisual quality aspects and non-interactive or conversational quality can be assessed. This Recommendation gives an overview of relevant aspects that need to be considered for subjective quality evaluation of multiparty telemeetings and provides guidance to recommendations describin
9、g the details of applicable methods and procedures. Aspects in this Recommendation are also applicable to two-party telemeetings. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T P.1301 2012-07-14 12 11.1002/1000/11687 2.0 ITU-T P.1301 2017-10-29 12 11.1002/1000/13403 Keyword
10、s Audio, audiovisual, asymmetric connections, conversation tests, guidance, multiparty conferencing, non-interactive tests, subjective quality evaluation, telemeetings, test methods, 3D audio and video. * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your w
11、eb browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/11830-en. ii Rec. ITU-T P.1301 (10/2017) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communi
12、cation technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The
13、World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1.
14、 In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a re
15、cognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. Th
16、e words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility
17、that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommen
18、dation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information an
19、d are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2017 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.1301 (10/2017) iii Table of Contents Page
20、 1 Scope . 1 2 References . 2 3 Definitions 3 3.1 Terms defined elsewhere 3 3.2 Terms defined in this Recommendation . 3 4 Abbreviations and acronyms 5 5 Conventions 5 6 General recommendations concerning the subjective quality evaluation of multiparty telemeetings 5 7 Multiparty specific aspects in
21、 subjective quality evaluation 6 8 Guidance to suitable test methods 6 8.1 Test method decision criteria 7 8.2 Flow charts to be used when selecting test methods 8 Annex A Set-up of a multiparty telemeeting assessment test . 14 A.1 Assessment of conversational quality Conversation tests . 14 A.2 Ass
22、essment of non-interactive quality Non-interactive tests 16 Annex B Assessment of telemeetings with text-based communication and graphical information means (e.g., web conferencing) 18 Annex C Assessment of video-only telemeetings. 20 Annex D Effect of transmission delays on telemeeting quality 21 D
23、.1 Background . 21 D.2 Existing test task recommendations . 21 D.3 Recommended test tasks . 22 D.4 Set-up of a delay test 22 D.5 Test subjects . 23 D.6 Training session 23 D.7 Instructions . 23 D.8 Test questions . 23 D.9 Objective measurements . 24 D.10 Effects of delay . 25 Annex E Assessment of s
24、patial audio and 3D video reproduction of multiparty telemeetings 26 E.1 General remarks 26 E.2 Recommendations concerning spatial audio 26 E.3 Remarks concerning 3D video . 26 Annex F Assessment of asymmetric multiparty telemeetings 27 F.1 Problem statement 27 iv Rec. ITU-T P.1301 (10/2017) Page F.
25、2 Translation of technical degradations into possible perceptual impairments 28 F.3 Influencing factors 30 F.4 Experiment design 31 F.5 Rating scales . 32 F.6 Test stimuli . 33 F.7 Data analysis . 33 Annex G Assessment of multiparty telemeetings with non-stationary quality . 36 Annex H Assessment of
26、 multiparty telemeetings using multi-dimensional scaling methods . 37 Appendix I Influential factors . 38 Appendix II Overview of multiparty non-interactive test stimuli and conversation test tasks 40 II.1 Non-interactive audio-only stimuli: . 40 II.2 Non-interactive video-only stimuli . 40 II.3 Non
27、-interactive audiovisual stimuli . 40 II.4 Audio-only conversation tasks . 40 II.5 Audiovisual conversation tasks 41 Appendix III Examples of multiparty conversation test tasks (audio-only and audiovisual): Free conversation . 42 Appendix IV Examples of multiparty conversation test tasks (audio-only
28、): Three-party conversation test scenarios 3CTs 44 IV.1 Introduction 44 IV.2 Test scenario development . 44 IV.3 Scenario validation . 45 IV.4 Cultural aspects 46 Appendix V Examples of multiparty conversation test tasks (audiovisual): Audiovisual multi-point tasks for three parties (Survival task,
29、Leavitt task, Brainstorming task) 47 V.1 Overview and most suitable task 47 V.2 Leavitt task . 47 V.3 Brainstorming task 48 V.4 Survival task . 48 Appendix VI Additional proposals for multiparty conversation test tasks (audiovisual): Extended survival tasks scenarios and celebrity name guessing task
30、 . 61 VI.1 Overview and background 61 VI.2 Modification of survival scenarios from Appendix V 61 VI.3 Three additional survival scenarios 68 VI.4 Celebrity name guessing task . 74 Appendix VII Additional proposals for multiparty conversation test tasks (audiovisual): Formal and informal multiparty v
31、ideo conferences 75 Rec. ITU-T P.1301 (10/2017) v Page Appendix VIII Overview of documents describing suitable test methods . 76 VIII.1 Baseline test methods on which the current Recommendation is based 76 VIII.2 Multiparty specific recommendations to adapt the baseline test methods . 77 VIII.3 Furt
32、her test methods, which are referred to in the annexes of this Recommendation 77 Bibliography. 78 Rec. ITU-T P.1301 (10/2017) 1 Recommendation ITU-T P.1301 Subjective quality evaluation of audio and audiovisual multiparty telemeetings 1 Scope This Recommendation concerns subjective quality assessmen
33、t of telemeeting systems that provide communication between multiple parties at remote locations. As multiparty telemeetings can differ in a large number of aspects, the assessment of such systems requires careful selection and control of the considered aspects and a precise description when reporti
34、ng results. The main aspects defining the scope of this Recommendation are: Number of participants and number of locations The term multiparty refers to more than two meeting participants who can be located at two or more locations. Hence several multiparty situations are possible on which methodolo
35、gical aspects could depend: two sites with more than one person at at least one site (multiparty point-to-point), more than two sites with one person at each site (multiparty one-per-site), and more than two sites with more than one person at at least one site (multiparty multi-point). Communication
36、 mode and rendering conditions The telemeeting systems considered in this Recommendation can provide audio-only, video-only (i.e., sign language or lip reading) or audiovisual communication. Telemeeting systems can render communication modes using different techniques such as mono channel vs. spatia
37、l sound reproduction or 2D video vs. 3D video display. Furthermore, web conferencing applications are considered as telemeeting systems that can provide additional text-based (chat, e-mail, etc.) and graphical information means (presentation slides, etc.). Evaluation mode and type of quality Evaluat
38、ion of multiparty telemeeting systems can focus on audio-only, video-only or audiovisual quality aspects and it can assess non-interactive or conversational/interactive quality. Hence, the assessment of multiparty telemeetings quality can be organized along five communication modes (audio, video, au
39、diovisual, text-based, graphical), three test modes (audio, video, audiovisual), and two types of qualities (non-interactive, conversational/interactive). Controlled and non-controlled environments Assessment tests can be conducted in a laboratory or in a real-life environment where the system is su
40、pposed to be used, for example, in a telepresence room. This Recommendation concerns testing in both controlled and non-controlled environments. Accordingly the test environments should be properly described and specified when reporting the test results. Symmetric and asymmetric set-ups All telemeet
41、ing participants can be connected with the same type of equipment (symmetric) or different (asymmetric) equipment types. This Recommendation concerns testing of both symmetric and asymmetric telemeeting set-ups. At the moment of writing this Recommendation, a number of ITU-T and ITU-R Recommendation
42、s are in force describing subjective quality evaluation methods; each of those methods focusing on individual communication modes, test modes or types of quality. Table 1 provides a corresponding overview. Note that in order to evaluate several quality aspects of a system intended for multiparty tel
43、emeetings several assessment tests may be required. 2 Rec. ITU-T P.1301 (10/2017) Table 1 Focus of ITU-T and ITU-R Recommendations in terms of type of quality, communication mode and test mode Type of quality Communication Mode Test Mode ITU-T/ITU-R Recommendations Non-interactive Audio-only Audio P
44、.800, P.806, P.880, BS.1116, BS.1285, BS.1534, P.1302, P.1310, P.1311 Video-only Video P.910, BT.500, BT.710, BT.1788, P.915, P.916 Audiovisual Audio P.800, P.880, BS.1116, BS.1285, BS.1534, P.913 Video P.910, BT.500, BT.710, BT.1788, P.913 Audiovisual P.911, P.1302, P.913 Text Video Graphical Video
45、 Conversational/Interactive Audio-only Audio P.800, P.805, P.1305, P.1310, P.1312 Video-only Video Audiovisual Audio P.800, P.805, P.1310, P.1312 Video Audiovisual P.920, P.1305 Text Video Graphical Video 2 References The following ITU-T Recommendations and other references contain provisions which,
46、 through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to investigate the possibility of appl
47、ying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T
48、 P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.805 Recommendation ITU-T P.805 (2007), Subjective evaluation of conversational quality. ITU-T P.806 Recommendation ITU-T P.806 (2014), A subjective quality test methodology using multiple
49、rating scales. ITU-T P.880 Recommendation ITU-T P.880 (2004), Continuous evaluation of time-varying speech quality. ITU-T P.910 Recommendation ITU-T P.910 (2008), Subjective video quality assessment methods for multimedia applications. ITU-T P.911 Recommendation ITU-T P.911 (1998), Subjective audiovisual quality assessment methods for multimedia applications. Rec. ITU-T P.1301 (10/2017) 3 ITU-T P.913 Recommendation ITU-T P.913 (2016), Methods for the subjective assessment of
