1、 International Telecommunication Union ITU-T P.1301TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (07/2012) SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Telemeeting assessment Subjective quality evaluation of audio and audiovisual multiparty telemeetings Recommendation ITU-T
2、P.1301 ITU-T P-SERIES RECOMMENDATIONS TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal characteristics Series P.30 P.300 Reference systems Series P.40 Objective measuring
3、 apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 P.800 Audiovisual quality in multimedia services Series P.900 Transmission performance
4、and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting assessment Series P.1300Statistical analysis, evaluation and reporting guidelines of quality measurements Series P.1400 F
5、or further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P.1301 (07/2012) i Recommendation ITU-T P.1301 Subjective quality evaluation of audio and audiovisual multiparty telemeetings Summary Recommendation ITU-T P.1301 concerns subjective quality assessment of telemeeting sy
6、stems that provide multiparty communication between distant locations, using audio-only, video-only, audiovisual, text-based or graphical means as communication modes. The term multiparty refers to more than two meeting participants who can be located at two or more than two locations. Evaluation of
7、 those systems can focus on audio-only, video-only, or audiovisual quality aspects and non-interactive or conversational quality can be assessed. This Recommendation gives an overview of relevant aspects that need to be considered for subjective quality evaluation of multiparty telemeetings and it p
8、rovides guidance to Recommendations describing the details of applicable methods and procedures. Aspects in this Recommendation are also applicable to two-party telemeetings. History Edition Recommendation Approval Study Group 1.0 ITU-T P.1301 2012-07-14 12 ii Rec. ITU-T P.1301 (07/2012) FOREWORD Th
9、e International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying techni
10、cal, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, i
11、n turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC.
12、 NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensu
13、re, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does
14、 not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning
15、 the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents
16、, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2013 All rights reserved. No part of this publicat
17、ion may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.1301 (07/2012) iii Table of Contents Page 1 Scope 1 2 References. 2 3 Definitions 3 3.1 Terms defined elsewhere 3 3.2 Terms defined in this Recommendation . 3 4 Abbreviations and acronyms 4 5 Co
18、nventions 4 6 General Recommendations concerning the subjective quality evaluation of multiparty telemeetings 5 7 Multiparty specific aspects in subjective quality evaluation 5 8 Guidance to suitable test methods 6 8.1 Test method decision criteria 6 8.2 Flow charts to be used when selecting test me
19、thods 7 Annex A Set-up of a multiparty telemeeting assessment test . 13 A.1 Assessment of conversational quality Conversation tests . 13 A.2 Assessment of non-interactive quality Non-interactive tests 15 Annex B Assessment of telemeetings with text-based communication and graphical information means
20、 (e.g., web conferencing) . 17 Annex C Assessment of video-only telemeetings. 19 Annex D Effect of transmission delays on telemeeting quality 20 D.1 Background . 20 D.2 Existing test task Recommendations 20 D.3 Recommended test tasks . 21 D.4 Set-up of a delay test 21 D.5 Test subjects . 22 D.6 Trai
21、ning session 22 D.7 Instructions . 22 D.8 Test questions . 22 D.9 Objective measurements . 23 D.10 Effects of delay . 23 Annex E Assessment of 3D audio and 3D video reproduction of multiparty telemeetings 25 Annex F Assessment of asymmetric multiparty telemeetings 26 F.1 Overview 26 F.2 Interactions
22、 of different group sizes and different communication modes 26 F.3 Remarks concerning the experiment design . 26 F.4 Remarks concerning scales . 27 iv Rec. ITU-T P.1301 (07/2012) Page F.5 Remarks concerning data analysis 27 Annex G Assessment of multiparty telemeetings with non-stationary quality .
23、28 Annex H Assessment of multiparty telemeetings using multi-dimensional scaling methods . 29 Appendix I Influential factors . 30 Appendix II Overview of multiparty non-interactive test stimuli and conversation test tasks 32 II.1 Non-interactive audio-only stimuli . 32 II.2 Non-interactive video-onl
24、y stimuli . 32 II.3 Non-interactive audiovisual stimuli . 32 II.4 Audio-only conversation tasks . 32 II.5 Audiovisual conversation tasks 33 Appendix III Examples of multiparty conversation test tasks (audio-only and audiovisual): Free conversation 34 Appendix IV Examples of multiparty conversation t
25、est tasks (audio-only): Three-party conversation test scenarios (3CTs) . 37 IV.1 Introduction 37 IV.2 Test scenario development . 37 IV.3 Scenario validation . 38 IV.4 Cultural aspects 39 Appendix V Examples of multiparty conversation test tasks (audiovisual): Audiovisual multi-point tasks for three
26、 parties (Survival task, Leavitt task, Brainstorming task) 40 V.1 Overview and most suitable task 40 V.2 Leavitt task . 40 V.3 Brainstorming task 41 V.4 Survival task . 41 Appendix VI Additional proposals for multiparty conversation test tasks (audiovisual): Formal and informal multiparty video conf
27、erences 54 Appendix VII Overview of documents describing suitable test methods . 55 Bibliography. 57 Rec. ITU-T P.1301 (07/2012) 1 Recommendation ITU-T P.1301 Subjective quality evaluation of audio and audiovisual multiparty telemeetings 1 Scope This Recommendation concerns subjective quality assess
28、ment of telemeeting systems that provide communication between multiple parties at remote locations. As multiparty telemeetings can differ in a large number of aspects, the assessment of such systems requires a proper selection and control of the considered aspects and a precise description when rep
29、orting results. The main aspects defining the scope of this Recommendation are: Number of participants and number of locations The term multiparty refers to more than two meeting participants who can be located at two, or more than two, locations. Hence, several multiparty situations are possible on
30、 which methodological aspects could depend: two sites with more than one person at at least one site (multiparty point-to-point), more than two sites with one person at each site (multiparty one-per-site), and more than two sites with more than one person at at least one site (multiparty multi-point
31、). Communication mode and rendering conditions The telemeeting systems considered in this Recommendation can provide audio-only, video-only (i.e., sign language or lip reading), or audiovisual communication. Telemeeting systems can render communication modes using different techniques such as mono c
32、hannel vs. spatial sound reproduction or 2D video vs. 3D video display. Furthermore, web conferencing applications are considered as telemeeting systems that can provide additional text-based (chat, e-mail, etc.), and graphical information means (presentation slides, etc.). Evaluation mode and type
33、of quality Evaluation of multiparty telemeeting systems can focus on audio-only, video-only, or audiovisual quality aspects and it can assess non-interactive or conversational/interactive quality. Hence, the assessment of multiparty telemeetings quality can be organized along five communication mode
34、s (audio, video, audiovisual, text-based, graphical), three test modes (audio, video, audiovisual), and two types of qualities (non-interactive, conversational/interactive). Controlled and non-controlled environments Assessment tests can be conducted in a laboratory or in a real-life environment whe
35、re the system is supposed to be used, for example, in a telepresence room. This Recommendation concerns testing in both controlled and not controlled environments. Accordingly, the test environments should be properly described and specified when reporting the test results. Symmetric and asymmetric
36、set-ups All telemeeting participants can be connected with the same type of equipment (symmetric) or different types of equipment (asymmetric). This Recommendation concerns testing of both symmetric and asymmetric telemeeting set-ups. At the moment of writing this Recommendation, a number of ITU-T a
37、nd ITU-R Recommendations are in force describing subjective quality evaluation methods; each of those methods focusing on individual communication modes, test modes, or types of quality. Table 1 provides a corresponding overview. Note that in order to evaluate several quality aspects of a system int
38、ended for multiparty telemeetings, several assessment tests might be needed to execute. 2 Rec. ITU-T P.1301 (07/2012) Table 1 Focus of ITU-T and ITU-R Recommendations in terms of type of quality, communication mode and test mode Type of quality Communication Mode Test Mode ITU-T/ITU-R Recommendation
39、s Non-interactive Audio Audio ITU T P.800, ITU-T P.880 ITU-R BS.1116, ITU-R BS.1285, ITU-R BS.1534 Video Video ITU-T P.910 ITU-R BT.500, ITU-R BT.710, ITU-R BT.1788 Audiovisual Audio ITU-T P.800, ITU-T P.880 ITU-R BS.1116, ITU-R BS.1285, ITU-R BS.1534 Video ITU-T P.910 ITU-R BT.500, ITU-R BT.710, IT
40、U-R BT.1788 Audiovisual ITU-T P.911 Text Video Graphical Video Audio Audio ITU-T P.800, ITU-T P.805 Video Video Audiovisual Audio ITU-T P.800, ITU-T P.805 Conversational/Interactive Video Audiovisual ITU-T P.920 Text Video Graphical Video 2 References The following ITU-T Recommendations and other re
41、ferences contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to i
42、nvestigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the
43、 status of a Recommendation. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.805 Recommendation ITU-T P.805 (2007), Subjective evaluation of conversational quality. ITU-T P.880 Recommendation ITU-T P.880 (2004), Continuous evaluati
44、on of time-varying speech quality. ITU-T P.910 Recommendation ITU-T P.910 (2008), Subjective video quality assessment methods for multimedia applications. ITU-T P.911 Recommendation ITU-T P.911 (1998), Subjective audiovisual quality assessment methods for multimedia applications. Rec. ITU-T P.1301 (
45、07/2012) 3 ITU-T P.920 Recommendation ITU-T P.920 (2000), Interactive test methods for audiovisual communications. ITU-R BS.1116 Recommendation ITU-R BS.1116-1 (10/97), Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems. ITU-R BS.1285 Rec
46、ommendation ITU-R BS.1285 (10/97), Pre-selection methods for the subjective assessment of small impairments in audio systems. ITU-R BS.1534 Recommendation ITU-R BS.1534-1 (01/03), Method for the subjective assessment of intermediate quality levels of coding systems. ITU-R BT.500 Recommendation ITU-R
47、 BT.500-13 (01.12), Methodology for the subjective assessment of the quality of television pictures. ITU-R BT.710 Recommendation ITU-R BT.710-4 (11.98), Subjective assessment methods for image quality in high-definition television. ITU-R BT.1788 Recommendation ITU-R BT.1788 (01/07), Methodology for
48、the subjective assessment of video quality in multimedia applications. 3 Definitions 3.1 Terms defined elsewhere None. 3.2 Terms defined in this Recommendation This Recommendation defines the following terms: 3.2.1 communication mode: The mode that the system under test is providing for the communic
49、ation between telemeeting participants. It can be audio-only, video-only (for hearing-impaired) or audiovisual. 3.2.2 conversational quality: The perceived quality when two or more test participants have a conversation. 3.2.3 interactive quality: Synonym for conversational quality. This term might appear to be more appropriate when considering video-only communications, e.g., sign language or lip reading communication, while the term conversational quality is usually used in the referen
