1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T P.1312 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (02/2016) SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Telemeeting assessment Method for the measurement of the communication effectiveness of mu
2、ltiparty telemeetings using task performance Recommendation ITU-T P.1312 ITU-T P-SERIES RECOMMENDATIONS TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal characteristics S
3、eries P.30 P.300 Reference systems Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 P.800 Audiovisual qua
4、lity in multimedia services Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting assessment Series P.1300 Statistical analysis, evaluati
5、on and reporting guidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than voice services Series P.1500 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P.1312 (02/2016) i Recommendation ITU-T
6、P.1312 Method for the measurement of the communication effectiveness of multiparty telemeetings using task performance Summary Recommendation ITU-T P.1312 describes a subjective test method for quantifying the effectiveness of telemeeting systems in conveying information in multiparty conversation s
7、cenarios. This method utilizes a predefined set of tasks designed to provoke rapid turn-taking and concurrent talking among participants. The goal of these tasks is to stress conferencing systems in order to clearly and measurably demonstrate their limits. The method measures the rate at which multi
8、ple participants exchange information to assess the effectiveness of communication systems compared to face-to-face communication. This constitutes an objectively-measured performance metric providing stable and repeatable results across different laboratories and that is highly discriminative acros
9、s various types of system properties. This method may be used to evaluate individual subcomponents of a telecommunication system or the system as a whole. The proposed method is especially sensitive to the ability to transmit multiple concurrent voices and maintain their independence. The test has d
10、emonstrated sensitivity to system parameters including duplex capability, spatialization, bandwidth and delay. As such, it is suitable for the assessment of multiparty telemeeting systems (see Recommendation ITU-T P.1301 for general guidelines and additional methods). History Edition Recommendation
11、Approval Study Group Unique ID* 1.0 ITU-T P.1312 2016-02-29 12 11.1002/1000/12751 Keywords Double-talk, face-to-face reference, interactive test method, system effectiveness, task performance measurement, telemeeting. * To access the Recommendation, type the URL http:/handle.itu.int/ in the address
12、field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/11830-en. ii Rec. ITU-T P.1312 (02/2016) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, informat
13、ion and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldw
14、ide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTS
15、A Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administ
16、ration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisi
17、ons are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to t
18、he possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside
19、of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest
20、 information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2016 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.1312 (02/2016) iii Table o
21、f Contents Page 1 Scope . 1 2 References . 1 3 Definitions 1 3.1 Terms defined elsewhere 1 3.2 Terms defined in this Recommendation . 2 4 Abbreviations and acronyms 2 5 Conventions 2 6 Task effectiveness test method . 2 6.1 Purpose . 2 6.2 Test principle 3 6.3 Reference condition and test scenarios
22、. 4 6.4 Test facilities and equipment 6 6.5 Test administrators . 6 6.6 Subjects . 7 6.7 Experiment design 7 6.8 Task performance measurement . 8 Annex I Construction of the pool of words for the word task 10 Appendix I Example of instructions . 11 Appendix II Example approaches subjects may use to
23、complete the tasks 13 Appendix III Example test results . 14 Bibliography. 16 Rec. ITU-T P.1312 (02/2016) 1 Recommendation ITU-T P.1312 Method for the measurement of the communication effectiveness of multiparty telemeetings using task performance 1 Scope This Recommendation describes a method for q
24、uantifying the amount of information that multiple interlocutors correctly exchange over a system. The measure serves to assess the effectiveness of a voice communication system by comparing the amount of correct information exchanged over the system relative to the amount of correct information the
25、se interlocutors achieve in face-to-face communication. This metric qualitatively evaluates a multiparty conferencing systems ability to transmit multiple voices and maintain their independence. The method has demonstrated sensitivity to system parameters including duplex capability, spatialization,
26、 bandwidth and delay. The method described herein applies only to voice communication; audiovisual communications is not covered by the scope of this Recommendation. The method does not include the measurement of technical properties of the systems. 2 References The following ITU-T Recommendations a
27、nd other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore enco
28、uraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone do
29、cument, the status of a Recommendation. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.805 Recommendation ITU-T P.805 (2007), Subjective evaluation of conversational quality. ITU-R BS.1116 Recommendation ITU-T BS.1116-3 (2015), Me
30、thods for the subjective assessment of small impairments in audio systems. 3 Definitions 3.1 Terms defined elsewhere This Recommendation uses the following terms defined elsewhere: 3.1.1 concurrent talk b-ITU-T P.1311: When far-end speech of two or more talkers occurs simultaneously at a given point
31、, typically the near end terminal. 3.1.2 multiparty b-ITU-T P.1301: More than two persons. Example: More than two persons are participating in a telemeeting, having a conversation, performing a test task together, etc. The term multiparty does not specify if the persons are distributed across two or
32、 more locations. If not explicitly stated differently, multiparty implicates that the persons are at two or more than two locations. When further specification is necessary, additional terms will be used or the number of locations will be explicitly stated. 2 Rec. ITU-T P.1312 (02/2016) 3.1.3 one pe
33、r site b-ITU-T P.1301: One person per connected location. Example: In a multiparty one-per-site telemeeting, more than two sites are connected with only one person present at each site. 3.2 Terms defined in this Recommendation This Recommendation defines the following terms: 3.2.1 performance score:
34、 The number of correct sequences conveyed by an individual, team or group. 3.2.2 system effectiveness: The ratio, expressed as a percentage, between the performance score achieved on a given system and the performance score obtained in face-to-face communication. 3.2.3 reader: A participant for whom
35、 the role is to read out loud a sequence of letters or words to his or her partner(s). 3.2.4 responder: A participant for whom the role is to register the sequence of letters or words his or her partner communicates. 3.2.5 face-to-face meeting: A meeting for which all participants are collocated. 3.
36、2.6 duplex: A communication channel for which signals can flow in both directions. 3.2.7 spatialization: The process of rendering audio sources in a three-dimensional environment. 4 Abbreviations and acronyms This Recommendation uses the following abbreviations and acronyms: SuT System-under-Test 5
37、Conventions The keywords “is required to“ indicate a requirement which must be strictly followed and from which no deviation is permitted if conformance to this Recommendation is to be claimed. The keywords “is recommended“ indicate a requirement which is recommended but which is not absolutely requ
38、ired. Thus this requirement need not be present to claim conformance. The keywords “is prohibited from“ indicate a requirement which must be strictly followed and from which no deviation is permitted if conformance to this Recommendation is to be claimed. The keywords “can optionally“ indicate an op
39、tional requirement which is permissible, without implying any sense of being recommended. This term is not intended to imply that the vendors implementation must provide the option and the feature can be optionally enabled by the network operator/service provider. Rather, it means the vendor may opt
40、ionally provide the feature and still claim conformance with this Recommendation. 6 Task effectiveness test method 6.1 Purpose The purpose of the Recommendation is to enable multiparty conversational tests of voice-only communication systems that are sensitive to subtle changes of system parameters,
41、 to produce results that are repeatable in different laboratories with predictable test duration. The test described in this Recommendation is based on the objectively-measured performance of participants when solving tasks over the communication system-under-test (SuT). This is unlike the conversat
42、ional tests of ITU-T P.805, which are based on subjective impressions reported by Rec. ITU-T P.1312 (02/2016) 3 participants. The method provides an objective performance measure that reduces the mutual influence of test conditions between experiments. Another important aspect of this test is that i
43、t relates the performance achieved using the SuT to the performance achieved through (in-person) face-to-face conversation. Using in-person meetings as the reference not only minimizes individual biases, but also leads to a metric that is directly relevant to a predominant use case of telecommunicat
44、ion systems, which is to substitute for in-person meetings. By measuring the rate of information exchange attained when using the SuT and relating it to the information exchange rate attained through in-person meetings, a measure of relative communication effectiveness is obtained that is easily int
45、erpretable and directly relevant to the systems intended use. The tasks used in the test are highly structured and designed to be challenging, which serves two purposes. Firstly, it emphasizes the aspects of a communication system that can only be assessed in conversational tests (as opposed to list
46、ening-only tests), thereby maximizing the information gained from the investment in a test. By encouraging concurrent talking, the systems duplex performance and ability to transmit multiple voices has a stronger effect on results than it would in less-demanding conversational tasks. Furthermore, by
47、 emphasizing rapid turn-taking, this method is sensitive to system delay. Secondly, the structured tests generate highly reproducible results, which necessarily should be the objective of any standardized test. The tests place low demands on social or verbal skills and are largely unaffected by list
48、ener training and experience. It is obvious that tasks specified in this Recommendation do not resemble a typical conversation. However, it covers important aspects of conversation such as: single talk, concurrent talk, turn taking, listening and understanding, back channels. Therefore, it is expect
49、ed that systems leading to superior performance in the tasks of this Recommendation also exhibit superior performance in real-world meetings, although the magnitude of the difference may be different. 6.2 Test principle This subjective test paradigm involves the participation of four or more subjects grouped into teams to complete tasks concurrently and on the same communication link. Depending on the conditions under test, subjects may be seated in the same or separate rooms to perform a series of tasks. Acoustic noise environments may be simulated
