ITU-T P 1310-2017 Spatial audio meetings quality evaluation (Study Group 12).pdf

1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T P.1310 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (03/2017) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Telemeeting assessment Spatial audio meetings quality evaluation Recomm

2、endation ITU-T P.1310 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal characteristics Series P.30 P.300 Reference syste

3、ms Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 Methods for objective and subjective assessment of sp

4、eech and video quality Series P.800 Audiovisual quality in multimedia services Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting asse

5、ssment Series P.1300 Statistical analysis, evaluation and reporting guidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than speech and video Series P.1500 For further details, please refer to the list of ITU-T Recommendations

6、. Rec. ITU-T P.1310 (03/2017) i Recommendation ITU-T P.1310 Spatial audio meetings quality evaluation Summary Recommendation ITU-T P.1310 concerns the quality assessment of telemeeting systems that apply spatial audio rendering techniques to facilitate communication between parties at remote locatio

7、ns. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T P.1310 2017-03-01 12 11.1002/1000/13181 Keywords Performance measures, spatial audio, subjective testing, telemeeting. * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your web brow

8、ser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/ 11830-en. ii Rec. ITU-T P.1310 (03/2017) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication

9、 technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World

10、Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In so

11、me areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recogniz

12、ed operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The word

13、s “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility that t

14、he practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation

15、 development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are

16、therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2017 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.1310 (03/2017) iii Table of Contents Page 1 Sco

17、pe . 5 2 References . 5 3 Definitions 5 3.1 Terms defined elsewhere 5 3.2 Terms defined in this Recommendation . 6 4 Abbreviations and acronyms 6 5 Conventions 6 6 General recommendations concerning assessment of spatial audio telemeeting systems 6 7 Guidance on appropriate test method . 6 7.1 Decis

18、ion criteria 6 7.2 Considerations on the test focus . 8 7.3 Selection of individual test methods . 9 Annex A Method to collect subjective quality ratings for spatial audio meetings . 11 A.1 Test facilities 11 A.2 Conversation tasks and non-interactive stimuli 11 A.3 Test participants 13 A.4 Experime

19、nt design 14 A.5 Scales 14 A.6 Instructions to subjects and training phases . 15 Annex B Assessing the accuracy of the spatial alignment of audiovisual systems using spatial audio 16 B.1 Introduction 16 B.2 Evaluation criteria concerning audiovisual alignment . 16 B.3 Rendering scenarios affected by

20、 audiovisual alignment 17 Annex C Listening test method to obtain task performance indicators for the intelligibility of concurrent speakers 19 Annex D Conversation test method for the measurement of communication effectiveness using task performance . 20 Annex E Listening test method to obtain task

21、 performance indicators for the cognitive load experienced in spatial audio meetings 21 E.1 Introduction and scope . 21 E.2 Test method overview 21 E.3 Speech material 21 E.4 Quotations . 21 E.6 Test design 22 E.7 Data validation 22 E.8 Data analysis . 23 Bibliography. 24 Rec. ITU-T P.1310 (03/2017)

22、 5 Recommendation ITU-T P.1310 Spatial audio meetings quality evaluation 1 Scope The focus of this Recommendation is on test methods involving test participants, which collect subjective ratings, task performance measures, and/or descriptors of communication aspects. For that purpose, this Recommend

23、ation extends the testing procedures for telemeeting systems described in ITU-T P.1301 by specifying test methods dedicated to spatial audio telemeeting systems and where appropriate referring to stand-alone methods that are also suitable for spatial audio, which are ITU-T P.1311, ITU-T P.1312. Furt

24、her test methods related to spatial audio will be under study within ITU and this Recommendation will be updated with the relevant references when new methods are available. In addition, this Recommendation also provides some guidance on testing spatial audio systems which include video communicatio

25、n. The focus is on the perceived alignment of the audio and the video source locations. 2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editio

26、ns indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid

27、ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.805 R

28、ecommendation ITU-T P.805 (2007), Subjective evaluation of conversational quality. ITU-T P.806 Recommendation ITU-T P.806 (2014), A subjective quality test methodology using multiple rating scales. ITU-T P.1301 Recommendation ITU-T P.1301 (2012), Subjective quality evaluation of audio and audiovisua

29、l multiparty telemeetings. ITU-T P.1311 Recommendation ITU-T P.1311 (2014), Method for determining the intelligibility of multiple concurrent talkers. ITU-T P.1312 Recommendation ITU-T P.1312 (2016), Method for the measurement of the communication effectiveness of multiparty telemeetings using task

30、performance. ITU-T P-Sup.26 ITU-T P-series Recommendations Supplement 26 (2012), Scenarios for the subjective quality evaluation of audio and audiovisual multiparty telemeetings. 3 Definitions 3.1 Terms defined elsewhere None. 6 Rec. ITU-T P.1310 (03/2017) 3.2 Terms defined in this Recommendation No

31、ne. 4 Abbreviations and acronyms None. 5 Conventions None. 6 General recommendations concerning assessment of spatial audio telemeeting systems It is recommended that subjective quality evaluations of spatial audio telemeetings are carried out as much as possible according to existing test methods r

32、ecommended by ITU-T and ITU-R. It is also recommended that the purpose of the test is taken into account when selecting and, if necessary, adapting appropriate test methods. Test purposes differentiate, for example, by the following aspects: rendering scenarios (e.g., audiovisual or audio only syste

33、ms); the focus on the system components under test (e.g., overall system or individual system components); the target variables (e.g., overall quality, timbral quality, spatial quality, cognitive load, or task performance); the desired measurement sensitivity (e.g., differentiating just spatial audi

34、o vs. non-spatial audio or differentiating several instances of spatial audio); or by the measurement paradigm (e.g., subjective ratings or performance measures). In case of multiparty tests, it is recommended that ITU-T P.1301 is consulted for multiparty-specific advice when selecting the appropria

35、te test methods. It is recommended that spatial-audio-specific considerations are taken into account when selecting and, if necessary, adapting appropriate test methods. Such considerations may be based on the spatial-audio-specific aspects described in this Recommendation. The selection, and if nec

36、essary adaptation of an appropriate test method may be done using the guidelines provided in this present Recommendation. Finally, practitioners should be aware that spatial audio telemeeting systems can and should be evaluated on several metrics. For that reason, it is recommended that practitioner

37、s evaluate which set of aspects is important for a complete evaluation of a system under test and choose the test methods accordingly. 7 Guidance on appropriate test method It is recommended that the actual test purpose at hand is first analysed in terms of the decision criteria in clause 7.1. Then,

38、 the guidance in clauses 7.2 and 7.3 can be applied to identify the most appropriate test method. 7.1 Decision criteria The choice for a particular test method described in this Recommendation depends on the following six decision criteria: Rec. ITU-T P.1310 (03/2017) 7 Rendering scenario: To utiliz

39、e their full potential, spatial audio rendering approaches differ between use cases, and thus the assessment methodology should be chosen accordingly. Two main rendering scenarios are considered : 1 audio-only or audiovisual systems using spatial audio for improved communication experience, independ

40、ently from the video signal; 2 audiovisual systems showing multiple participants for which the rendered audio space is expected to be aligned with the visual location of the participants. Test Focus: Spatial audio telemeeting systems in real networks consist of more components than just the spatial

41、audio capturing and rendering engines, e.g., codecs, bandwidth, noise and echo reduction. For that reason, the test method should be aligned with the test focus; that is the evaluation of: 1 the overall system; or 2 individual system components. Target variables: The media signal quality of spatial

42、audio has two main aspects; its spatial quality component and its non-spatial quality component: The spatial quality component refers to the quality features related to the spatial attributes of sound. Examples of such attributes are immersion, envelopment, localization blur and source width. The no

43、n-spatial quality component refers to the quality features related to the non-spatial attributes of sound. Examples of such attributes are loudness, timbre, noisiness and sharpness. Furthermore, the added value of spatial audio in telemeetings goes beyond media signal quality since it benefits commu

44、nicative aspects such as conversation flow, communication effort, cognitive load and task performance. As a result, the perceived overall quality may be different from the perceived media signal quality. For that reason, the selected test method should be appropriate for the desired target variable(

45、s). The variables considered here are: 1 Overall quality (consisting of media-signal quality and communication quality components). 2 Media-signal-quality component (consisting of non-spatial quality and spatial quality components). 3 Communication-quality component. Measurement sensitivity: Assessm

46、ent methods that have been proposed and used in the state-of-the-art can be divided into two categories in terms of their sensitivity: 1 Methods that are able to show the added value of spatial audio systems, i.e., non-spatial audio vs. spatial audio. 2 Methods that are in addition able to distingui

47、sh different variants of spatial audio systems. This needs to be considered when selecting a test method. Test paradigm: listening-only tests, conversation tests Both, listening-only (non-interactive) tests and conversation tests can be used for evaluating spatial audio telemeeting systems. 8 Rec. I

48、TU-T P.1310 (03/2017) Listening-only tests have the advantage of being simpler to implement and easier to conduct. They enable the assessment of many short stimuli and tend to allow test subjects to focus more on the quality assessment task at hand. Conversation tests allow the experimenter to set t

49、he test subjects into a more natural and realistic situation by using a partial or a full communication system. These tests enable the evaluation of conversational aspects, beyond pure signal quality, and the assessment of all components and properties of a full system. Measurement paradigm: An additional selection criterion is the measurement paradigm of a test method, which needs to be aligned with the assessment goals. 1 If the assessment goals are to evaluate the quality experience consciously