ETSI TR 103 503-2018 Speech and multimedia Transmission Quality (STQ) Procedures for Multimedia Transmission Quality Testing with Parallel Task including Subjective Testing (V1 1 1.pdf

1、 ETSI TR 103 503 V1.1.1 (2018-03) Speech and multimedia Transmission Quality (STQ); Procedures for Multimedia Transmission Quality Testing with Parallel Task including Subjective Testing TECHNICAL REPORT ETSI ETSI TR 103 503 V1.1.1 (2018-03) 2Reference DTR/STQ-255 Keywords assessment, listening qual

2、ity, parallel task ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice The present document can be

3、downloaded from: http:/www.etsi.org/standards-search The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any ex

4、isting or perceived difference in contents between such versions and/or in print, the only prevailing document is the print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be s

5、ubject to revision or change of status. Information on the current status of this and other ETSI documents is available at https:/portal.etsi.org/TB/ETSIDeliverableStatus.aspx If you find errors in the present document, please send your comment to one of the following services: https:/portal.etsi.or

6、g/People/CommiteeSupportStaff.aspx Copyright Notification No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI. The content of the PDF version shall not be modified without

7、 the written authorization of ETSI. The copyright and the foregoing restriction extend to reproduction in all media. ETSI 2018. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTMand the ETSI logo are trademarks of ETSI registered for the benefit of its Members. 3GPPTM and LTETMare trademarks of ETSI r

8、egistered for the benefit of its Members and of the 3GPP Organizational Partners. oneM2M logo is protected for the benefit of its Members. GSMand the GSM logo are trademarks registered and owned by the GSM Association. ETSI ETSI TR 103 503 V1.1.1 (2018-03) 3Contents Intellectual Property Rights 4g3F

9、oreword . 4g3Modal verbs terminology 4g3Introduction 4g31 Scope 5g32 References 5g32.1 Normative references . 5g32.2 Informative references 5g33 Abbreviations . 6g34 Subjective speech quality assessment, intelligibility and listening effort: existing approaches . 7g34.1 Introduction 7g34.2 Classific

10、ation of parallel tasks in scientific publications 7g34.2.1 Current approaches . 7g34.2.2 Mentally oriented tasks . 8g34.2.3 Physically oriented tasks . 8g34.2.4 Hybrid tasks 9g35 Procedures for subjective testing deploying parallel task 9g35.1 General considerations . 9g35.1.1 Introduction. 9g35.1.

11、2 Task Class 1 (activity driven) . 9g35.1.3 Task Class 2 (purpose driven). 10g35.1.4 Additional comments to Task Class 1 and Task Class 2 classification . 10g35.2 Test Environment . 10g35.2.1 Real environment 10g35.2.2 Lab with simulated parallel task . 10g35.2.3 VR based testing . 11g35.3 Subjectiv

12、e testing procedure. 11g35.4 Result Analysis and Reporting . 11g35.4.1 Introduction. 11g35.4.2 Special reported items . 11g3Annex A: Examples of test scenarios incorporating parallel task . 12g3A.1 Example scenario 1 - psychomotor experiment A (hybrid general task) . 12g3A.2 Example scenario 2 - psy

13、chomotor experiment B (hybrid general task) . 13g3A.3 Example scenario 3 - tasting experiment A (hybrid purpose oriented task). 14g3A.4 Example scenario 4 - tasting experiment B (hybrid purpose oriented task) . 14g3A.5 Example scenario 5 stationary bicycle (physically oriented, purpose oriented task

14、) 15g3A.6 Example scenario 6 - virtual reality deployment (physically oriented general task) . 15g3History 17g3ETSI ETSI TR 103 503 V1.1.1 (2018-03) 4Intellectual Property Rights Essential patents IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The i

15、nformation pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards“, which is available from

16、the ETSI Secretariat. Latest updates are available on the ETSI Web server (https:/ipr.etsi.org/). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or

17、 the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Trademarks The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indic

18、ated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks. Foreword This Technical

19、 Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality (STQ). The present document describes auditory test methodologies for the prediction of perceived audio signal quality under parallel task conditions. Modal verbs terminology In the present document

20、 “should“, “should not“, “may“, “need not“, “will“, “will not“, “can“ and “cannot“ are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions). “must“ and “must not“ are NOT allowed in ETSI deliverables except when used in direct citati

21、on. Introduction Subjective testing of speech quality and intelligibility is standardized at ETSI, ANSI, ITU-T and ITU-R. Tests are performed in defined environments using listening/conversational rigorous procedures (Recommendation ITU-T P.800 i.16, Recommendation ITU-T P.805 i.21, Recommendation I

22、TU-T P.835 i.18, Recommendation ITU-R BS.1534-3 i.22, Recommendation ITU-R BS.1116 i.23, etc.), and they require relaxed, fresh, fit and concentrated naive or expert listeners seated comfortably in usually artificially looking listening room/booth. However, such a test does not correspond to the nor

23、mal use of the tested technologies. Voice services are often used in sports, driving, work, public transport, or other noisy or less convenient environments. Users are tired, stressed or concentrate on another, often important, task. In an attempt to bring laboratory tests closer to reality, the so-

24、called dual-task or parallel-task tests are introduced, in these test participants are asked to perform multiple different tasks at the same time. ETSI ETSI TR 103 503 V1.1.1 (2018-03) 51 Scope The present document describes the methods for assessment of subjective audio (including speech) quality a

25、nd speech intelligibility under parallel task condition. This approach can be used to evaluate the perceived listening quality or speech intelligibility in situations which better mimics real operation of the tested telecommunication equipment or algorithm. The present document describes possible pa

26、rallel task generation and scenarios, the test design and reference conditions used to evaluate the quality or intelligibility subjectively. Several parallel task scenarios are covered: Physically oriented. Mentally oriented. Hybrid. 2 References 2.1 Normative references Normative references are not

27、 applicable in the present document. 2.2 Informative references References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the

28、referenced document (including any amendments) applies. NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long term validity. The following referenced documents are not necessary for the application of the present document but they

29、assist the user with regard to a particular subject area. i.1 V. Durin and L. Gros: “Measuring speech quality impact on tasks performance“, Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, pp. 2074-2077, 2008. i.2 A. Serampalis, S. Kalluri, B. Edwards, and E. Hafter: “Objective measures of

30、listening effort in noise“, J. Speech, Lang. Hear. Res., vol. 52, no. October 2009, pp. 1230-1240, 2009. i.3 G. P. Sonntag, T. Portele, and F. Haas: “Comparing the comprehensibility of different synthetic voices in a dual task experiment“, Proc. Third Work. Speech Synth. Jenolan Caves House, Blue Mt

31、., pp. 5-10, 1998. i.4 L. Gros, N. Chateau, and S. Busson: “The impact of real environments on transmitted speech quality judgments“, Quality, vol. 0, pp. 45-50, 2003. i.5 D. Guse, S. Egger, A. Raake, and S. Moller: “Web-QOE under real-world distractions: Two test cases“, 2014 6th Int. Work. Qual. M

32、ultimed. Exp. QoMEX 2014, pp. 220-225, 2014. i.6 S. L. Beilock, T. H. Carr, C. MacMahon, and J. L. Starkes: “When paying attention becomes counterproductive: Impact of divided versus skill-focused attention on novice and experienced performance of sensorimotor skills“, J. Exp. Psychol. Appl., vol. 8

33、, no. 1, pp. 6-16, 2002. i.7 J. Holub: “Low Bit-rate Coded Speech Intelligibility - Comparison of Laboratory Test Results and Results of Test with Parallel Task“, in Future Forces Forum, 2016. ETSI ETSI TR 103 503 V1.1.1 (2018-03) 6i.8 D. L. Strayer and W. A. Johnston: “Driven to distraction: Dual-t

34、ask studies of simulated driving and conversing on a cellular telephone“, Psychol. Sci., vol. 12, no. 6, pp. 462-466, 2001. i.9 S. Choi, A. Lotto, D. Lewis, B. Hoover, and P. Stelmachowicz: “Attentional Modulation of Word Recognition by Children in a Dual-Task Paradigm“, J. Speech Lang. Hear. Res.,

35、vol. 51, no. 4, p. 1042, Aug. 2008. i.10 Y.-H. Wu, E. Stangl, X. Zhang, J. Perkins, and E. Eilers: “Psychometric Functions of Dual-Task Paradigms for Measuring Listening Effort“, Ear Hear., vol. 37, no. 6, pp. 660-670, 2016. i.11 C. Kwak and W. Han: “Comparison of Single-Task versus Dual-Task for Li

36、stening Effort“, J. Audiol. Otol., Oct. 2017. i.12 L. Gros, N. Chateau, and A. Mac: “Assessing speech qualityuni2009: a new approach Methodology“, 2005. i.13 K. S. Helfer, J. Chevalier, and R. L. Freyman: “Aging, spatial cues, and single- versus dual-task performance in competing speech perception“,

37、 J. Acoust. Soc. Am., vol. 128, no. 6, pp. 3625-3633, Dec. 2010. i.14 K. Bunton and C. K. Keintz: “The use of a dual-task paradigm for assessing speech intelligibility in clients with Parkinson disease“, J. Med. Speech. Lang. Pathol., vol. 16, no. 3, pp. 141-155, Sep. 2008. i.15 ITU-T Handbook: “Pra

38、ctical procedures for subjective testing“, 2011. i.16 Recommendation ITU-T P.800 (08/1996): “Methods for subjective determination of transmission quality“. i.17 Recommendation ITU-T P.807 (02/2016): “Subjective test methodology for assessing speech intelligibility“. i.18 Recommendation ITU-T P.835 (

39、11/2003): “Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm“. i.19 Council of Europe (2011): “Common European Framework of Reference for Languages: Learning, Teaching, Assessment“ Council of Europe. i.20 Recommendation ITU-T P.1400 (03/

40、2013): “Statistical analysis, evaluation and reporting guidelines of quality measurements“. i.21 Recommendation ITU-T: P.805: “Subjective evaluation of conversational quality“, Geneva 2007. i.22 Recommendation ITU-R: BS.1534: “Method for the subjective assessment of intermediate quality levels of co

41、ding systems“, Geneva 2015. i.23 Recommendation ITU-R: BS.1116: “Methods for the subjective assessment of small impairments in audio systems“, Geneva 2015. 3 Abbreviations For the purposes of the present document, the following abbreviations apply: ECG ElectroCardioGraphy EEG ElectroEncephaloGraphyP

42、C Personal Computer QoE Quality of Experience STD STandard Deviation VR Virtual Reality ETSI ETSI TR 103 503 V1.1.1 (2018-03) 74 Subjective speech quality assessment, intelligibility and listening effort: existing approaches 4.1 Introduction Subjective testing of speech quality and intelligibility f

43、ollows strictly standardized procedures. Tests are performed in defined environments using listening/conversational rigorous procedures (i.16, i.18, etc.) and it requires relaxed, fresh, fit and concentrated naive or expert listeners comfortably seated in a listening room/booth with proper acoustic

44、lining to minimize e.g. inherent background noise and room reverberation. However, such a test does not correspond to normal use of the tested technologies. Voice services are often used during sports, driving, work, etc. Users are tired, stressed or concentrated on another, often important, task. T

45、o bring laboratory tests closer to reality, the so-called dual-task or parallel-task tests are introduced, where test participants are asked to perform multiple different tasks at the same time. The test results obtained during parallel task test differ from regular subjective tests. The differences

46、 are sometimes contra-intuitive and cannot be explained e.g. by decreased level of subjects attention. The parallel task should be designed to distract subjects in a similar way as the activity performed during the real (targeted) situation. Limitations are given by requirements on repeatability, sp

47、ace- and movement- restrictions in the lab, etc. 4.2 Classification of parallel tasks in scientific publications 4.2.1 Current approaches Parallel tasks found in scientific literature can be divided into three types: Mentally oriented tasks, Physically oriented tasks, and Hybrid tasks. Selected avai

48、lable experiments of those three categories are discussed in Table 1. Table 1: Resource summary Reference Test type Parallel task Parallel task type Language i.1 Speech intelligibility Memorizing digits Mentally oriented N/A i.2 Speech intelligibility Memorizing digits Mentally oriented English i.3

49、Speech intelligibility Pressing colour buttons Mentally oriented German i.4 QoE test Pressing colour buttons Mentally oriented English i.5 QoE testTraveling in public transport; watching a TV Mentally oriented; Hybrid German i.6 Other Memorizing tones; memorizing words Mentally oriented N/A i.7 Speech intelligibility Laser shooting simulator Hybrid English i.8 Other Telephone call Hybrid Englishi.9 Speech intelligibility Word repetition; Memorizing digits Mentally oriented English i.10 Spe