1、 ETSI EG 202 396-3 V1.6.1 (2017-01) Speech and multimedia Transmission Quality (STQ); Speech Quality performance in the presence of background noise; Part 3: Background noise transmission - Objective test methods ETSI GUIDE ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 2 Reference REG/STQ-249 Keywords noi
2、se, QoS, quality, speech ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice The present document c
3、an be downloaded from: http:/www.etsi.org/standards-search The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of
4、any existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document ma
5、y be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at https:/portal.etsi.org/TB/ETSIDeliverableStatus.aspx If you find errors in the present document, please send your comment to one of the following services: https:/portal.e
6、tsi.org/People/CommiteeSupportStaff.aspx Copyright Notification No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI. The content of the PDF version shall not be modified w
7、ithout the written authorization of ETSI. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2017. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTMand the ETSI logo are Trade Marks of ETSI registered for the benefit of its
8、 Members. 3GPPTM and LTE are Trade Marks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. GSM and the GSM logo are Trade Marks registered and owned by the GSM Association. ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 3 Contents Intellectual Property Rights 5g3For
9、eword . 5g3Modal verbs terminology 5g31 Scope 6g32 References 6g32.1 Normative references . 6g32.2 Informative references 6g33 Symbols and abbreviations . 8g33.1 Symbols 8g33.2 Abbreviations . 8g34 Speech signals to be used . 9g35 Selection of the data within the scope of the wideband objective mode
10、l: Experts evaluation . 10g35.1 Selection process 10g35.2 Results 10g35.3 French database 11g36 Description of the wideband objective test method . 11g36.1 Introduction 11g36.2 Speech sample preparation and nomenclature 12g36.2.1 Speech sample preparation . 12g36.2.2 Nomenclature 15g36.3 Additional
11、Training data 16g36.4 Principles of Relative Approach and Relative Approach 16g36.5 Objective N-MOS. 19g36.5.1 Introduction. 19g36.5.2 Description of N-MOS algorithm . 20g36.5.3 Comparing subjective and objective N-MOS results 23g36.6 Objective S-MOS . 24g36.6.1 Introduction. 24g36.6.2 Description o
12、f S-MOS Algorithm . 25g36.6.3 Comparing Subjective and Objective S-MOS Results 28g36.7 Objective G-MOS. 29g36.7.1 Description of G-MOS Algorithm 29g36.7.2 Comparing subjective and objective G-MOS results 30g37 Validation of the Wideband Objective Test Method 31g37.1 Introduction 31g37.2 ETSI EG 202
13、396-2 Database Results Analysis . 33g37.2.1 Comparing subjective and objective N-MOS results 33g37.2.2 Comparing subjective and objective S-MOS results 33g37.2.3 Comparing Subjective and Objective G-MOS Results . 34g37.3 Orange Validation Database results Analysed . 35g37.3.0 Introduction. 35g37.3.1
14、 Comparing subjective and objective N-MOS results 35g37.3.2 Comparing subjective and objective S-MOS results 36g37.3.3 Comparing Subjective and Objective G-MOS Results . 36g38 Objective Model for Narrowband Applications . 37g38.0 Introduction 37g38.1 File pre-processing . 37g38.2 Adaptation of the C
15、alculations . 38g38.3 Prediction results 39g3Annex A: Detailed post evaluation of listening test results 41g3ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 4 Annex B: Results of PESQ and TOSQA2001 - Analysis of ETSI EG 202 396-2 database 44g3Annex C: Comparison of objective MOS versus auditory MOS for the
16、complete STF 294 database . 51g3Annex D: Comparison of objective MOS versus auditory MOS for rejected conditions 53g3Annex E: Void 55g3Annex F: Detailed STF 294 subjective and objective validation test results 56g3Annex G: Void 59g3Annex H: Extension of the Speech Quality Test Method to Narrowband:
17、Adaptation, Training and Validation . 60g3Annex I: Void 62g3Annex J: Summary of Czech samples not used for model training 63g3J.0 Introduction 63g3J.1 Selection process - Czech database 63g3J.2 General differences between the databases 65g3J.3 Comparison of the objective method results for Czech and
18、 French samples . 68g3J.4 Czech conditions results analysis . 73g3J.4.1 Comparing subjective and objective N-MOS results . 73g3J.4.2 Comparing subjective and objective S-MOS results 73g3J.4.3 Comparing Subjective and Objective G-MOS Results. 74g3J.5 Language Dependent Robustness of G-MOS. 75g3J.6 Re
19、gression Coefficients for Czech data . 76g3J.7 Post selection 77g3Annex K: Relative Approach Non-Linear Transformation . 81g3Annex L: Bibliography 82g3History 83g3ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 5 Intellectual Property Rights IPRs essential or potentially essential to the present document ma
20、y have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI s
21、tandards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (https:/ipr.etsi.org/). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs no
22、t referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Foreword This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality (STQ). The present document is a deli
23、verable of ETSI Specialized Task Force (STF) 294 entitled: “Improving the quality of eEurope wideband speech applications by developing a performance testing and evaluation methodology for background noise transmission“. The present document is part 3 of a multi-part deliverable covering Speech and
24、multimedia Transmission Quality (STQ); Speech Quality performance in the presence of background noise, as identified below: Part 1: “Background noise simulation technique and background noise database“; Part 2: “Background noise transmission - Network simulation - Subjective test database and result
25、s“; Part 3: “Background noise transmission - Objective test methods“. Modal verbs terminology In the present document “should“, “should not“, “may“, “need not“, “will“, “will not“, “can“ and “cannot“ are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the ex
26、pression of provisions). “must“ and “must not“ are NOT allowed in ETSI deliverables except when used in direct citation. ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 6 1 Scope The present document aims to identify and define testing methodologies which can be used to objectively evaluate the performance
27、of narrowband and wideband terminals and systems for speech communication in the presence of background noise. Background noise is a problem in mostly all situations and conditions and need to be taken into account in both, terminals and networks. The present document provides information about the
28、testing methods applicable to objectively evaluate the speech quality in the presence of background noise. The present document includes: The description of the experts post evaluation process chosen to select the subjective test data being within the scope of the objective methods. The results of t
29、he performance evaluation of the currently existing methods described in Recommendations ITU-T P.862 i.16 and P.862.1 i.17 and in TOSQA2001 i.19 which is chosen for the evaluation of terminals in the framework of ETSI VoIP speech quality test events i.8, i.9, i.10 and i.11. The method which is appli
30、cable to objectively determine the different parameters influencing the speech quality in the presence of background noise taking into account: - the speech quality; - the background noise transmission quality; - the overall quality. The present document is to be used in conjunction with: - ETSI ES
31、202 396-1 i.1 which describes a recording and reproduction setup for realistic simulation of background noise scenarios in lab-type environments for the performance evaluation of terminals and communication systems. - ETSI EG 202 396-2 i.2 which describes the simulation of network impairments and ho
32、w to simulate realistic transmission network scenarios and which contains the methodology and results of the subjective scoring for the data forming the basis of the present document. - French speech sentences as defined in Recommendation ITU-T P.501 i.13 for wideband and English speech sentences as
33、 defined in Recommendation ITU-T P.501 i.13 for narrowband. 2 References 2.1 Normative references Normative references are not applicable in the present document. 2.2 Informative references References are either specific (identified by date of publication and/or edition number or version number) or
34、non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies. NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their
35、long term validity. The following referenced documents are not necessary for the application of the present document but they assist the user with regard to a particular subject area. i.1 ETSI ES 202 396-1: “Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence
36、 of background noise; Part 1: Background noise simulation technique and background noise database“. ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 7 i.2 ETSI EG 202 396-2: “Speech Processing, Transmission and Quality Aspects (STQ); Speech Quality performance in the presence of background noise; Part 2: Bac
37、kground Noise Transmission - Network Simulation - Subjective Test Database and Results“. i.3 Recommendation ITU-T P.835: “Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm“. i.4 Recommendation ITU-T P.800: “Methods for subjective determi
38、nation of transmission quality“. i.5 Recommendation ITU-T P.831: “Subjective performance evaluation of network echo cancellers“. i.6 Genuit, K.: “Objective Evaluation of Acoustic Quality Based on a Relative Approach“, InterNoise 96, Liverpool, UK. i.7 Recommendation ITU-T SG 12 Contribution 34: “Eva
39、luation of the quality of background noise transmission using the “Relative Approach“. i.8 ETSI 2ndSpeech Quality Test Event: “Anonymized Test Report“, ETSI Plugtests, HEAD acoustics, T-Systems Nova. NOTE: Available at: http:/www.etsi.org/WebSite/OurServices/Plugtests/History.aspx. Also available as
40、 ETSI TR 102 648-3. i.9 ETSI 3rdSpeech Quality Test Event: “Anonymized Test Report “IP Gateways“. NOTE: Available at: http:/www.etsi.org/WebSite/OurServices/Plugtests/History.aspx. i.10 ETSI 3rdSpeech Quality Test Event: “Anonymized Test Report “IP Phones“. i.11 ETSI 4thSpeech Quality Test Event: “A
41、nonymized Test Report “IP Gateways and IP Phones“. NOTE: Available at: http:/www.etsi.org/WebSite/OurServices/Plugtests/History.aspx. i.12 F. Kettler, H.W. Gierlich, F. Rosenberger: “Application of the Relative Approach to Optimize Packet Loss Concealment Implementations“, DAGA, March 2003, Aachen,
42、Germany. i.13 Recommendation ITU-T P.501: “Test Signals for Use in Telephonometry“. i.14 R. Sottek, K. Genuit: “Models of Signal Processing in human hearing“, International Journal of Electronics and Communications (AE) volume 59, 2005, p. 157-165. NOTE: Available at: http:/www.elsevier.de/aeue. i.1
43、5 SAE International - Document 2005-01-2513: “Tools and Methods for Product Sound Design of Vehicles“ R. Sottek, W. Krebber, G. Stanley. i.16 Recommendation ITU-T P.862: “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrowband teleph
44、one networks and speech codecs“. i.17 Recommendation ITU-T P.862.1: “Mapping function for transforming P.862 raw result scores to MOS-LQO“. i.18 Recommendation ITU-T P.862.2: “Wideband extension to Recommendation P.862 for the assessment of wideband telephone networks and speech codecs“. i.19 Recomm
45、endation ITU-T SG 12 Contribution 19: “Results of objective speech quality assessment of wideband speech using the Advanced TOSQA2001“. i.20 Recommendation ITU-T G.722: “7 kHz audio-coding within 64 kbit/s“. i.21 Recommendation ITU-T G.722.2: “Wideband coding of speech at around 16 kbit/s using Adap
46、tive Multi-Rate Wideband (AMR-WB)“. i.22 Recommendation ITU-T P.56: “Objective measurement of active speech level“. i.23 Recommendation ITU-T P.57: “Artificial ears“. ETSI ETSI EG 202 396-3 V1.6.1 (2017-01) 8 i.24 M. Spiegel: “Theory and problems of statistics“, McGraw Hill, 1998. i.25 Void. i.26 M.
47、 Kendall: “Rank correlation methods“, Charles Griffin Speech quality performance in the presence of background noise: Background noise transmission for mobile terminals-objective test methods“. i.33 Hastie T.; Tibshirani R. and Friedman J.: “The Elements of Statistical Learning: Data Mining, Inferen
48、ce, and Prediction“, New York: Springer-Verlag, 2001. i.34 ETSI EG 202 396-3 (V1.1.1 to V1.3.1): “Speech Processing, Transmission and Quality Aspects (STQ); Speech Quality performance in the presence of background noise; Part 3: Background noise transmission - Objective test methods“. 3 Symbols and
49、abbreviations 3.1 Symbols For the purposes of the present document, the following symbols apply: 2Variance 3.2 Abbreviations For the purposes of the present document, the following abbreviations apply: AMR Adaptive MultiRate ASL Active Speech Level NOTE: According to Recommendation ITU-T P.56 i.22. BGN BackGround Noise CDF Cumulative Density Function dB SPL Sound Pressure Level re 20 Pa in dB DB Data Base DUT Device Under Test EFR Enhance Full Rate FR Full Rate G-MOS Global MOS N
