1、 ETSI TR 102 251 V1.1.1 (2003-10)Technical Report Speech Processing, Transmission and Quality Aspects (STQ);Anonymous Test Reportfrom 2ndSpeech Quality Test Event 2002ETSI ETSI TR 102 251 V1.1.1 (2003-10) 2 Reference DTR/STQ-00051 Keywords QoS, speech ETSI 650 Route des Lucioles F-06921 Sophia Antip
2、olis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present document can be downloaded from: http:/www.etsi.org The presen
3、t document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dispute, the reference shall be the printing on ETSI printers o
4、f the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at http:/portal.etsi.org/tb/stat
5、us/status.asp If you find errors in the present document, send your comment to: editoretsi.org Copyright Notification No part may be reproduced except as authorized by written permission. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Sta
6、ndards Institute 2003. All rights reserved. DECTTM, PLUGTESTSTM and UMTSTM are Trade Marks of ETSI registered for the benefit of its Members. TIPHONTMand the TIPHON logo are Trade Marks currently being registered by ETSI for the benefit of its Members. 3GPPTM is a Trade Mark of ETSI registered for t
7、he benefit of its Members and of the 3GPP Organizational Partners. ETSI ETSI TR 102 251 V1.1.1 (2003-10) 3 Contents Intellectual Property Rights4 Foreword.4 1 Scope 5 2 References 5 3 Abbreviations .6 4 Management summary .6 5 Organizational topics6 5.1 Time and location.6 5.2 Contacts7 6 Responsibi
8、lities 7 7 Test description 8 7.1 General test description 8 7.2 Measurement scenarios 9 7.2.1 Measurements using electrical interfaces .9 7.2.1.1 Measurement setup9 7.2.1.2 Measurement conditions .9 7.2.2 Acoustical measurements .10 7.2.2.1 Measurement setup10 7.2.3 Measurement conditions .12 7.3 M
9、easurement methodology 13 7.4 Test signals.15 7.4.1 Voice signals.15 7.4.2 Test signals according to ITU-T Recommendation P.501 15 7.5 Assessment methods.23 7.5.1 Instrumental assessment of listening quality 23 7.5.2 TOSQA and PESQ results and precision23 7.5.3 Instrumental computational assessment
10、using speech-like (P.501) test signals.24 8 Results 25 8.1 Estimation of one-way speech quality25 8.1.1 G.711 Codec .25 8.1.2 G.729 Codec .26 8.2 Delay measurements.29 8.3 Transmission parameters, double talk performance and background noise transmission 29 8.3.1 Tests with G.711 Codec30 8.3.2 Echo
11、canceller performance test with G.711 Codec .38 8.3.3 Tests with G.729A Codec.42 History 46 ETSI ETSI TR 102 251 V1.1.1 (2003-10) 4 Intellectual Property Rights IPRs essential or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essenti
12、al IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates a
13、re available on the ETSI Web server (http:/webapp.etsi.org/IPR/home.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI
14、 Web server) which are, or may be, or may become, essential to the present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech Processing, Transmission and Quality Aspects (STQ). The tests conducted during the 2ndETSI TIPHON VoIP Speech Quality Test Eve
15、nt have been optimized and adapted considering the feedback from all participating manufactures after the 1stETSI VoIP Speech Quality Test Event, concentrating on a more detailed evaluation of specific parameters, optimizing the number of network conditions and discussing the manufacturers implement
16、ation based on the test results already during the test event. In order to ensure consistency of the measurements of the 1stETSI VoIP Speech Quality Test Event and to allow comparisons of results as well, instrumental measures for electrical and acoustical VoIP scenarios were conducted with TOSQA200
17、1. In addition, instrumental measurements were conducted according to ITU-T Recommendation P.862 for those scenarios where P.862 is suitable. Instrumental measurements were performed for one-way speech transmission as well as double-talk situations by evaluating background noise transmission perform
18、ance, PLC implementations and echo cancelling characteristics in detail. The measurements conducted by HEAD acoustics were based on test signals and test procedures as described in TS 101 329-5 1 using test signals and analysis procedures described in ITU-T Recommendations P.501 3, P.502 4 and G.168
19、. Furthermore, each company conducted a “free-style“ testing block to evaluate specific test conditions. In addition to the tests, an extra day tutorial session was held for each manufacturer. Within ETSI, two bodies are actively involved in this event. ETSI Project TIPHON (Telecommunications and In
20、ternet Protocol Harmonization Over Networks) and ETSI Technical Committee STQ (Speech Processing, Transmission and Quality Aspects) look for the improvement of quality aspects in the voice transmission area. ETSI ETSI TR 102 251 V1.1.1 (2003-10) 5 1 Scope The present document contains the anonymous
21、Test Report from the 2nd Speech Quality Test Event 2002. 2 References For the purposes of this Technical Report (TR) the following references apply: 1 ETSI TS 101 329-5: “Telecommunications and Internet Protocol Harmonization Over Networks (TIPHON) Release 3; End-to-end Quality of Service in TIPHON
22、systems; Part 5: Quality of Service (QoS) measurement methodologies“. 2 ETSI EG 201 377-1: “Speech Processing, Transmission and Quality Aspects (STQ); specification and measurement of speech transmission quality; part 1: Introduction to objective comparison measurement methods for one-way speech qua
23、lity across networks“. 3 ITU-T Recommendation P.501: “Test signals for use in telephonometry“. 4 ITU-T Recommendation P.502: “Objective test methods for speech communication systems using complex test signals“. 5 ITU-T Recommendation P.58: “Head and torso simulator for telephonometry“. 6 ITU-T Recom
24、mendation P.57: “Artificial ears“. 7 ETSI TIPHON temporary document 17TD135: “Subjective and Objective Speech Quality Evaluation on Speech Data recorded“ at the SuperOp 99 event in Hawaii. Sophia Antipolis, March 2000. 8 ITU-T Recommendation P.64: “Determination of sensitivity/frequency characterist
25、ics of local telephone systems“. 9 ITU-T Recommendation P.79: “Calculation of loudness ratings for telephone sets“. 10 ITU-T Recommendation G.122: “Influence of national systems on stability and talker echo in international connections“. 11 ITU-T Recommendation P.56: “Objective measurement of active
26、 speech level“. 12 ITU-T Recommendation P.830: “Subjective performance assessment of telephone-band and wideband digital codecs“. 13 ITU-T Recommendation P.810: “Modulated noise reference unit (MNRU)“. 14 ITU-T COM12-D41 Deutsche Telekom AG (Q9/12): “Enhancement of P.862 results by post-processing u
27、sing “TOCQ“. 15 ITU-T Recommendation P.800: “Methods for subjective determination of transmission quality“. 16 Genuit, K.: “Objective Evaluation of Acoustic Quality based on a Relative Approach“, InterNoise Proceedings, Liverpool, UK, 1996. 17 Sottek, R.: “Modelle zur Signalverarbeitung im menschlic
28、hen Gehr“, PHD thesis RWTH Aachen, 1993. ETSI ETSI TR 102 251 V1.1.1 (2003-10) 6 3 Abbreviations For the purposes of the present document, the following abbreviations apply: ACQUA Advanced Communication Quality Analysis (HEAD acoustics test system) ASL Active Speech Level CSS Composite Source Signal
29、 HATS Head And Torso Simulator IRS Intermediate Reference System MNRU Modulated Noise Reference Unit MOS Mean Opinion Score RTP Real Time Transport Protocol TMOS TOSQA Mean Opinion Score (output of TOSQA) TOSQA Telecommunications Objective Speech Quality Assessment VAD Voice Activity Detection 4 Man
30、agement summary The present document describes the test methodologies, the assessment methods and the results of the measurements which were carried out during the 2nd ETSI Speech Quality Test Event. The tests were conducted by T-Systems Nova GmbH Berkom (Berkom) and HEAD acoustics GmbH (HEAD acoust
31、ics). The aim of the test event was to determine the speech quality of various Voice over IP equipment under certain IP network conditions. During the test event, speech material as well as measurement data were collected by transferring voice samples and artificial signals across the Voice over IP
32、setup. This material was analyzed and the results are reported in the present documentation. The analysis of the collected data can be split in two parts. In the first part the assessment of the one-way speech quality (listening quality) was performed by instrumental assessments. The one-way speech
33、transmission quality was evaluated by processing real speech samples and analyzing it using the TOSQA2001 algorithm and in addition using the current ITU-T standard P.862 “PESQ“. TOSQA2001 and PESQ lead to MOS-comparable results on a scale from 1 to 5 or -1 to 5, respectively. For all measurements a
34、t the acoustical interface using artificial ears (e.g. measurements of IP-Phones) only TOSQA2001 was used, because PESQ is not applicable for evaluation of acoustical recorded signals according to P.862 documentation. PESQ and TOSQA were used in all measurement scenarios that were based on measureme
35、nts at the electrical interfaces at sending and receiving side (e.g. Gateway-Gateway). In the second part more detailed analyses were carried out. The various strategies of PLC implementations were investigated in detail as well as VAD- and comfort noise implementations. Furthermore additional situa
36、tions were evaluated: The system performance with background noise during double talk and specifically the echo canceller characteristics. These tests were based on TIPHON TS 101 329-5 1 and ITU-T Recommendations P.501 3, P.502 4 and G.168.These test show the performance of the different implementat
37、ions with respect to speech quality in detail and can be used to optimize the system design. 5 Organizational topics 5.1 Time and location The measurements were conducted during the 2ndETSI Speech Quality Test Event at ETSI Headquarters in Sophia Antipolis, France from 16thto 24thof April 2002. ETSI
38、 ETSI TR 102 251 V1.1.1 (2003-10) 7 5.2 Contacts The ETSI responsibilities are covered by: ETSI: Philippe Cousin e-mail: phillippe.cousinetsi.org Tel: +33 4 92 94 43 06 List of authors: Berkom: Harald Klaus e-mail: harald.klaust- Tel: +49 30 3497 2382 Dr. Jens Berger e-mail: jens.bergert- Tel: +49 3
39、0 3497 2374 Oliver Eisfeldt e-mail: oliver.eisfeldtt- Tel: +49 30 3497 2646 HEAD acoustics: Dr. H.W. Gierlich e-mail: h.w.gierlichhead-acoustics.de Tel: +49 2407 577-26 Frank Kettler e-mail: frank.kettlerhead-acoustics.de Tel: +49 2407 577-26 Frank Rosenberger e-mail: frank.rosenbergerhead-acoustics
40、.de Tel: +49 2407 577-26 6 Responsibilities The present document contains results of both types of measurements, for measurement at the electrical network-interface of the used PBX as well as at the acoustical interface using a telephone-handset which was connected to the PBX. IP-phones were connect
41、ed to the IP-network. Technical details for the various types of measurements can be found later in the present document. This clause is intended to provide information about the responsibility of the test labs (Berkom, HEAD acoustics) for the respective measurements, quality assessments and parts o
42、f the reports. So, if you have any questions concerning the report you may contact the responsible test lab. The contact information can be found in the former clause. The data acquisition for the evaluation of the one-way speech transmission quality at the acoustical interfaces as well as at the el
43、ectrical interface was performed by HEAD acoustics. The assessment of all TMOS values was carried out by Berkom. The data acquisition and the evaluation of the various transmission parameters, double-talk and background noise performance using artificial test signals were performed by HEAD acoustics
44、, at the electrical interface as well as at the acoustical interface. The IP-network simulator NIST-Net were provided and controlled by Berkom. For this test event the latest NIST-Net version 2.0.10 was used. ETSI ETSI TR 102 251 V1.1.1 (2003-10) 8 7 Test description 7.1 General test description The
45、 tests are based on the ETSI TIPHON specification as described in the latest version of TS 101 329-5 1. The test plan is subdivided in different parts. Measurements were done with three basic configurations: Electrical - Electrical Connection. Acoustical - Electrical Connection. Acoustical - Acousti
46、cal Connection. Measurements were conducted using two kinds of input signals: Real speech samples (English for demonstration, German for Listening Quality measurements). Artificial test signals (according to ITU-T Recommendation P.501 3). This results in the following matrix of input signal processi
47、ng: Electrical - Electrical Connection: Voice samples and artificial test signals. Acoustical - Electrical Connection: Voice samples and artificial test signals. Acoustical - Acoustical Connection: Voice samples and artificial test signals. After recording the samples were analyzed by T-Systems Nova
48、, Berkom using the Telecommunications Objective Speech Quality Assessment method TOSQA2001 2. This measure leads to objective TMOS values which allow direct comparisons to the TOSQA values of corresponding test conditions of the 1stETSI VoIP Quality Test Event. The TOSQA2001 analysis was carried out
49、 for all kind of measurement scenarios. In addition to the TOSQA2001 analysis, speech quality measures according to ITU-T Recommendation P.862 were performed for those scenarios where electrical - electrical scenarios were built up. Instrumental measurements for the chosen scenarios were performed by HEAD acoustics using sophisticated speech like test signals and analysis methods as published and described in 3, 4 and 12. The description of the signals is include
