ETSI TR 102 648-3-2007 Speech Processing Transmission and Quality Aspects (STQ) Test Methodologies for ETSI Test Events and Results Part 3 2nd ETSI Plugtests Speech Quality Test Ev_1.pdf

1、 ETSI TR 102 648-3 V1.1.1 (2007-02)Technical Report Speech Processing, Transmission and Quality Aspects (STQ);Test Methodologies for ETSI Test Events and Results;Part 3: 2ndETSI Plugtests Speech Quality Test Event ReportETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 2 Reference DTR/STQ-00079-3 Keywords int

2、eroperability, quality, speech, VoIP ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual

3、 copies of the present document can be downloaded from: http:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document For

4、mat (PDF). In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current

5、status of this and other ETSI documents is available at http:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced e

6、xcept as authorized by written permission. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2007. All rights reserved. DECTTM, PLUGTESTSTM and UMTSTM are Trade Marks of ETSI registered for the benefit of its Members. TIP

7、HONTMand the TIPHON logo are Trade Marks currently being registered by ETSI for the benefit of its Members. 3GPPTM is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 3 Contents Intellectual Property Right

8、s4 Foreword.4 1 Scope 5 2 References 5 3 Abbreviations .6 4 Summary 6 5 Introduction 7 5.1 Time and location.7 6 Responsibilities 7 7 Test description 8 7.1 General test description 8 7.2 Measurement scenarios 9 7.2.1 Measurements using electrical interfaces .9 7.2.1.1 Measurement setup9 7.2.1.2 Mea

9、surement conditions .9 7.2.2 Acoustical measurements .10 7.2.2.1 Measurement setup10 7.2.3 Measurement conditions .12 7.3 Measurement methodology 13 7.4 Test signals.15 7.4.1 Voice signals.15 7.4.2 Test signals according to ITU-T Recommendation P.501 15 7.5 Assessment methods.23 7.5.1 Instrumental a

10、ssessment of listening quality 23 7.5.2 TOSQA and PESQ results and precision23 7.5.3 Instrumental computational assessment using speech-like (P.501) test signals24 8 Results 25 8.1 Estimation of one-way speech quality25 8.1.1 G.711 Codec .25 8.1.2 G.729 Codec .26 8.2 Delay measurements.29 8.3 Transm

11、ission parameters, double talk performance and background noise transmission 29 8.3.1 Tests with G.711 Codec30 8.3.2 Echo canceller performance test with G.711 Codec .38 8.3.3 Tests with G.729A Codec.42 History 46 ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 4 Intellectual Property Rights IPRs essential

12、or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Es

13、sential, IPRs notified to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/webapp.etsi.org/IPR/home.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETS

14、I. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech Pr

15、ocessing, Transmission and Quality Aspects (STQ). The present document is part 3 of a multi-part deliverable. Full details of the entire series can be found in part 1 18. ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 5 1 Scope The present document contains the anonymous Test Report from the 2ndSpeech Qual

16、ity Test Event 2002. 2 References For the purposes of this Technical Report (TR) the following references apply: NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 1 ETSI TS 101 329-5: “Telecommunications and Inter

17、net Protocol Harmonization Over Networks (TIPHON) Release 3; End-to-end Quality of Service in TIPHON systems; Part 5: Quality of Service (QoS) measurement methodologies“. 2 ETSI EG 201 377-1: “Speech Processing, Transmission and Quality Aspects (STQ); specification and measurement of speech transmis

18、sion quality; Part 1: Introduction to objective comparison measurement methods for one-way speech quality across networks“. 3 ITU-T Recommendation P.501: “Test signals for use in telephonometry“. 4 ITU-T Recommendation P.502: “Objective test methods for speech communication systems using complex tes

19、t signals“. 5 ITU-T Recommendation P.58: “Head and torso simulator for telephonometry“. 6 ITU-T Recommendation P.57: “Artificial ears“. 7 ETSI TIPHON temporary document 17TD135: “Subjective and Objective Speech Quality Evaluation on Speech Data recorded“ at the SuperOp 99 event in Hawaii. Sophia Ant

20、ipolis, March 2000. 8 ITU-T Recommendation P.64: “Determination of sensitivity/frequency characteristics of local telephone systems“. 9 ITU-T Recommendation P.79: “Calculation of loudness ratings for telephone sets“. 10 ITU-T Recommendation G.122: “Influence of national systems on stability and talk

21、er echo in international connections“. 11 ITU-T Recommendation P.56: “Objective measurement of active speech level“. 12 ITU-T Recommendation P.830: “Subjective performance assessment of telephone-band and wideband digital codecs“. 13 ITU-T Recommendation P.810: “Modulated noise reference unit (MNRU)

22、“. 14 ITU-T COM12-D41 Deutsche Telekom AG (Q9/12): “Enhancement of P.862 results by post-processing using “TOCQ“. 15 ITU-T Recommendation P.800: “Methods for subjective determination of transmission quality“. 16 Genuit, K.: “Objective Evaluation of Acoustic Quality based on a Relative Approach“, Int

23、erNoise Proceedings, Liverpool, UK, 1996. 17 Sottek, R.: “Modelle zur Signalverarbeitung im menschlichen Gehr“, PHD thesis RWTH Aachen, 1993. 18 ETSI TR 102 648-1: “Speech Processing, Transmission and Quality Aspects (STQ); Test Methodologies for ETSI Test Events and Results; Part 1: VoIP Speech Qua

24、lity Testing“. ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 6 19 ITU-T Recommendation P.862: “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs“. 20 ITU-T Recommendation G.168: “Digital network

25、echo cancellers“. 21 ETSI TBR 008: “Integrated Services Digital Network (ISDN); Telephony 3,1 kHz teleservice; Attachment requirements for handset terminals“. 22 ITU-T Recommendation G.114: “One-way transmission time“. 3 Abbreviations For the purposes of the present document, the following abbreviat

26、ions apply: ACQUA Advanced Communication Quality Analysis ASL Active Speech Level BRI Basic Rate Interface CSS Composite Source Signal DSS1 European ISDN Protocol ERL Echo Return Loss HATS Head And Torso Simulator IP Internet Protocol IRS Intermediate Reference System ISDN Integrated Services Digita

27、l Network MNRU Modulated Noise Reference Unit MOS Mean Opinion Score NIST-Net Network Simulation Tool from National Institute of Standards and Technology PABX Private Automatic Branch Exchange PLC Packet Loss Concealment RTP Real Time Transport Protocol TMOS TOSQA Mean Opinion Score NOTE: Output of

28、TOSQA. TOSQA Telecommunications Objective Speech Quality Assessment VAD Voice Activity Detection VAD Voice Activity Detection 4 Summary The present document describes the test methodologies, the assessment methods and the results of the measurements which were carried out during the 2ndETSI Speech Q

29、uality Test Event. The tests were conducted by T-Systems Nova GmbH Berkom (Berkom) and HEAD acoustics GmbH (HEAD acoustics). The aim of the test event was to determine the speech quality of various Voice over IP equipment under certain IP network conditions. During the test event, speech material as

30、 well as measurement data were collected by transferring voice samples and artificial signals across the Voice over IP setup. This material was analysed and the results are reported in the present documentation. The analysis of the collected data can be split in two parts. In the first part the asse

31、ssment of the one-way speech quality (listening quality) was performed by instrumental assessments. The one-way speech transmission quality was evaluated by processing real speech samples and analyzing it using the TOSQA2001 algorithm and in addition using the current ITU-T Recommendation P.862 19 “

32、PESQ“. TOSQA2001 and PESQ lead to MOS-comparable results on a scale from 1 to 5 or -1 to 5, respectively. For all measurements at the acoustical interface using artificial ears (e.g. measurements of IP-Phones) only TOSQA2001 was used, because PESQ is not applicable for evaluation of acoustical recor

33、ded signals according to P.862 documentation. PESQ and TOSQA were used in all measurement scenarios that were based on measurements at the electrical interfaces at sending and receiving side (e.g. Gateway-Gateway). ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 7 In the second part more detailed analyses w

34、ere carried out. The various strategies of PLC implementations were investigated in detail as well as VAD- and comfort noise implementations. Furthermore additional situations were evaluated. The system performance with background noise during double talk and specifically the echo canceller characte

35、ristics. These tests were based on TS 101 329-5 1 and ITU-T Recommendations P.501 3, P.502 4 and G.168 20.These test show the performance of the different implementations with respect to speech quality in detail and can be used to optimize the system design. 5 Introduction The tests conducted during

36、 the 2ndETSI TIPHON VoIP Speech Quality Test Event have been optimized and adapted considering the feedback from all participating manufactures after the 1stETSI VoIP Speech Quality Test Event, concentrating on a more detailed evaluation of specific parameters, optimizing the number of network condi

37、tions and discussing the manufacturers implementation based on the test results already during the test event. In order to ensure consistency of the measurements of the 1stETSI VoIP Speech Quality Test Event and to allow comparisons of results as well, instrumental measures for electrical and acoust

38、ical VoIP scenarios were conducted with TOSQA2001. In addition, instrumental measurements were conducted according to ITU-T Recommendation P.862 for those scenarios where P.862 is suitable. Instrumental measurements were performed for one-way speech transmission as well as double-talk situations by

39、evaluating background noise transmission performance, PLC implementations and echo cancelling characteristics in detail. The measurements conducted by HEAD acoustics were based on test signals and test procedures as described in TS 101 329-5 1 using test signals and analysis procedures described in

40、ITU-T Recommendations P.501 3, P.502 4 and G.168. Furthermore, each company conducted a “free-style“ testing block to evaluate specific test conditions. In addition to the tests, an extra day tutorial session was held for each manufacturer. Within ETSI, two bodies are actively involved in this event

41、. ETSI Project TIPHON (Telecommunications and Internet Protocol Harmonization Over Networks) and ETSI Technical Committee STQ (Speech Processing, Transmission and Quality Aspects) look for the improvement of quality aspects in the voice transmission area. 5.1 Time and location The measurements were

42、conducted during the 2ndETSI Speech Quality Test Event at ETSI Headquarters in Sophia Antipolis, France from 16thto 24thof April 2002. 6 Responsibilities The present document contains results of both types of measurements, for measurement at the electrical network-interface of the used PBX as well a

43、s at the acoustical interface using a telephone-handset which was connected to the PBX. IP-phones were connected to the IP-network. Technical details for the various types of measurements can be found later in the present document. This clause is intended to provide information about the responsibil

44、ity of the test labs (Berkom, HEAD acoustics) for the respective measurements, quality assessments and parts of the reports. So, if you have any questions concerning the present document you may contact the responsible test lab. The contact information can be found in the former clause. The data acq

45、uisition for the evaluation of the one-way speech transmission quality at the acoustical interfaces as well as at the electrical interface was performed by HEAD acoustics. The assessment of all TMOS values was carried out by Berkom. The data acquisition and the evaluation of the various transmission

46、 parameters, double-talk and background noise performance using artificial test signals were performed by HEAD acoustics, at the electrical interface as well as at the acoustical interface. The IP-network simulator NIST-Net were provided and controlled by Berkom. For this test event the latest NIST-

47、Net version 2.0.10 was used. ETSI ETSI TR 102 648-3 V1.1.1 (2007-02) 8 7 Test description 7.1 General test description The tests are based on the ETSI TIPHON specification as described in the latest version of TS 101 329-5 1. The test plan is subdivided in different parts. Measurements were done wit

48、h three basic configurations: Electrical - Electrical Connection. Acoustical - Electrical Connection. Acoustical - Acoustical Connection. Measurements were conducted using two kinds of input signals: Real speech samples (English for demonstration, German for Listening Quality measurements). Artifici

49、al test signals (according to ITU-T Recommendation P.501 3). Electrical - Electrical Connection: Voice samples and artificial test signals. Acoustical - Electrical Connection: Voice samples and artificial test signals. Acoustical - Acoustical Connection: Voice samples and artificial test signals. After recording the samples were analysed by T-Systems Nova, Berkom using the Telecommunications Objective Speech Quality Assessment method TOSQA2001 2. This measure leads to objective TMOS values wh