1、 ETSI TR 102 648-2 V1.1.1 (2007-02)Technical Report Speech Processing, Transmission and Quality Aspects (STQ);Test Methodologies for ETSI Test Events and Results;Part 2: 1stETSI Plugtests Speech Quality Test Event ReportETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 2 Reference DTR/STQ-00079-2 Keywords int
2、eroperability, quality, speech, VoIP ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual
3、 copies of the present document can be downloaded from: http:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document For
4、mat (PDF). In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current
5、status of this and other ETSI documents is available at http:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced e
6、xcept as authorized by written permission. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2007. All rights reserved. DECTTM, PLUGTESTSTM and UMTSTM are Trade Marks of ETSI registered for the benefit of its Members. TIP
7、HONTMand the TIPHON logo are Trade Marks currently being registered by ETSI for the benefit of its Members. 3GPPTM is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. ETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 3 Contents Intellectual Property Right
8、s4 Foreword.4 1 Scope 5 2 References 5 3 Abbreviations .6 4 Summary 6 5 Overview 7 6 Test Description .8 6.1 General Test Description8 6.2 Measurement Scenarios8 6.2.1 Measurements using Electrical Interfaces.8 6.2.1.1 Measurement Setup.8 6.2.1.2 Measurement Conditions 9 6.2.2 Measurements using Aco
9、ustical Interfaces.10 6.2.2.1 Measurement Setup.10 6.2.2.2 Measurement Conditions 12 6.3 Measurement Methodology12 6.4 Test Signals 13 6.4.1 Voice Signals 13 6.4.2 Artificial Test Signals .13 6.5 Assessment Methods 19 6.5.1 Auditory Assessment 19 6.5.2 Instrumental Assessment 19 6.5.3 Instrumental C
10、omputational Assessment Using Speech-like (P.501) Test Signals 20 7 Results 20 7.1 Auditory Reference Test 20 7.1.1 Performance of the Auditory Test.20 7.1.1.1 TOSQA Results.20 7.2 Speech Quality Estimation Using Voice Signals22 7.2.1 G.711 Codec .23 7.2.2 G.723 Codec .23 7.2.3 G.729 Codec .23 7.2.4
11、 Summary of Results24 7.3 Advanced Measurements on Communicational Quality 25 7.3.1 Parameters determining speech sound quality under single talk conditions .25 7.3.2 Transmission Characteristics for Background Noise26 7.3.3 Transmission Performance under Double Talk Conditions 26 7.3.4 Detailed Ana
12、lysis of Echo during Double Talk 29 8 Conclusion31 History 32 ETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 4 Intellectual Property Rights IPRs essential or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly
13、 available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
14、 server (http:/webapp.etsi.org/IPR/home.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or m
15、ay be, or may become, essential to the present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech Processing, Transmission and Quality Aspects (STQ). The present document is part 2 of a multi-part deliverable. Full details of the entire series can be f
16、ound in part 1 19. ETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 5 1 Scope The present document contains the anonymous Test Report from the 1stETSI Plugtests Speech Quality Test Event. 2 References For the purposes of this Technical Report (TR) the following references apply: NOTE: While any hyperlinks in
17、cluded in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 1 ITU-T Recommendation P.800: “Methods for subjective determination of transmission quality“. 2 ETSI EG 201 377-1: “Speech Processing, Transmission and Quality Aspects (STQ); Specification and
18、 measurement of speech transmission quality; Part 1: Introduction to objective comparison measurement methods for one-way speech quality across networks“. 3 ITU-T Recommendation P.501: “Test signals for use in telephonometry“. 4 ITU-T Recommendation P.502: “Objective test methods for speech communic
19、ation systems using complex test signals“. 5 ITU-T Recommendation P.58: “Head and torso simulator for telephonometry“. 6 ITU-T Recommendation P.57: “Artificial ears“. 7 ETSI TIPHON temporary document 17TD135: “Subjective and objective speech quality evaluation on speech data recorded at the SuperOp
20、99 event in Hawaii. Sophia Antipolis, March 2000“. 8 ITU-T Recommendation P.64: “Determination of sensitivity/frequency characteristics of local telephone systems“. 9 ITU-T Recommendation P.79: “Calculation of loudness ratings for telephone sets“. 10 ITU-T Recommendation G.122: “Influence of nationa
21、l systems on stability and talker echo in international connections“. 11 ITU-T Recommendation P.56: “Objective measurement of active speech level“. 12 ITU-T Recommendation P.830: “Subjective performance assessment of telephone-band and wideband digital codecs“. 13 ITU-T Recommendation P.810: “Modula
22、ted noise reference unit (MNRU)“. 14 21TD68: “Proposal for 2nd Speech quality test event“, Reinhard Scholl. 15 21TD95: “Preliminary test report“, T-Nova Berkom End-to-end Quality of Service in TIPHON systems; Part 5: Quality of Service (QoS) measurement methodologies“. 19 ETSI TR 102 648-1: “Speech
23、Processing, Transmission and Quality Aspects (STQ); Test Methodologies for ETSI Test Events and Results; Part 1: VoIP Speech Quality Testing“. 20 ITU-T Recommendation P.340: “Transmission characteristics and speech quality parameters of hands-free terminals“. ETSI ETSI TR 102 648-2 V1.1.1 (2007-02)
24、6 21 ETSI TBR 8: “Integrated Services Digital Network (ISDN); Telephony 3,1 kHz teleservice; Attachment requirements for handset terminals“. 22 ITU-T COM12-117E, March 2000. 3 Abbreviations For the purposes of the present document, the following abbreviations apply: AGC Automatic Gain Control ASL Ac
25、tive Speech Level CAS Communication Analysis System CSS Composite Source Signal ERL Echo Return Loss HATS Head And Torso Simulator IP Internet Protocol IRS Intermediate Reference System ISDN Integrated Services Digital Network JLR Junction Loudness Rating MNRU Modulated Noise Reference Unit MOS Mean
26、 Opinion Score NOTE: Output of TOSQA. NIST National Institute of Standards and Technology OLR Overall Loudness Rating OVL Over-Load Point PBX Public Branch Exchange PLC Packet Loss ConcealmentPVS PC Voice Switch RLR Receive Loudness Rating RTP Real time Transport Protocol SLR Send Loudness Rating TM
27、OS TOSQA Mean Opinion Score TOSQA Telecommunications Objective Speech Quality Assessment VAD Voice Activity Detection 4 Summary The European Telecommunications Standards Institute (ETSI) organized a special test event for VoIP (Voice over Internet Protocol) speech quality in Sophia Antipolis, France
28、, from 23rdof October to 1stNovember, 2000. T-Nova Deutsche Telekom Innovationsgesellschaft mbH Berkom, in collaboration with HEAD acoustics GmbH, performed speech quality measurements on VoIP equipment of different manufacturers. Texas Instruments Incorporated and Alcatel co-sponsored the test even
29、t. The aim of the test event was to determine the speech quality of various Voice over IP equipment under certain IP network conditions. During the test event, speech material as well as measurement data were collected by transferring voice samples and artificial signals across the Voice over IP set
30、up. Speech quality was measured by both instrumental (objective) and auditory (subjective) methods. Both methods were used to measure the one-way speech quality (listening quality). The important transmission parameters determining conversational quality like double talk performance, background nois
31、e transmission and echo performance were accessed using sophisticated test signals and enhanced analysis methods as described in TS 101 329-5 18 and recent ITU-T Recommendations. The one-way speech transmission quality was evaluated by processing real speech samples and analysing it using the TOSQA
32、algorithm. To validate the TOSQA algorithm, auditory reference tests according to ITU-T Recommendations of the P.800 series were carried out. Correlations of 91,6 % and 93,6 % for listening quality and connection quality, respectively, demonstrate the high accuracy of TOSQA for VoIP transmission sce
33、narios tested here. A subset of speech recordings were carried out using the HATS (head and torso simulator) HMS II.3 of HEAD acoustics equipped with type 3.4 artificial ears. For these conditions a separate auditory test was conducted and the speech material was also assessed by the new version TOS
34、QA2001 terminal extension. Here a correlation of 98 % was derived. ETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 7 Instrumental measurements using sophisticated test signals and analysis methods according to recent ITU-T Recommendations of the P.500 series were conducted covering all conversational aspect
35、s like single talk and double talk periods or echoes. These tests are specially designed to analyse and optimize parameters determining conversational quality, quality of background noise transmission, the performance of echo cancellers and others. These measurements were carried out at the acoustic
36、al interface using IP terminals or standard ISDN telephones mounted to the HATS and at the electrical interface for gateway testing. The results provide important information for the manufactures about conversational speech quality of their equipment. In particular, the tests determined parameters l
37、ike: distortions, AGC (automatic gain control), VAD (voice activity detection) or PLC (packet loss concealment) implementations under single talk conditions; double talk performance influenced by level variations, clipping and echoes; echo canceller performance determined by convergence characterist
38、ics, spectral echo attenuation, NLP implementation; quality of background noise transmission, clipping, voice activity detection or the design of comfort noise injection. Based on the results the following tests and test conditions for conversational speech quality are suggested for standardization.
39、 Specific echo canceller tests for the VoIP equipment including low Echo Return Losses (ERL) of 6 dB (simulating worst case echo conditions in networks) and high ERL 40 dB (simulating typical ISDN connections). On the one hand the implemented echo cancellers should guarantee a sufficient echo attenu
40、ation but on the hand the echo cancellers should not degrade the performance of the network for high ERL values in the echo path. These echo canceller tests should be carried out and analysed under single and double talk conditions. The occurrence of signal gaps (clipping) under double talk conditio
41、ns should especially be tested under network condition including high ERL values. Again the implemented signal processing in VoIP equipment should not degrade the network performance if no packet loss and no delay jitter is introduced during the test. The quality of background noise transmission tog
42、ether with implemented comfort noise injection should be tested. The tests should determine the adaptation of injected comfort noise on the actual background noise level and spectrum. The results of this test event are being published in the present document. Parts of it will be included in the docu
43、ment ETSI TIPHON 05013 TR 101 329-6 “Actual measurement test results“. The report will also be presented at ETSI STQ and ITU-T Study Group 12. The data will provide input for new or enhanced standards and recommendations for enhanced VoIP communications. Furthermore, the results can be used for opti
44、mization of the manufacturers“ VoIP equipment to improve the overall speech quality. Due to the benefit of such an event it is strongly recommended to continue the process of end-to-end speech quality testing. To support this idea a second ETSI VoIP test event is currently being prepared and planned
45、. 5 Overview The present document describes the test methodologies, the assessment methods and the results of the measurements which were carried out during the 1stETSI VoIP speech quality test event. The aim of the test event was to determine the speech quality of various Voice over IP equipment un
46、der certain IP network conditions. During the test event, speech material as well as measurement data were collected by transferring voice samples and artificial signals across the Voice over IP setup. This material was analysed and the results are reported in the present document. The analysis of t
47、he collected data can be split in two parts. In the first part the assessment of the one-way speech quality (listening quality) was performed by both, auditory and instrumental assessments. In the second part the analysis of various transmission parameters, double talk performance and background noi
48、se transmission was performed and different transmission parameters were indicated. ETSI ETSI TR 102 648-2 V1.1.1 (2007-02) 8 The one-way speech transmission quality was evaluated by processing real speech samples and analysing it using the TOSQA algorithm. TOSQA leads to MOS-comparable results. To
49、validate the TOSQA results an auditory reference test was carried out. A detailed description of the relationship of a reference MOS evaluation according ITU-T Recommendation P.800 series Recommendations and the relating TOSQA results is given. In the second main part of the document, measurement results based on recent ITU-T Recommendations (P.500 series, P.340 20) are included. These measurement results provide information about various transmission parameters from which double talk performance and background noise tran
