1、 ETSI TR 102 648-1 V1.1.1 (2006-12)Technical Report Speech Processing, Transmission and Quality Aspects (STQ);Test Methodologies for ETSI Test Events and Results;Part 1: VoIP Speech Quality TestingETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 2 Reference DTR/STQ-00079-1 Keywords interoperability, quality,
2、 speech, VoIP ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present
3、document can be downloaded from: http:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of d
4、ispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and othe
5、r ETSI documents is available at http:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced except as authorized by
6、written permission. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2006. All rights reserved. DECTTM, PLUGTESTSTM and UMTSTM are Trade Marks of ETSI registered for the benefit of its Members. TIPHONTMand the TIPHON log
7、o are Trade Marks currently being registered by ETSI for the benefit of its Members. 3GPPTM is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 3 Contents Intellectual Property Rights4 Foreword.4 Introduct
8、ion 4 1 Scope 6 2 References 6 3 Abbreviations .8 4 The general structure of ETSI speech quality test events.8 4.1 Tests and test sessions 9 4.2 Test Reports10 5 Test description 10 5.1 General Test Description10 5.2 Tests Based on Instrumental Assessment of Speech Samples11 5.3 Tests Based on Speec
9、h like Test Signals according to ITU-T Recommendation P.501.12 6 Detailed Test Plan 13 6.1 Electrical - Electrical Measurements13 6.2 Electrical - Acoustical Measurements 15 6.3 Acoustical - Acoustical Measurements 18 7 Representation and Documentation of Test Results.20 7.1 Gateway Pies 21 7.2 Term
10、inal Pies26 History 31 ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 4 Intellectual Property Rights IPRs essential or potentially essential to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-mem
11、bers, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/webapp.etsi.org/IPR/home
12、.asp). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the
13、present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech Processing, Transmission and Quality Aspects (STQ). The present document is part 1 of a multi-part deliverable covering the procedures and results of the ETSI VoIP Speech Quality Test Events, a
14、s identified below: Part 1: “VoIP Speech Quality Testing“; Part 2: “Results of the 1stETSI VoIP Speech Quality Test Event“; Part 3: “Results of the 2ndETSI VoIP Speech Quality Test Event“; Part 4: “Results of the 3rdETSI VoIP Speech Quality Test Event“; Part 5: “Results of the 4thETSI VoIP Speech Qu
15、ality Test Event“. Introduction ETSI VoIP speech quality test events have been organized by ETSI Plugtest (formerly ETSI Bake-Off Service) since the year 2000. The main goals of the events always have been: Test of different VoIP implementations, terminals and gateways under the identical conditions
16、 for all participating manufacturers. Use and improve existing ESTI standards and give appropriate feedback to the ETSI standards bodies. Give a feedback to the manufacturers of VoIP equipment with respect to their performance in comparison to other vendors. The idea behind the ETSI VoIP test events
17、 is to measure, analyze and compare speech quality parameters for VoIP equipment. All conversational aspects like speech sound quality, echo measurements, double talk performance and the transmission quality in the presence of background noise are considered. These test events can be regarded as ver
18、y useful for all sides: The manufacturer: participate in a tutorial about speech quality measures; have one exclusive testing day; learn how the equipment performs in various test conditions; can derive useful information for system optimization in a special “consulting part“ during the testing day;
19、 get all his individual results including detailed information about potential improvements; ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 5 can compare his individual results to the results of all other participants being published in an anonymous test report. Comparison to the published results can be m
20、ade to the results of former SQTEs. Standardization bodies: achieve important data about current speech quality testing methods; can demonstrate the trend and development of speech quality testing methods; obtain an overview about conversational speech quality of current VoIP implementations availab
21、le; can demonstrate the trend and development of VoIP speech quality by comparing the results to those from the previous events. The present document describes the latest procedures used in the ETSI VoIP test events. It is used as a guideline for the implementation of such tests. ETSI ETSI TR 102 64
22、8-1 V1.1.1 (2006-12) 6 1 Scope The present document is a guideline for tests to be conducted in the ETSI VoIP speech quality test events. The present document describes: the general test conditions; the test setup; the test methodologies; and the result representations. Testing as described in the p
23、resent document ensures comparability of the results between tests performed in different test events. The test principles described in the present document are applicable for tests: between the two acoustic interfaces of a connection; between the acoustic interface and the electrical access point o
24、f a connection; between two electric interfaces of a VoIP connection. The present document covers narrowband connections and to some extent wideband connections. Besides gateways and handset terminals hands-free terminals are addressed. Conference configurations are out of scope of the present docum
25、ent. 2 References For the purposes of this Technical Report (TR) the following references apply: NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 1 2nd ETSI VoIP Speech Quality Test Event, Test Specification, Ver
26、sion 2.01, ETSI Plugtests, T-Systems Nova GmbH, Berkom, HEAD acoustics, April 2002. 2 1st ETSI VoIP Speech Quality Test Event, Test Specification, ETSI Bake-off Service, Deutsche Telekom Berkom, HEAD acoustics, October 2000. 3 ETSI TS 101 329-5: “Telecommunications and Internet Protocol Harmonizatio
27、n Over Networks (TIPHON) Release 3; End-to-end Quality of Service in TYPHON systems; Part 5: Quality of Service (QoS) measurement methodologies“. 4 “Proposal for enhancing the test program of the 3rd speech quality test event, HEAD acoustics“, ETSI, STQ#15 meeting, 8th to 12th December 2003, Dsseldo
28、rf, Germany. 5 ITU-T Recommendation P.800.1: “Mean Opinion Score (MOS) terminology“. 6 ITU-T Recommendation P.501: “Test Signals for Use in Telephonometry“. 7 ETSI EG 201 377-1: “Speech Processing, Transmission and Quality Aspects (STQ); specification and measurement of speech transmission quality;
29、Part 1: Introduction to objective comparison measurement methods for one-way speech quality across networks“. 8 ITU-T Recommendation P.862: “Perceptual evaluation of speech quality (PESQ), an objective method for end-to-end speech quality assessment of narrowband telephone networks and speech codecs
30、“. 9 ITU-T Recommendation P.800: “Methods for subjective determination of transmission quality“. ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 7 10 ITU-T Recommendation P.502: “Objective test methods for speech communication systems, using complex test signals“. 11 ITU-T Recommendation P.58: “Head and tor
31、so simulator for telephonometry“. 12 ITU-T Recommendation P.57: “Artificial ears“. 13 ITU-T Recommendation P.64: “Determination of sensitivity/frequency characteristics of local telephone systems“. 14 ETSI TIPHON temporary document 17TD135: “Subjective and objective speech quality evaluation on spee
32、ch data recorded at the SuperOp 99 event in Hawaii“. Sophia Antipolis, March 2000. 15 ITU-T Recommendation P.56: “Objective measurement of active speech level“. 16 ITU-T Recommendation P.79: “Calculation of loudness ratings for telephone sets“. 17 ITU-T Recommendation G.122: “Influence of national s
33、ystems on stability and talker echo in international connections“. 18 F. Kettler, H.W. Gierlich, F. Rosenberger:“Application of the Relative Approach to Optimize Packet Loss Concealment Implementations“ DAGA, March 2003, Aachen, Germany. 19 ITU-T Recommendation G.168: “Digital network echo canceller
34、s“. 20 ITU-T Recommendation P.340: “Transmission characteristics and speech quality parameters of hands-free terminals“. 21 ITU-T Recommendation P.505: “One-view visualization of speech quality measurement results“. 22 F. Kettler; F. Rosenberger; H.W. Gierlich: “Speech Quality “Quick Check“ for VoIP
35、 Terminals“, DAGA, March 22.-28., 2004, Strasbourg, France. 23 ETSI EG 201 377-2: “Speech Processing, Transmission and Quality Aspects (STQ); Specification and measurement of speech transmission quality; Part 2: Mouth-to-ear speech transmission quality including terminals“. 24 ITU-T Recommendation P
36、.862.1: “Mapping function for transforming P.862 raw result scores to MOS-LQO“. 25 3rd ETSI VoIP Speech Quality Test Event, Test Specification, ETSI Plugtests Service, HEAD acoustics, April 2004. 26 3rd ETSI VoIP Speech Quality Test Event, Test Specification, ETSI Plugtests Service, Anonymous Test R
37、eport Gateways. 27 3rd ETSI VoIP Speech Quality Test Event, Test Specification, ETSI Plugtests Service, Anonymous Test Report Terminals. 28 ETSI TR 102 526: “Speech Processing, Transmission and Quality Aspects: Wideband telephony considerations“. 29 ITU-T Recommendation P.862.2: “Wideband extension
38、to Recommendation P.862 for the assessment of wideband telephone networks and speech codecs“. 30 ITU-T Recommendation P.862.3: “Application guide for objective quality measurement based on Recommendations P.862, P.862.1 and P.862.2“. ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 8 3 Abbreviations For the
39、purposes of the present document, the following abbreviations apply: AEC Acoustic Echo Cancellers AGC Automatic Gain Control BGNT BackGround Noise Transmission CN Comfort Noise DSS1 Digital Subscriber System No.1 DT Double Talk ETSI European Telecommunications Standards Institute HATS Head And Torso
40、 Simulators HFT Hand-Free Telephone IP Internet Protocol ISDN Itegrated Services Digital Network ITU-T International Telecommunication Union - Telecommunication Standardization Sector JLR Junction Loudness Rating MOS Mean Opinion Score MOS-LQO Mean Opinion Score - Listening speech Quality Objective
41、MOS-LQON Mean Opinion Score - Listening speech Quality Objective, Narrowband MOS-LQOM Mean Opinion Score - Listening speech Quality Objective, Mixed MOS-LQOW Mean Opinion Score - Listening speech Quality Objective, Wideband NISTnet National Institute of Standards and Technology network emulation too
42、l NLP Non-Linear Processor OLR Overall Loudness Rating PESQ Perceptual Evaluation of Speech Quality PLC Packet Loss Concealment RLR Receiving Loudness Rating SLR Sending Loudness Rating SQTE Speech Quality Test Event TOSQA2001 Telecommunications Objective Speech Quality Assessment, version 2001 TMOS
43、 TOSQA Mean Opinion Score NOTE: Output of TOSQA2001. VAD Voice Activity Detection 4 The general structure of ETSI speech quality test events The goal of the ETSI SQTEs is to evaluate the performance of different VoIP equipment with respect to speech quality and conversational quality. ETSI and ITU-T
44、 standardized methods are used in order to determine the different parameters influencing the speech and conversational quality. It is expected that manufacturers bring either IP gateways or IP terminals for the tests. The IP terminals may be operated with handset, headset, in loud-hearing or hands-
45、free mode. One test lab typically performs speech quality measurements on the VoIP equipment of different manufacturers. These tests are instrumental (objective) and described in the present document. The test program of the 3rdETSI VoIP SQTE 26, which was an extension to the 1stand 2ndprogram of th
46、e events 1, 2 and the TS 101 329-5 3 is kept as a basis since it was well accepted by all participants and exact comparisons can be made to the previous events. The program is described in this technical report. A typical test event covers: Two days preparation for each manufacturer. A day tutorial
47、with presentations is given by the test lab about speech covering speech quality aspects, testing methods, test signals including appropriate analysis methods and algorithms. The whole program for the testing day and the settings that need to be configured in the equipment are also discussed in the
48、tutorial. An exclusive testing day being subdivided in predefined test sessions and so-called “freestyle“ sessions. ETSI ETSI TR 102 648-1 V1.1.1 (2006-12) 9 The test sessions are carried out with fix test conditions (packet loss rates, jitter conditions etc.) and system settings on those the partic
49、ipating manufacturers have agreed on (e.g. speech coder). Note that this agreement is necessary in order to compare the results. During the freestyle session each manufacturer can choose the tests or test conditions which he decides to be most interesting for him. A consulting part is offered during the testing day. This shall give guidelines for system optimization. It is therefore highly recommended for participants to: bring implementations that allow th
