1、STD-ITU-T RECMN P-833-ENGL 3998 I 48bE!59% Ob5742b lL9 INTERNATIONAL TELECOMMUNICATION UNION ITU-T TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU P.831 (I 2/98) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Methods for objective and subjective assessment of
2、quality Subjective performance evaluation of network echo cancellers ITU-T Recommendation P.831 (Previously CCITT Recommendation) STD-ITU-T RECMN P.831-ENGL 1998 I 48b2591 Ob57427 055 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS - Vocabu
3、lary and effects of transmission parameters on customer opinion of Series P.10 transmission quality Subscribers lines and sets Series P.30 Transmission standards Series P.40 Objective measuring apparatus Series P.50 Objective electro-acoustical measurements Series P.60 II Measurements related to spe
4、ech loudness Series P.70 P.300 P.500 Audiovisual quality in multimedia services Series P.900 - For further details, please refer to ITU-T List of Recommendations. STD-ITU-T RECMN P-83L-ENGL 1998 = 4862591 0657428 T9L ITU-T RECOMMENDATION P.831 SUBJECTIVE PERFORMANCE EVALUATION OF NETWORK ECHO CANCEL
5、LERS Summary This Recommendation describes methods and procedures for conducting subjective performance evaluations of network echo cancellers. The deployment of digital technology in the Public Switched Telephone Network (PSTN) has had numerous advantages for users of the network as well as for net
6、work operators. These new technologies come at the price of increased transmission time, which increases the likelihood that any echo impairment will be annoying to voice users of the network. Hence, the deployment of echo cancellers in the network is widespread. Recommendations G.165 and G.168 defi
7、ne certain instrumental tests that must be met to ensure minimum performance of an echo canceller. However, there has been some concern that those tests do not address fully the echo cancellation needs of voice users of the network. Subjective testing is a commonly used method of assessing the perfo
8、rmance of digital devices, including digital speech codecs and Digital Circuit Multiplication Equipment (DCME). This Recommendation defines natural extensions of those techniques to the subjective evaluation of echo cancellers. Source ITU-T Recommendation P.831 was prepared by ITU-T Study Group 12 (
9、1997-2000) and was approved under the WTSC Resolution No. 1 procedure on the 3rd of December 1998. Keywords Echo cancellation, subjective performance, speech transmission quality. Recommendation P.831 (12/98) 1 FOREWORD ITU (International Telecommunication Union) is the United Nations Specialized Ag
10、ency in the field of telecommunications. The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of the ITU. The ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a
11、worldwide basis. The World Telecommunication Standardization Conference (WTSC), which meets every four years, establishes the topics for study by the ITU-T Study Groups which, in their turn, produce Recommendations on these topics. The approval of Recommendations by the Members of the ITU-T is cover
12、ed by the procedure laid down in WTSC Resolution No. 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with IS0 and IEC. NOTE In this Recommendation the term recognized operating agency (ROA) includes any indivi
13、dual, company, corporation or governmental organization that operates a public correspondence service. The terms Administration, ROA and public correspondence are defined in the Constitution of the ITU (Geneva, 1992). INTELLECTUAL PROPERTY RIGHTS The ITU draws attention to the possibility that the p
14、ractice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. The ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation
15、 development process. As of the date of approval of this Recommendation, the ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, impiementors are cautioned that this may not represent the latest information and
16、are therefore strongly urged to consult the TSB patent database. O ITU 1999 Ail rights reserved. No part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from the ITU. 11 Recom
17、mendation P.831 (12/98) STD-ITU-T RECMN P.831-ENGL 1998 111 48b2591i Ob57430 b4T - CONTENTS Page 1 1.1 1.2 1.3 1.4 1.5 2 2.1 2.2 2.3 3 3.2 3.3 3.4 4 4.1 4.2 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7 General . Scope References Terms and definitions . Abbreviations . Conventions Overview of subjective testing pro
18、cedures for Echo Cancellers Conversational test . Talking-and-listening test . Third-party listening tests . General considerations for subjective evaluation of echo cancellers . Echo canceller parameters to evaluate Test equipment and calibration Selection of subjects . Analysis of results Conversa
19、tion tests . Purpose . 4.1.1 Advantages 4.1.2 Disadvantages Test design, Set-up and procedure Test design and circuit conditions . 4.2.1 4.2.2 Reference conditions . 4.2.3 Task . 4.2.4 Opinion scales and questionnaires Talking-and-listening test . Purpose . Parameters Set-up Description of test proc
20、edure 5.4.1 Initial convergence 5.4.2 Steady state conditions Advantages . Disadvantages . Reference conditions Recommendation P.831 (12/98) i 1 1 2 2 3 3 5 7 7 7 8 8 8 9 9 9 10 10 10 11 11 12 12 13 13 13 13 . 111 STD-ITU-T RECMN P-63B-ENGL 1996 9 4662591 Ob5743L 58b 5.8 6 6.1 6.2 6.3 6.4 6.5 6.6 6.
21、7 6.8 6.9 6.10 7 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 Precautions . Third-party listening Test A . Purpose . Parameters Set.up Recording procedure for original source material Recording procedure for listening test material . Description of the listening test procedure . 6.6.1 Playback Advantages . D
22、isadvantages . Reference conditions Precautions . Third-party listening Test B . Purpose . 6.6.2 Subjects . Impairments to evaluate . Set-up Advantages . Disadvantages . Reference conditions Precautions . Descnption of test procedure 7.8.1 Recording Evaluation . 7.9.1 Playback 7.9.2 Subjects . Annex
23、 A . Questionnaires for use in conversational tests A . 1 Introduction A.2 Sample questions Appendix I - Example test conditions for echo canceller evaluations Page 14 14 14 14 15 16 17 18 18 18 18 18 19 19 19 19 20 20 20 21 21 21 21 21 22 22 22 23 23 23 24 iv Recommendation P.831 (12198) Recommenda
24、tion P.831 SUBJECTIVE PERFORMANCE EVALUATION OF NETWORK ECHO CANCELLERS (Geneva, 1998) 1 General 1.1 Scope This Recommendation describes procedures to be used to assess the subjective performance of echo cancellers. The methods defined here may be used to assess the extent to which an echo canceller
25、 operates effectively for voice users of the PSTN. In particular, the intent is not to define methods that may be used to assess the effects of delay, nor is it the intent to define rules for echo canceller application. These issues are addressed in Recommendations G. 1 14 and G. 13 1 , respectively
26、. Further, this Recommendation does not define specific values for echo canceller parameters (e.g. convergence time) to yield satisfactory subjective performance. The procedures defined here may also be appropriate for evaluating the subjective performance of other signal processing devices that may
27、 be deployed in the PSTN (e.g. Automatic Level Control devices). These issues are under study in ITU-T Study Group 12. 1.2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At th
28、e time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; all users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations listed below. A list of the
29、currently valid ITU-T Recommendations is regularly published. - - - - - - - - - ITU-T Recommendation P.800 (1 996), Methods for subjective determination of CCITT Recommendation G. 164 (1 988), Echo suppressors. ITU-T Recommendation G.165 (1993), Echo cancellers. ITU-T Recommendation G. 168 (1997), D
30、igital network echo cancellers. ITU-T Recommendation P.5 1 (1996), Artifzcial mouth. ITU-T Recommendation P.56 (1993), Objective measurement ofactive speech level. ITU-T Recommendation P.57 (1 996), Artificial ears. ITU-T Recommendation P.58 (1 996), Head and torso simulator for telephonometry. ITU-
31、T Recommendation P.501 (1996), Test signalsfor use in telephonometry. transmission quality. ITU-T Recommendation P.8 1 O (1 996), Modulated Noise Reference Unit (34NRU). ITU-T Recommendation P. 830 (1 996), Subjective performance assessment of telephone- band and wideband digital codecs. ITU Handboo
32、k on Telephonometvy, 2nd edition, Geneva 1992. - - - Recommendation P.831 (12/98) 1 m 48b2591 Ob57433 359 1.3 Terms and definitions This Recommendation defines the following terms: 1.3.1 1.3.2 echo canceller is intended to operate. 1.3.3 1.3.4 near-end: The side of an echo canceller which contains t
33、he echo. This includes all transmission facilities and equipment (including the hybrid and terminating telephone) which is included in the echo path. 1.3.5 syllable clipping or temporal clipping: Loss of speech energy caused by voicelspeech activated devices. For echo cancellers, the primary source
34、of temporal clipping is the NLP. In this instance, clipping does not refer to amplitude limiting. 1.3.6 talker echo loudness rating: The loudness loss in the talker echo path. (see Recommendation G. 100) 1.3.7 third-party listening test: A listening-only subjective test (see Recommendation P.800) in
35、 which the listener hears recordings from the “center“ of the connection under evaluation, In conventional listening-only tests, the listener is positioned at one end of the connection under study. 1.3.8 conversation test: A subjective test in which two participants have a conversation, as described
36、 in Annex AP.800 and in the Handbook on Telephonometry. 1.3.9 talking-and-listening test: A subjective test in which a participant talks while simultaneously listening for impairments (e.g. echo). 1.3.10 untrained subject: See 3.3.1. 1.3.11 experienced subject: See 3.3.2. double-talk When near-end a
37、nd far-end speech occur simultaneously. far-end: The side of an echo canceller which does not contain the echo path on which the live network: A telephony network in commercial use. 1.4 Abbreviations This Recommendation uses the following abbreviations: ACR ATM DCME DCR DMOS EC ERL ERLE HATS LRGP MN
38、RU MOS MRP NLP OLR 2 Absolute Category Rating Asynchronous Transfer Mode Digital Circuit Multiplication Equipment Degradation Category Rating Degradation Mean Opinion Score Echo Canceller Echo Return Loss Echo Return Loss Enhancement Head and Torso Simulator Lip Ring Guard Position Modulated Noise R
39、eference Unit Mean Opinion Score Mouth Reference Point Non-Linear Processor Overall Loudness Rating Recommendation P.831 (12/98) STO-ITU-T RECMN P.83L-ENGL II998 E 48b2591 Ob57434 295 PCME Packetized Circuit Multiplication Equipment PLMN Public Land Mobile Network PSTN Public Switched Telephone Netw
40、ork RLR Receiving Loudness Rating SLR Sending Loudness Rating TELR Talker Echo Loudness Rating 1.5 Conventions As described in this Recommendation, subjective evaluation of echo cancellers may be conducted using listening-only, talking-and-listening, or conversational methods. The test procedures de
41、fined in this Recommendation may also be useful for evaluation of other active speech signal processing device s. 2 Table 1 lists four subjective testing methods that have been found suitable for evaluating the subjective performance of Echo Cancellers (ECs). Each testing method is described briefly
42、 in this clause. Detailed descriptions are found in other clauses. Each listing in Table 1 includes the possible applications of that type of test. Four classes for application of the methods are identified: Overview of subjective testing procedures for Echo Cancellers o evaluation of overall opinio
43、n and/or quality of connections with ECs from the viewpoint of a identification of EC parameters that are important for voice users of the PSTN (e.g. tandem selection of values for those parameters (e.g. speed of convergence, minimum amount of typical voice user of the PSTN; operation of ECs, echo r
44、eturn loss enhancement); echo return loss enhancement); O o a diagnostic evaluation of specific problems involving ECs. Each of the testing methods may be used with untrained or experienced participants. However, each procedure may have different application depending on the amount of technical expe
45、rience a particular participant has with ECs. For example, conversational tests with untrained participants would be especially useful for “global“ evaluation of EC performance. On the other hand, a conversational test between experts might serve as a diagnostic tool. In general, it is not recommend
46、ed that Talking-and-Listening Tests, and Listening-Only Tests be performed in isolation. A complete evaluation of echo canceller performance must take int.0 account conversational interactions between subjects. Recommendation P.831 (12/98) 3 STD-ITU-T RECMM P.63L-ENGL 1998 M 48b259L Ob57435 L2L Tabl
47、e llp.831- Applications of subjective testing methods Overall Opinion and/or Quality Parameter Value Selection Parameter Identification Testing method Conversational Test Talking-and-Listening Test Diagnostic Parameter Value Selection Parameter Identification Third-party Listening Type B Diagnostic
48、Overall Opinion and/or Quality Parameter Value Selection Parameter Identification Untrained subjects I Experienced subjects Diagnostic Parameter Value Selection Parameter Identification Diagnostic Overall Opinion and/or Quality Parameter Value Selection Parameter Identification Parameter Value Selec
49、tion Diagnostic Diagnostic Parameter Value Selection Parameter Identification Diagnostic Parameter Value Selection Parameter Identification NOTE - Third-party listening Type A - see 2.3, below. Third-party listening Type B - see 2.3, below. In order to use the suite of test methods effectively, it will be helpful to understand the relationship among the various types of tests (e.g. MOS on conversational vs. MOS on Listening Type A). This topic is currently under study. 2.1 Conversational test Conversational test procedures for ECs are described in clause 4. Conversational tests have the
