ITU-T P 830-1996 Subjective Performance Assessment of Telephone- Band and Wideband Digital Codecs - Telephone Transmission Quality Methods for Objective and Subjective Assessment o.pdf

1、ITU-T RECMN*P-830 96 4862591 Ob18033 TIT = INTERNATIONAL TELECOMMUNICATION UNION ITU-T TELECOMMUNICATION STAN DARD IZATION SECTOR OF ITU P.830 (02/96) TELEPHONE TRANSMISSION QUALITY METHODS FOR OBJECTIVE AND SUBJECTIVE ASSESSMENT OF QUALITY SUBJECTIVE PERFORMANCE ASSESSMENT DIGITAL CODECS OF TELEPHO

2、NE-BAND AND WIDEBAND ITU-T Recommendation P.830 (Previously “CCITT Recommendation”) COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T RECMN*P.830 96 = LiBb257L Ob18034 9-56 D FOREWORD The ITU-T (Telecommunication Standardization Sec

3、tor) is a permanent organ of the International Telecommunication Union (ITLJ). The ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommen- dations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Stand

4、ardization Conference (WTSC), which meets every four years, establishes the topics for study by the IT-T Study Groups which, in their turn, produce Recommendations on these topics. The approval of Recommendations by the Members of the IT-T is covered by the procedure laid down in WTSC Resolution No.

5、 1 (Helsinki, March 1-12, 1993). IT-T Recommendation P.830 was revised by IT-T Study Group 12 (1993-1996) and was approved under the WTSC Resolution No. 1 procedure on the 6th of February 1996. NOTE! In this Recommendation, the expression “Administration” is used for conciseness to indicate both a t

6、elecommunication administration and a recognized operating agency. O ITU 1996 All rights reserved. No part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from the IW. COPYRIG

7、HT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling Services8 9 10 11 ITU-T RECMN*PnB30 b W 4862591 Ob18035 892 m CONTENTS Scope References Definitions Abbreviations . Conventions Summary of subjective testing procedures . Source recordings . 7.1 Recordin

8、g system 7.2 Speech samples . Selection of experiment parameters . 8.1 Codec conditions 8.2 Reference conditions Experiment design Listening test procedure . 10.1 Receiving system 10.2 Opinion scales . 10.3 Electrical noise Analysis of results Annex A . Comparison of different SNR definitions . Anne

9、x B . Sets of parameters to determine codec performance . Narrow-band systems (300-3400 Hz) . B.l B.2 Wideband systems (100-7000 Hz) Annex C - PCM codec corresponding to one qdu C.1 Common analogue interface . C.2 Digital interface C.3 Analogue interface Annex D - Modified IRS send and receive chara

10、cteristics . Bibliography Page 1 1 2 2 3 3 3 3 4 5 5 11 14 14 14 14 15 15 17 17 18 18 19 19 19 19 20 21 Recommendation P.830 (02/96) 1 COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T RECMNxP.830 96 = 4862593 0618036 729 = SUMMARY

11、This revised Recommendation describes methods and procedures for conducting subjective performance evaluations of digital speech codecs. Revisions encompassed by this version of the Recommendation are to include new information that reflects current practices in subjective evaluation of digital code

12、cs, including an expanded section on creating source recordings and addition of two annexes. One annex describes an implementation of a PCM codec (A-/p-law) that generates one quantization distortion unit (qdu) of distortion to input signals. The other new annex describes the Modified IRS transmit a

13、nd receive characteristics. These characteristics are recommended as the transmit and receive responses to be used in situations where the codec being tested is intended for use in fully digital circuits. INTRODUCTION Digital telephony in the Public Switched Telephone Network (PSTN) is now commonpla

14、ce. Since the 1960s. digital transmission in the PSTN has been synonymous with 64 kbit/s A-law or p-law PCM (see Recommendation G.711). The combination of better knowledge in signal processing techniques and advances in technology, most notably LSWLSI techniques, has lead to an increased interest in

15、 more efficient coding methods than PCM. Examples of this development include the introduction of 32 kbits ADPCM (Recommendation G.726), 16 kbits LDCELP (Recommendation G.728), and a 64 kbit/s, 7 kHz wideband codec (Recommendation G.722). Considerable interest is now focused on coding of speech at b

16、it rates below 16 kbit/s. Subjective testing is the most widely used method of assessing the performance of digital codecs. When the transmission path is digital and/or non-linear, simple objective measurements, such as those specified in Recommendation G.712, are insufficient to ensure adequate tra

17、nsmission performance. The aim of a subjective testing methodology is to measure the degradation contributed by the non-linear part of the transmission path, and hence to ensure that the performance of the complete system is satisfactory. To be suitable for this purpose, the measurements must be: a)

18、 reliable; and b) carried out in a way that takes account of major interactions between the non-linear part and the other parts of the transmission system. This implies both the ability to assign a unique numerical contribution to each digital process and the ability to use this assigned contributio

19、n in conjunction with other impairments to estimate telephone connection performance. KEYWORDS Subjective testing, subjective evaluation. 11 Recommendation P.830 (OZ96) COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T RECMN*P.830 9

20、6 4862593 ObL037 bb5 Recommendation P.830*) SUBJECTIVE PERFORMANCE ASSESSMENT OF TELEPHONE-BAND AND WIDEBAND DIGITAL CODECS (Helsinki, 1993; revised in 1996) 1 Scope This Recommendation is based on Annex BP.80 and on experience gained from several international experiments 5, 6, 9, lo, 12, 13, 15 an

21、d 17. Further, this Recommendation defines a specific testing method for evaluating digital processes. This method allows the subjective effects of distortion introduced by the speech coding process to be taken into account when assessing the performance of the evolving international network. This R

22、ecommendation contains advice on how to assess the performance of digital codecs. This Recommendation must be read in conjunction with Recommendation P.80. Additional information is available in sources listed in the Bibliography. A variety of methods are possible to characterize the performance of

23、digital processes. At the present time these comprise listening-only tests involving: 1) opinion (category) ratings; 2) 3) articulation tests. pair or multiple comparisons; and For most applications the -T recommends the use of the Absolute Category Rating (ACR) method using the Listening Quality sc

24、ale. However, there are times when other scales and rating methods are more suitable and appropriate (e.g. the Listening Effort scale has been found useful when interest centers on good correlation with measures of conversational performance), and these are used as well in this Recommendation. Only

25、where there is a deviation from the use of the ACR method using the Listening Quality scale will it be stated. NOTE - The Degradation Category Rating (DCR) method 8 is described in detail in Annex DP.80. This method is purported to be suited when the absolute quality of the source material is low (e

26、.g. speech corrupted by background noise) or when digitai impairments are small. It may therefore be particularly useful to sene for system optimization. It should also be noted that a threshold method for direct comparison, described in detail in Annex EP.80, is again applicable for system optimiza

27、tion. Furthermore, there may be a need for supplementing listening-only tests with conversation tests planned and conducted in accordance with Recommendation P.80. However, until there has been further consideration of the circumstances in which conversational tests are necessary, and further invest

28、igation is made regarding the choice of conditions that should be covered in such experiments, no detailed plans can be included in this Recommendation. 2 References The following Recommendations contain provisions which, through reference in this text, constitute provisions of this Recommendation.

29、At the time of publication, the editions indicated are valid. All Recommendations and other references are subject to revision; all users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations listed below. A list of

30、the currently valid ITU-T Recommendations is regularly published. IEC Publication 581.7 (1986). High fidelis, audio equipment and system; Minimum performance requirements - Part 7: Loudspeakers. ITU-T Recommendation G. 1 13 (1996), Transmission impainnenrs. ITU-T Recommendation G.191 (1993), Softwar

31、e tools for speech und audio coding standardization. - - - Previously P.83. Recommendation P.830 (02/96) 1 COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T RECMN*P.830 96 W 4862591 Ob18038 5Tl ITU-T Recommendation G. 192 (1996). A

32、common digital parallel interfoce for speech standardization activities. CCIT Recommendation G.711 (1988), Pulse Code Modulation (PCM of voicefrequencies. CCiTT Recommendation G.7 12 (1992). Transmission performunce characteristics of pulse code modulation. CCiTT Recommendation G.722 (1988), 7 Mz au

33、dio-coding within 64 kbids. CCIT Recommendation G.726 (1990), 40, 32, 24 and 16 bids Adaptive Differential Pulse Code Modulation (ADPCM). CC Recommendation G.728 (1992), Coding of speech at 16 kbids using low-delay code excited linear prediction. -T Recommendation 0.41 (1994), Psophometer for use on

34、 telephone-type circuits. CCiTT Recommendation O. 13 1 (1988). Quantizing distortion measuring equipment using a pseudo- random noise test signal. CCT Recommendation 0.132 (1988). Quantizing distortion measuring equipment using a sinusoidal test signal. ITU-T Recommendation O. 133 (1993). Equipment

35、for measuring the performance of PCM encoders und decoders. -T Recommendation P.10 (1993). Vocabulary of terms on telephone transmission quality und telephone sets. CCT Recommendation P.48 (1988), Specification for un intermediate reference system. ITU-T Recommendation P.50 (1993). Artificial voices

36、. -T Recommendation P.56 (1993). Objective measurement of active speech level. ITU-T Recommendation P.80 (1993), Methods for subjective determination of transmission quality. ITU-T Recommendation P.810 (1996), Modulated Noise Reference Unit (MNRU. ITU-T Recommendation P.84 (1993). Subjective listeni

37、ng test method for evaluating digital circuit multiplication and packetized voice systems. CCIT Recommendation 4.35 (1988), Technical characteristics of tones for the telephone service. 3 Definitions For the purposes of this Recommendation, the following definitions apply: 3.1 3.2 described in Recom

38、mendation P.810. dBov: dB relative to the overload point of a digital system. Q: The ratio, in dB, of speech power to modulated noise power in the Modulated Noise Reference Unit, as 3.3 3.4 QN: Q for a narrow-band Modulated Noise Reference Unit. Qw: Q for a wideband Modulated Noise Reference Unit. 4

39、 Abbreviations For the purposes of this Recommendation, the following abbreviations are used: ACR Absolute Category Rating ADPCM BER Bit Error Rate DCR Degradation Category Rating eif equipment impairment factor Adaptive Differential Pulse Code Modulation 2 Recommendation P.830 (OU96) COPYRIGHT Inte

40、rnational Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling Services ITU-T RECNNxP-830 76 486259L Ob38039 438 JRS ITU-T STL LDCELP MNRU Modulated Noise Reference Unit MOS Mean Opinion Score PCM Pulse Code Modulation PSTN Public Switched Telephone Network qdu quantizatio

41、n distortion unit RMS Root Mean Square SNR Signal-to-Noise Ratio Intermediate Reference System (Recommendation P.48) ITU-T Software Tools Library Low-Delay Code Excited Linear Prediction 5 Conventions Subjective evaluation of digital speech codecs may, in principle, be conducted using listening-only

42、 or conversational methods of subjective testing. As a practical matter, listening-only tests are the only feasible method of subjective testing during the development of speech codecs when a real-time implementation of the codec is not available. This Recommendation discusses listening-only methods

43、 of subjective evaluation of speech codecs. 6 Summary of subjective testing procedures When performing a subjective evaluation of a speech codec, consideration must be given to the intended application of the codec under test. The system used for recording speech materials, the selection of experime

44、ntal test conditions, and the system used for delivery of processed speech materials to the listeners must be chosen so that they capture the important end-to-end characteristics of the telephone connection. In particular, the characteristics of the recording and receiving systems should reflect the

45、 characteristics of the transmitter and receiver that will be used in the intended application of the codec. Performing subjective evaluations of digital codecs proceeds via a number of steps: 1) 2) 3) Design of the experiment; 4) 5) Analysis of results. Preparation of source speech materials, inclu

46、ding recording of talkers; Selection of experimental parameters to exercise the features of the codec that are of interest; Selection of a test procedure and conduct of the experiment; Each of these steps is described below. 7 Source recordings 7.1 Recording system As noted above, it is important to

47、 use a representative system for the transmitting end of the system under test. When testing narrow-band (300-34Hz) digital codecs for network applications, the ITU-T recommends the use of a modified Intermediate Reference System (IRS) transmit characteristic. Such a device is constructed by removin

48、g the SRAEN filter from the IRS transmitter. The transmit Characteristic of the modified IRS is found in Annex D. A software implementation of the modified IRS transmit characteristic is available in the ITU-T Software Tools Library (ITU-T Sn). NOTE - The ITU-T Software Tools Library contains a numb

49、er of tools that are useful for the development of speech and audio coding standards. These tools are also useful when preparing materials for use in subjective evaluations of speech codecs. The tools that are available in the -T STL are described in Recommendation G. 19 1. Recommendation P.830 (0296) 3 COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesIn some circumstances (e.g. when it is expected that low frequencies will be sharply limited), it will be necessary or de