1、 INTERNATIONAL TELECOMMUNICATION UNION ITU-T P.835TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (11/2003) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Methods for objective and subjective assessment of quality Subjective test methodology for evaluating spe
2、ech communication systems that include noise suppression algorithm ITU-T Recommendation P.835 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Seri
3、es P.10 Subscribers lines and sets Series P.30 P.300 Transmission standards Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of quality
4、 Series P.80 P.800 Audiovisual quality in multimedia services Series P.900 For further details, please refer to the list of ITU-T Recommendations. ITU-T Rec. P.835 (11/2003) i ITU-T Recommendation P.835 Subjective test methodology for evaluating speech communication systems that include noise suppre
5、ssion algorithm Summary This Recommendation describes a methodology for evaluating the subjective quality of speech in noise and is particularly appropriate for the evaluation of noise suppression algorithms. The methodology uses separate rating scales to independently estimate the subjective qualit
6、y of the Speech Signal alone, the Background Noise alone, and Overall Quality. Source ITU-T Recommendation P.835 was approved on 13 November 2003 by ITU-T Study Group 12 (2001-2004) under the ITU-T Recommendation A.8 procedure. Keywords Coded speech in background noise, noise preprocessor, noise sup
7、pression algorithm, speech quality evaluation, subjective testing. ii ITU-T Rec. P.835 (11/2003) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications. The ITU Telecommunication Standardization Sector (ITU-T) is a permane
8、nt organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishe
9、s the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standard
10、s are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recomme
11、ndation may contain certain mandatory provisions (to ensure e.g. interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are u
12、sed to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intell
13、ectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not receive
14、d notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementors are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database. ITU 2004 All rights reserved. N
15、o part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. ITU-T Rec. P.835 (11/2003) iii CONTENTS Page 1 Scope 1 2 References. 1 3 Definitions 2 4 Abbreviations 2 5 Experimental design . 2 5.1 Speech material 2 5.2 Listening session 5 5.3 Dat
16、a analysis. 6 5.4 Presentation and interpretation of results . 7 Appendix I Procedure for proper mixing of speech and noise samples 7 I.1 General . 7 I.2 Parameters 7 I.3 Speech and background noise files. 8 I.4 Speech and noise input filters. 8 I.5 P.56 speech level adjustment 8 I.6 Basic noise lev
17、el adjustment 8 Appendix II Example of Instructions to subjects 9 ITU-T Rec. P.835 (11/2003) 1 ITU-T Recommendation P.835 Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm 1 Scope Typically, Noise Suppression Algorithms (NSA) operate on
18、a noisy speech waveform and attempt to reduce the noise or background component without adversely affecting the speech or signal component of the waveform. This goal can often be realized for relatively low levels of noise suppression. For higher levels of noise suppression, however, NSAs often adve
19、rsely affect the speech component as more noise is suppressed: there tends to be increasing degradation of the speech or signal component as more of the noise or background component is removed. In this situation, subjects can often become confused as to what they should be responding to in their ra
20、tings of the overall “quality“ of the waveforms: while the background may have been improved because there is less noise present in the waveform, the speech signal may have been degraded in the process. In a single-scale rating method, the ACR, for example, each individual subject weights the signal
21、 and the background components in determining his ratings of overall speech quality. This weighting process introduces additional error variance in the subjects ratings of overall quality resulting in decreased reliability in those ratings. The methodology described in this Recommendation reduces th
22、e listeners uncertainty by requiring him to successively attend to and rate the waveform on: the speech signal, the background noise, and the overall effect: speech + background. While this methodology has been shown to be reliable and valid for evaluating NSAs, it should not be restricted to testin
23、g NSA. The methodology can be used for the more general case of evaluating conditions of speech in background noise. It is particularly applicable in those cases where it is unknown whether a system includes a noise preprocessor. 2 References The following ITU-T Recommendations and other references
24、contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation are therefore encouraged to investigat
25、e the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it, as a stand-alone document, the status o
26、f a Recommendation. ITU-T Recommendation G.191 (2000), Software tools for speech and audio coding standardization. ITU-T Recommendation P.56 (1993), Objective measurement of active speech level. ITU-T Recommendation P.800 (1996), Methods for subjective determination of transmission quality. ITU-T Re
27、commendation P.810 (1996), Modulated noise reference unit (MNRU). ITU-T Recommendation P.830 (1996), Subjective performance assessment of telephone-band and wideband digital codecs. 2 ITU-T Rec. P.835 (11/2003) 3 Definitions This Recommendation defines the following term: 3.1 dBov: dB relative to ov
28、erload. 4 Abbreviations This Recommendation uses the following abbreviations: ACR Absolute Category Rating ANOVA ANalysis Of VAriance D/A Digital-to-Analogue MANOVA Multiple ANalysis Of VAriance MOS Mean Opinion Score NSA Noise Suppression Algorithm RMS Root Mean Square SNR Signal-to-Noise Ratio SPL
29、 Sound Pressure Level 5 Experimental design 5.1 Speech material 5.1.1 Source speech material The source speech material should be meaningful sentences representative of the language under test and including multiple speech samples for both male and female talkers. 5.1.2 Processing Standard laborator
30、y procedures shall be followed to ensure that the processed speech and noise type samples are mixed and filtered properly (see ITU-T Rec. G.191 (Software Tool library) and Appendix I). 5.1.3 Reference conditions The reference conditions shall be selected to independently vary the signal and backgrou
31、nd ratings through their entire range of scale values. For example, speech in background noise should be varied along two dimensions, Speech-to-Noise Ratio (SNR) for varying the background ratings and MNRU for varying the signal ratings. Figure 1 illustrates the relative independence of the signal s
32、core and the correlation of the overall score to the background score when MNRU is varied while keeping SNR constant. Figure 2 illustrates the relative independence of the background score and the correlation of the overall score to the signal score when SNR is varied while keeping MNRU constant. Fi
33、gure 3 shows that the introduction of these combined reference conditions provide a full context within this two-dimensional perceptual space (signal/background). ITU-T Rec. P.835 (11/2003) 3 P.835_F01SNR = 40 dB, MNRU varies12345816243240MNRU (dBQ)RatingSignalBackgroundOverallFigure 1/P.835 Referen
34、ce condition: SNR constant, MNRU varies P.835_F02MNRU = 40 dB, SNR varies12345010203040SNR (dB)RatingSignalBackgroundOverallFigure 2/P.835 Reference condition: MNRU constant, SNR varies 4 ITU-T Rec. P.835 (11/2003) P.835_F03SNR and MNRU vary1234510/16SNR (dB) / MNRU (dBQ)RatingSignalBackgroundOveral
35、l20/24 30/32 40/40Figure 3/P.835 Reference condition: SNR and MNRU vary 5.1.4 Speech sample presentation Each trial contains a three-sentence sample of speech laid out in the general format illustrated in the example given in Figure 4. Each sample is comprised of three sub-samples, where each sub-sa
36、mple is followed by a silent voting period. In the example shown in the figure, each sub-sample is approximately 4 s in duration including 1 s of background noise alone, 2 s of speech + noise, 1 s of background, and an appropriate silent voting interval. In practice, the sub-samples should be as lon
37、g as necessary for the subjects to make reliable ratings. For the first two sub-samples, listeners rate either the signal or the background depending on the rating scale order specified for that trial. For the signal, subjects are instructed to attend only to the speech signal and rate the speech on
38、 the five-category distortion scale shown in Figure 5. For the background, subjects are instructed to attend only to the background and rate the background on the five-category intrusiveness scale shown in Figure 6. For the third sub-sample in each trial, subjects are instructed to listen to the spe
39、ech + background and rate it on the five-category overall quality scale shown in Figure 7, the Mean Opinion Score (MOS) used with the ACR. To control for the effects of rating scale order, the order of the rating scales shall be balanced across the experiment, i.e., scale order should be “Signal, Ba
40、ckground, Overall Effect“ for half of the trials, and “Background, Signal, Overall Effect“ for the other half. Furthermore, rating scale order should be counter-balanced across listening panels. P.835_F04Sentence 1signalratingSentence 2backgroundratingSentence 3overallquality-Sub-sample 1- Vot.Int.
41、-Sub-sample 2- Vot.Int. -Sub-sample 3- Vot.Int.Figure 4/P.835 Example of the timing of the speech materials in a P.835 trial NOTE 1 Experiments have shown that the sequence duration may be 4 s or 8 s, without influencing the results. The use of the shorter duration reduces the overall test duration.
42、 NOTE 2 Experiments have shown that sentences 1, 2 and 3 in Figure 4 may be the same in a complete sequence or may be different. This factor does not influence the results. ITU-T Rec. P.835 (11/2003) 5 5.2 Listening session 5.2.1 Listeners At least 32 nave listeners shall participate in the tests. A
43、ll the listeners shall be native speakers of the language used for the test and no listener shall have participated in a subjective experiment in the previous three months. 5.2.2 Audio presentation Audio presentation shall comply with the guidelines given in ITU-T Rec. P.800. These guidelines includ
44、e the listening system, listening levels, test duration and listening environment. 5.2.3 Instructions and rating scales Listeners shall receive written instruction in the rating tasks to be performed in the methodology. The instructions are provided in text form to avoid ambiguity and differences ac
45、ross experiments and across listening panels within an experiment. The instructions should show examples of the three rating scales involved in the methodology. Examples of the three rating scales in English are shown in Figure 5 for the Speech Signal rating, Figure 6 for the Background Noise rating
46、, and Figure 7 for the Overall Quality rating. The rating scales and category descriptors in languages other than English should provide a close translation of those shown in the example figures. Session 1 Block 1 Trial 1 Attending ONLY to the SPEECH SIGNAL, select the category which best describes
47、the sample you just heard. the SPEECH SIGNAL in this sample was 5 NOT DISTORTED 4 SLIGHTLY DISTORTED 3 - SOMEWHAT DISTORTED 2 FAIRLY DISTORTED 1 VERY DISTORTED Figure 5/P.835 Speech signal rating scale 6 ITU-T Rec. P.835 (11/2003) Session 1 Block 1 Trial 1 Attending ONLY to the BACKGROUND, select th
48、e category which best describes the sample you just heard. the BACKGROUND in this sample was 5 NOT NOTICEABLE 4 SLIGHTLY NOTICEABLE 3 NOTICEABLE BUT NOT INTRUSIVE 2 SOMEWHAT INTRUSIVE 1 VERY INTRUSIVE Figure 6/P.835 Background noise rating scale Select the category which best describes the sample yo
49、u just heard for purposes of everyday speech communication. the OVERALL SPEECH SAMPLE was 5 EXCELLENT 4 GOOD 3 - FAIR 2 POOR 1 BAD Figure 7/P.835 Overall quality rating scale (same as the MOS rating scale) used in the ACR procedure (see ITU-T Rec. P.800) An example of an Instructions sheet is given in Appendix II, in the case of the order “Signal, Background noise, Overall Quality“. It shall be adapted in the case of the order “Background noise, Signal, Overall Quality“. 5.2.4 Vot
