1、 Rec. ITU-R BS.1387-1 1 RECOMMENDATION ITU-R BS.1387-1 Method for objective measurements of perceived audio quality (1998-2001) The ITU Radiocommunication Assembly, considering a) that conventional objective methods (e.g. for measuring signal-to-noise ratio and distortion) are no longer adequate for
2、 measuring the perceived audio quality of systems which use low bit-rate coding schemes or which employ analogue or digital signal processing; b) that low bit-rate coding schemes are rapidly being deployed; c) that not all implementations conforming to a specification or standard guarantee the best
3、quality achievable with that specification or standard; d) that formal subjective assessment methods are not suitable for continuous monitoring of audio quality, e.g. under operational conditions; e) that objective measurement of perceived audio quality may eventually complement or supersede convent
4、ional objective test methods in all areas of measurement; f) that objective measurement of perceived audio quality may usefully complement subjective assessment methods; g) that, for some applications, a method which can be implemented in real time is necessary, recommends 1 that for each applicatio
5、n listed in Annex 1 the method given in Annex 2 be used for objective measurement of perceived audio quality. Foreword This Recommendation specifies a method for objective measurement of the perceived audio quality of a device under test, e.g. a low bit-rate codec. It is divided into two Annexes. An
6、nex 1 gives the user a general overview of the method and includes four Appendices. Appendix 1 describes applications and test signals. Appendix 2 lists the Model Output Variables and discusses limitations of use and accuracy. Appendix 3 gives the outline of the model while Appendix 4 describes the
7、principles and characteristics of objective perceptual audio quality measurement methods in general. Annex 2 provides the implementer with a detailed description of the method using two versions of the psycho-acoustic model that were developed during the integration phase where six models were combi
8、ned. In Appendix 1 of Annex 2 the validation process of the objective measurement method is described. Appendix 2 of Annex 2 gives an overview of all the databases that were used in the development and validation of the method. 2 Rec. ITU-R BS.1387-1 TABLE OF CONTENTS Page Foreword 1 Table of conten
9、ts 2 Annex 1 Overview. 7 1 Introduction. 7 2 Applications 7 3 Versions 8 4 The subjective domain 8 5 Resolution and accuracy . 10 6 Requirements and limitations . 10 Appendix 1 to Annex 1 Applications 10 1 General 10 2 Main applications 11 2.1 Assessment of implementations 11 2.2 Perceptual quality
10、line up 11 2.3 On-line monitoring 11 2.4 Equipment or connection status 12 2.5 Codec identification 12 2.6 Codec development. 12 2.7 Network planning 12 2.8 Aid to subjective assessment. 13 2.9 Summary of applications 13 3 Test signals . 13 3.1 Selection of natural test signals. 14 3.2 Duration 15 4
11、 Synchronization 15 5 Copyright issues 15 Appendix 2 to Annex 1 Output variables 15 1 Introduction. 15 2 Model Output Variables 15 3 Basic Audio Quality 16 4 Coding Margin 17 5 User requirements . 17 Rec. ITU-R BS.1387-1 3 Page Appendix 3 to Annex 1 Model outline 17 1 Audio processing 18 1.1 User-de
12、fined settings. 18 1.2 Psycho-acoustic model 18 1.3 Cognitive model 19 Appendix 4 to Annex 1 Principles and characteristics of objective perceptual audio quality measurement methods. 20 1 Introduction and history 20 2 General structure of objective perceptual audio quality measurement methods 21 3 P
13、sycho-acoustical and cognitive basics 21 3.1 Outer and middle ear transfer characteristic . 22 3.2 Perceptual frequency scales 22 3.3 Excitation 23 3.4 Detection . 24 3.5 Masking. 25 3.6 Loudness and partial masking. 25 3.7 Sharpness 25 3.8 Cognitive Processing. 26 4 Models incorporated . 27 4.1 DIX
14、 27 4.2 NMR 28 4.3 OASE 28 4.4 Perceptual Audio Quality Measure (PAQM) 29 4.5 PERCEVAL 30 4.6 POM 30 4.7 The Toolbox Approach . 31 Annex 2 Description of the Model 32 1 Outline 32 1.1 Basic Version 33 1.2 Advanced Version. 33 2 Peripheral Ear Model 34 2.1 FFT-based Ear Model . 34 2.1.1 Overview 34 2
15、.1.2 Time Processing . 35 2.1.3 FFT . 35 2.1.4 Outer and middle ear 36 4 Rec. ITU-R BS.1387-1 Page 2.1.5 Grouping into critical bands . 36 2.1.6 Adding internal noise . 43 2.1.7 Spreading 43 2.1.8 Time domain spreading 45 2.1.9 Masking Threshold. 45 2.2 Filter bank-based ear model 46 2.2.1 Overview
16、46 2.2.2 Subsampling . 47 2.2.3 Setting of Playback Level. 47 2.2.4 DC-rejection-filter 47 2.2.5 Filter Bank 48 2.2.6 Outer and middle ear filtering 50 2.2.7 Frequency domain spreading 51 2.2.8 Rectification . 53 2.2.9 Time domain smearing (1) Backward masking 53 2.2.10 Adding of internal noise . 53
17、 2.2.11 Time domain smearing (2) Forward masking . 53 3 Pre-processing of excitation patterns 54 3.1 Level and pattern adaptation . 54 3.1.1 Level adaptation . 54 3.1.2 Pattern adaptation . 55 3.2 Modulation 56 3.3 Loudness . 56 3.4 Calculation of the error signal. 57 4 Calculation of Model Output V
18、ariables 57 4.1 Overview. 57 4.2 Modulation difference. 58 4.2.1 RmsModDiffA58 4.2.2 WinModDiff1B. 59 4.2.3 AvgModDiff1Band AvgModDiff2B59 4.3 Noise Loudness . 59 4.3.1 RmsNoiseLoudA. 60 4.3.2 RmsMissingComponentsA. 60 4.3.3 RmsNoiseLoudAsymA. 60 4.3.4 AvgLinDistA. 60 4.3.5 RmsNoiseLoudB. 60 4.4 Ban
19、dwidth . 60 4.4.1 Pseudocode. 61 4.4.2 BandwidthRefBand BandwidthTestB61 Rec. ITU-R BS.1387-1 5 Page 4.5 Noise-to-mask ratio. 62 4.5.1 Total NMRB. 62 4.5.2 Segmental NMRB. 62 4.6 Relative Disturbed FramesB62 4.7 Detection Probability 62 4.7.1 Maximum filtered probability of detection (MFPDB) . 64 4.
20、7.2 Average distorted block (ADBB) . 64 4.8 Harmonic structure of error. 64 4.8.1 EHSB65 5 Averaging 65 5.1 Spectral averaging. 65 5.1.1 Linear average 65 5.2 Temporal averaging 65 5.2.1 Linear average 66 5.2.2 Squared average . 66 5.2.3 Windowed average . 66 5.2.4 Frame selection. 67 5.3 Averaging
21、over audio channels. 67 6 Estimation of the perceived basic audio quality . 67 6.1 Artificial neural network. 68 6.2 Basic Version 68 6.3 Advanced Version. 70 7 Conformance of Implementations. 71 7.1 General 71 7.2 Selection 71 7.3 Settings for the conformance test 71 7.4 Acceptable tolerance interv
22、al 72 7.5 Test items 72 Appendix 1 to Annex 2 Validation process. 73 1 General 73 2 Competitive phase. 74 3 Collaborative phase. 75 4 Verification . 75 4.1 Comparison of SDG and ODG values 76 4.2 Correlation 76 4.3 Absolute Error Score (AES) . 79 4.4 Comparison of ODG versus the confidence interval
23、80 4.5 Comparison of ODG versus the tolerance interval . 84 6 Rec. ITU-R BS.1387-1 Page 5 Selection of the optimal model versions. 86 5.1 Pre-selection criteria based on correlation 86 5.2 Analysis of number of outliers 87 5.3 Analysis of severeness of outliers. 88 6 Conclusion 89 Appendix 2 to Anne
24、x 2 Descriptions of the reference databases 89 1 Introduction. 89 2 Items per database. 91 3 Experimental conditions . 91 3.1 MPEG90 92 3.2 MPEG91 92 3.3 ITU92DI 92 3.4 ITU92CO 92 3.5 ITU93 92 3.6 MPEG95 93 3.7 EIA95 93 3.8 DB2 . 93 3.9 DB3 . 93 3.10 CRC97. 94 4 Items per condition for DB2 and DB3 9
25、4 4.1 DB2 . 94 4.2 DB3 . 97 Glossary 97 Abbreviations . 98 References 99 Bibliography. 100 Rec. ITU-R BS.1387-1 7 ANNEX 1 Overview 1 Introduction Audio quality is one of the key factors when designing a digital system for broadcasting. The rapid introduction of various bit-rate reduction schemes has
26、 led to significant efforts in establishing and refining procedures for subjective assessments, simply because formal listening tests have been the only relevant method for judging audio quality. The experience gained was the foundation for Recommendation ITU-R BS.1116, which then became the basis f
27、or most listening tests of this type. Since subjective quality assessments are both time consuming and expensive, it is desirable to develop an objective measurement method in order to produce an estimate of the audio quality. Traditional objective measurement methods, like Signal-to-Noise-Ratio (S/
28、N) or Total-Harmonic-Distortion (THD) have never really been shown to relate reliably to the perceived audio quality. The problems become even more evident when the methods are applied on modern codecs which are both non-linear and non-stationary. A number of methods for making objective perceptual
29、measurements of perceived audio quality have been introduced during the last decade. But none of the methods were thoroughly validated, and consequently neither standardized nor widely accepted. In 1994, ITU-R identified an urgent need to establish a standard in this area and the work was initiated.
30、 An open call for proposals was issued and the following six candidates for measurement methods were received: Disturbance Index (DIX), Noise-to-Mask Ratio (NMR), Perceptual Audio Quality Measure (PAQM), Perceptual Evaluation (PERCEVAL), Perceptual Objective Measure (POM) and The Toolbox Approach. T
31、he methods are described in Appendix 4 to Annex 1. The measurement method in this Recommendation is the result of a process where the performance of each of the above six methods was studied, and the most promising tools extracted and integrated into one single method. The recommended method has bee
32、n carefully validated at a number of test sites. It has proven to generate both reliable and useful information for several applications. One must, however, keep in mind that the objective measurement method in this Recommendation is not generally a substitute for arranging a formal listening test.
33、2 Applications The basic concept for making objective measurements with the recommended method is illustrated in Fig. 1 below. 1387-01Device under testSignal under testReference signalObjectivemeasurementmethodAudio quality estimateFIGURE 1Basic concept for making objective measurements8 Rec. ITU-R
34、BS.1387-1 The measurement method in this Recommendation is applicable to most types of audio signal processing equipment, both digital and analogue. It is, however, expected that many applications will focus on audio codecs. The following 8 classes of applications have been identified: TABLE 1 Appli
35、cations 3 Versions In order to achieve an optimal fit to different cost and performance requirements, the objective measurement method recommended in this Recommendation has two versions. The Basic Version is designed to allow for a cost-efficient real-time implementation, whereas the Advanced Versi
36、on has a focus on achieving the highest possible accuracy. Depending on the implementation, this additional accuracy increases the complexity approximately by a factor of four compared to the Basic Version. Table 1 gives some guidance on which version to apply for each of the applications. 4 The sub
37、jective domain Formal subjective listening tests, e.g. those based on Recommendation ITU-R BS.1116, are carefully designed to come as close as possible to a reliable estimate of the judgement of the audio quality. One could, however, not expect the result from a subjective listening test to fully re
38、flect the actual perception. Figure 2 illustrates the imperfections implicit in both the subjective and the objective domain. It is obviously not possible to validate an objective method directly. Instead, objective measurement methods are validated against subjective listening tests. Application Br
39、ief description Version 1 Assessment of implementations A procedure to characterize different implementations of audio processing equipment, in many cases audio codecs Basic/Advanced 2 Perceptual quality line up A fast procedure which takes place prior to taking a piece of equipment or a circuit int
40、o service Basic 3 On-line monitoring A continuous process to monitor an audio transmission in service Basic 4 Equipment or connection status A detailed analysis of a piece of equipment or a circuit Advanced 5 Codec identification A procedure to identify the type and implementation of a particular co
41、dec Advanced 6 Codec development A procedure which characterizes the performance of the codec in as much detail as possible Basic/Advanced 7 Network planning A procedure to optimize the cost and performance of a transmission network under given constraints Basic/Advanced 8 Aid to subjective assessme
42、nt A tool for screening critical material to include in a listening test Basic/Advanced Rec. ITU-R BS.1387-1 9 1387-02FIGURE 2Validation conceptsTheactual perceptionObjectivemeasurementsSubjectiveassessmentsThe objective measurement method in this Recommendation has been focused on applications whic
43、h are normally assessed in the subjective domain by applying Recommendation ITU-R BS.1116. The basic principle of that particular test method can be briefly described as follows: the listener can select between three sources (A, B and C). The known Reference Signal is always available as source A. T
44、he hidden Reference Signal and the Signal Under Test are simultaneously available but are randomly assigned to B and C, depending on the trial. The listener is asked to assess the impairments on B compared to A, and C compared to A, according to the continuous five-grade impairment scale. One of the
45、 sources, B or C, should be indiscernible from source A; the other one may reveal impairments. Any perceived differences between the reference and the other source must be interpreted as an impairment. Normally, only one attribute, Basic Audio Quality, is used. It is defined as a global attribute th
46、at includes any and, all detected differences between the reference and the Signal Under Test. The grading scale shall be treated as continuous with anchors derived from the ITU-R five-grade impairment scale given in Recommendation ITU-R BS.562 as shown below. 1387-03FIGURE 3The ITU-R five-grade imp
47、airment scaleImperceptiblePerceptible but not annoyingSlightly annoyingAnnoyingVery annoying5.04.03.02.01.0The analysis of the results from a subjective listening test is in general based on the Subjective Difference Grade (SDG) defined as: SDG = GradeSignal Under Test GradeReference SignalThe SDG v
48、alues should ideally range from 0 to 4, where 0 corresponds to an imperceptible impairment and 4 to an impairment judged as very annoying. 10 Rec. ITU-R BS.1387-1 5 Resolution and accuracy The Objective Difference Grade (ODG) is the output variable from the objective measurement method and correspon
49、ds to the SDG in the subjective domain. The resolution of the ODG is limited to one decimal. One should however be cautious and not generally expect that a difference between any pair of ODGs of a tenth of a grade is significant. The same remark is valid when looking at results from a subjective listening test. There is no single figure which fully describes the accuracy of the objective measurement method. Instead, one has to consider a number of different figures of merit. One of them is the correlation between SDGs and ODGs. It is impor
