1、 Rec. 813 1 RECOMMENDATION 813 METHODS FOR OBJECTIVE PICTURE QUALITY ASSESSMENT IN RELATION TO IMPAIRMENTS FROM DIGITAL CODING OF TELEVISION SIGNALS (Question 64/11) (1992) Rec. 813 The CCIR, considering a) that with the increasing application of digital coding and bit-rate reduced transmission, the
2、 assessment of coding impairments is of critical importance; b) that objective measurements are necessary for specific evaluations as routine supervisions or system optimizations; c) that several types of objective measurements may be defined for different digital systems and applications; d) that t
3、he adoption of standardized methods is of importance in the exchange of information between various laboratories, recommends 1. that the general methods described in Annex 1 for obtaining objective measurements of digital picture quality should be used; 2. that the choice of the appropriate objectiv
4、e measurements for a system or an application should be made on the basis of the information given in Annex 1. ANNEX 1 1. Introduction With the increasing application of digital coding and bit-rate reduced transmission, the assessment of coding impairments is of critical importance. An understanding
5、 of these assessment methods is relevant not only to the performance of new coding equipment, but also to an interpretation of measurements made on such equipments and to specifications for target performance. Moreover digital codecs as with all adaptive or non-linear digital processes, cannot be fu
6、lly characterized with traditional television test signals or patterns. Quality of codecs for distribution application can be measured objectively, quality specifications being expressed against the subjective judgement of observers. Studies indicate the desirability of establishing relationships be
7、tween objective measurements of signals impaired by digital coding, and the visual quality of the picture thus obtained. This Annex gives progress towards this end, which is proving more difficult to achieve as codec complexity increases. The quality of a codec designed for contribution applications
8、 however, could in theory be specified in terms of objective performance parameters because its output is destined not for immediate viewing, but for studio post-processing, storing and/or coding for further transmission. Because of the difficulty of defining this performance for a variety of post-p
9、rocessing operations, the approach preferred has been to specify the performance of a chain of 2 Rec. 813 equipment, including a post-processing function, which is thought to be representative of a practical contribution application. This chain might typically consist of a codec, followed by a studi
10、o post-processing function (or another codec in the case of basic contribution quality assessment), followed by yet another codec before the signal reaches the observer. Adoption of this strategy for the specification of codecs for contribution applications means that the measurement procedures give
11、n in this Annex can also be used to assess them. 2. Digital codec classification The function of digital coding is to reduce the bit rate needed to represent a sequence of images while ensuring minimal loss in picture quality. Coding equipment does this, first by removing as much statistical redunda
12、ncy from the images as possible (i.e. no loss in quality occurs as a result of this conceptual first stage). Then, if more bit-rate reduction is necessary, some distortion has to be introduced into the picture, although one of the objectives of codec design is to hide this distortion by exploiting c
13、ertain perceptual insensitivities of the human visual system. It is convenient to divide codecs into two classes, those using fixed word-length coding and those using variable word-length coding (see definitions in 3.1 and 3.2 respectively). The latter class is more efficient and complex, and includ
14、es all recently proposed systems for coding 4:2:2 video to the range 30-45 Mbit/s. The former class is however sufficient to permit 4:2:2 video to be reduced to 140 Mbit/s while still preserving the quality demanded for contribution applications. A further sub-division of these classes is also usefu
15、l, into intrafield (or spatial) codecs and interframe (including interfield) codecs, which contain frame (or field) stores permitting them to exploit the redundancy which exists between successive picture frames (or fields). There is emerging a third class of codec which employs variable word-length
16、 coding but which is being designed for variable bit-rate networks. These codecs can in principle, preserve a constant decoded image quality subject to the bounds of peak network demand. The quality-testing of such codecs would have to take into account the nature of the network used and the statist
17、ics of the data injected by all of its users, and remains to be studied. 3. Objective assessments of codecs in terms of perceived picture impairments 3.1 Fixed word-length codecs With fixed word-length codecs a fixed number of bits is used to represent a fixed number of source picture samples. For e
18、xample in fixed word-length PCM or DPCM codecs, a fixed number of bits is allocated to each picture sample, and in fixed word-length transform or vector quantization codecs, a fixed number of bits is allocated to each block of picture samples. 3.1.1 Methods based on the use of synthetic test signals
19、 In these codecs the impairment introduced into each received picture sample of an image is dependent upon the values of those samples in the locality surrounding it, either in the same field (for an intrafield codec) or in the same and previous fields (for an interframe codec). It is therefore poss
20、ible, using suitably chosen two or three dimensional digital test signals, to artificially provoke the degradations characteristic of digital image coding. Some of these degradation factors have acquired names such as false contouring, granular noise, blur, blocking impairments, etc., relating to th
21、eir interpretation by observers. Having provoked these distortions, their magnitudes can be objectively measured and, using experience gained from subjective assessments these measurements could then be related to some quantification of codec quality. Relating the degradation factors to their interp
22、retation by observers may prove difficult in interframe coding systems or systems employing some adaptive processing because they can vary at any moment, with motion or adaptation of the coding algorithm. In a proposed method, the subjective assessment test first uses scales derived from pairs of op
23、posite adjectives (the semantic differential method), and then the results are analysed by principal component analysis to extract the picture quality degradation factors. The classification results can be tested by applying multiple regression analysis which relates the factors to subjective judgem
24、ents. A list of picture quality degradation factors is presented in Table 1. Rec. 813 3 TABLE 1 Examples of picture quality degradation factors for digital system and corresponding physical measures (units) While these methods appear to have conveniences for codec assessment and also to offer a tool
25、 to the codec designer, they are difficult to relate to the performance of a codec for real pictures for the following reasons: the complex composition of real picture sequences cannot be satisfactorily modelled by a practical number of synthetic test signals; degradations can be numerous in charact
26、er and difficult to classify because of their subtle nature (for example, a particular distortion may be visible only in textured parts of an image moving in a particular way); meaningful objective measurements of degradations can be difficult to define (for example, for motion portrayal). It should
27、 be noted that the duration of the period in which objective measures are taken should correspond to the observation window provided by the duration of the presentation in subjective tests. 3.1.2 Methods based on natural picture material and coding error Natural picture sequences can be thought of a
28、s being composed of a number of different regions, each with different local content and each exercising different fixed word-length codecs in different ways. Therefore the content of an image sequence will have a significant impact on the quality perceived by a viewer. It is also possible, where a
29、comparison is to be made between two codecs, for the image sequence content to determine which appears the better. Not only does this underline the importance of the choice of test images for subjective assessments (see Recommendation 500) but also that an objective measure of the performance of a p
30、articular codec must consider image content, if there is to be a correlation between subjective and objective assessment results. The most common forms of objective quality measurement are based on the coding error of a codec; that is, the difference between an input picture sequence and its decoded
31、 output. This difference signal (often amplified) can itself be displayed as an image sequence and this can provide a useful development aid to the codec specialist. It should not however be used as material for subjective assessments. 3.1.3 Methods based on normalized mean square error A frequently
32、 used objective measure of decoded image quality is mean square coding error. This is the average, over every picture sample in a sequence, of the square of the coding error and is usually normalized with respect to (the square of) the full amplitude range of the picture samples. Sometimes the norma
33、lized mean square error (NMSE) is quoted as a coding noise figure evaluated as 10 log (NMSE). The popularity of the NMSE measure stems from its Picture quality degradation factor Physical measure Image blur Step response rise time Edge busyness Step response jitter width False contouring Sp-pto mini
34、mum quantizing p-p Granular noise Equivalent analogue signal-to-noise ratio expressed in terms of Sp-p / N r.m.s.“Dirty window” effect Maximum noise amplitude Movement blur Rise time of a moving edge Jerkiness Field or frame difference in terms of moving edge position 4 Rec. 813 mathematical conveni
35、ence but it must be regarded with caution as a measure of decoded quality. It cannot distinguish, for example, between a few large coding errors (which may be annoying to an observer) and a large number of small coding errors (which may be imperceptible). Weighting of the coding error signal (perfor
36、med after a log operation) prior to the NMSE evaluation, with a filter derived from a visual model, has been attempted and has achieved improved correlation with subjective assessment results. The NMSE is a useful practical tool in codec development where it is often required to compare coding metho
37、ds which are very similar (i.e. those which use minor variants of the same algorithm and where impairment processes can be assumed to be identical). 3.1.4 Methods based on visual models The sensitivity of the human visual system to coding error in a particular region of an image is strongly influenc
38、ed by the characteristics of the image material itself in that region. The inability to recognize this fact is the major failing of the NMSE measure. To give just one example of this influence: it is known that an observers sensitivity to coding error noise is reduced when the spectrum of that noise
39、 approximately coincides with the spectrum of the “background” image. These properties of the visual system are those which are being exploited in codec design when subjective experiments or psychovisual data are used to optimize system parameters. In order to further the correlation between objecti
40、ve measures of picture quality and that judged by human observers it is necessary to develop a visual model which can interpret local coding error in the context of the background image and which can combine all these local assessments to form a global quality rating. This approach is applicable to
41、both fixed and variable word-length codecs and is considered in 3.2.3. 3.2 Variable word-length codecs Television codecs which require to reduce their source image data by more than a factor of about two, use methods based upon variable word-length coding. These codecs have increased efficiency beca
42、use they possess the flexibility to allocate dynamically coding bits to the parts of an image sequence where they are most effective in maintaining decoded image quality. There are several ways in which codecs can do this; the use of variable length entropy codes is not necessarily implied. 3.2.1 Me
43、thods based on the use of synthetic test signals Because of the flexibility of these codecs, the impairment which they introduce into each coded sample is dependent not only on the values of samples in the same locality, but also on the history of previous samples extending a frame or more into the
44、past. This means that for either intrafield or interframe variable word-length codecs it is not meaningful to attempt codec characterization by trying to provoke local distortions with local test signals and making objective measurements on them. If, however, the adaptation modes of a variable word-
45、length codec can be artificially held (requiring access to its internal workings), each mode may be characterized separately. Knowledge of the codecs adaptation switching, when it is presented with natural scenes, could then be used to objectively determine its performance. It is possible to contriv
46、e moving synthetic test sequences which take a codec to the point where it produces visible distortion, but even if objective measurements could be defined to characterize these distortions (see reservations in 3.1.1), their interpretation could only be made in the context of that entire test sequen
47、ce. This raises questions about how typical of natural scenes it is, and whether a codec designer would have the opportunity to optimize its performance to suit known test material. 3.2.2 Methods based on natural picture material and coding error It is important in any assessment of variable word-le
48、ngth codecs that natural picture sequences be used. Bearing in mind the ability of these codecs to direct the utilization of coding bits throughout the image, careful consideration should be given to the content of every part of the image sequence when judging its criticality (see Recommendation 500
49、). It is recommended that any objective assessments be based on the coding error of a codec where the inputs are a number of natural test pictures. The normalized mean square error method discussed in 3.1.3 may also be applied to the coding error from variable word-length codecs but such results should be for specialist interpretation only and even then, only as a supplement to subjective assessments. Similarly objective comparisons between codecs based on the NMSE should only be undertaken by specialists in codec design and only where techniques to be compared have very minor
