1、 Rec. ITU-R BT.1676 1 RECOMMENDATION ITU-R BT.1676 Methodological framework for specifying accuracy and cross-calibration of video quality metrics (Question ITU-R 44/6) (2004) The ITU Radiocommunication Assembly, considering a) that digital TV and HDTV utilizing bit-rate reduction technologies such
2、as MPEG-2, DV and others have achieved widespread use; b) that the Radiocommunication Sector is responsible for setting the overall quality perfor-mance of broadcasting chains; c) that impairments to television pictures can be shown to correlate with measurable features of the signals; d) that overa
3、ll picture quality is related to the combination of all impairments; e) that in the case of digital TV it is necessary, in particular, to assess the performance of bit-rate reduction methods both in terms of subjective and objective parameters; f) that for television systems a number of objective pi
4、cture quality parameters as well as associated performance measurement and monitoring methods have been developed for the studio environment and in broadcasting; g) that full reference objective picture quality measurement methods are useful in evaluating studio and broadcasting systems; h) that dat
5、a sets of test materials, subjective scores and objective values are used in the validation testing of objective picture quality measurement methods; j) that there are a number of proposed full reference video quality metrics (VQMs) that can be used to provide objective picture quality ratings; k) t
6、hat there are a number of well-known statistical evaluation methods documented in the literature that can be used to validate and compare VQMs based on data sets of test materials, subjective scores and objective values; l) that when one or more VQMs are accepted as normative in ITU Recommendations
7、there will still be a need to estimate the mathematical accuracy (resolving power) of the VQM being used; m) that cross-calibration of full reference objective picture quality measurement methods based on available data sets is important for international exchange of measurement and monitoring resul
8、ts, 2 Rec. ITU-R BT.1676 recommends 1 that the calculations specified in Annex 1 be used to estimate the accuracy and cross-calibration of objective picture quality measurements utilizing the full reference method; 2 that the calculations specified in Annex 1 may be used as one of several methods to
9、 deter-mine the accuracy in evaluation and validation of various objective picture quality measurements utilizing the full reference method. Annex 1 Method for specifying accuracy and cross-calibration of VQMs 1 Scope VQMs are intended to provide calculated values that are strongly correlated with v
10、iewer subjective assessments. This Recommendation provides: Methods for curve fitting VQM objective values to subjective data in order to facilitate the accuracy calculation and to produce a normalized objective value scale that can be used for cross correlation between different VQMs. An algorithm
11、(based on statistical analysis relative to subjective data) to quantify the accuracy of a given VQM. A simplified root mean square error calculation to quantify the accuracy of a VQM when the subjective data has roughly equal variance across the VQM scale. A method to plot classification errors to d
12、etermine the relative frequencies of “false tie”, “false differentiation”, “false ranking”, and “correct decision” for a given VQM. The methods specified in this Recommendation are based on objective and subjective evaluation of component video such as defined by Recommendation ITU-R BT.601 using me
13、thods such as described in Recommendation ITU-R BT.500 Methodology for the subjective assessment of the quality of television pictures. A data set for a VQM will consist of objective values and mean subjective scores for a variety of motion video sources (SRCs) processed by a variety of hypothetical
14、 reference circuits (HRCs). An example of such a data set is given in ITU-T Document COM 9-80-E, Final report from the video quality experts group on the validation of objective models of video quality assessment. The methods specified in this Recommendation are directly applicable to a defined data
15、 set as described above. For measurements not specifically part of the data set the methods specified in this Recommendation provide a reasonable estimate of accuracy and cross-calibration for applications that can be considered to be similar to and within the scope of the defined data set. Rec. ITU
16、-R BT.1676 3 The methods specified in this Recommendation are appropriate for use in combination with other statistical calculations in order to evaluate the usefulness of a VQM. Informative information regarding the use of the methods is presented in Appendix 1. A complete verification process by s
17、uitable independent laboratories is required for a VQM to be considered for inclusion as a normative part of an ITU-R Recommendation. 2 Accuracy of a VQM In order to use an objective VQM, one must know whether the score difference between two processed videos is statistically significant. Hence, a q
18、uantification is needed of the accuracy (or resolving power) of the VQM. To visualize this resolving power, it helps to begin with a scatter plot in which the abscissa of each point is a VQM score from a particular video SRC and distortion HRC, and the ordinate is a subjective score from a particula
19、r viewing of the SRC/HRC. Each SRC/HRC combination (associated with a particular VQM score) contains a distribution of mean subjective scores, S, based on a number of viewers, which represents (approximately) the relative probabilities of S for the particular SRC/HRC combination. The resolving power
20、 of a VQM can be defined as the difference between two VQM values for which the corresponding subjective-score distributions have means that are statistically different from each other (typically at the 0.95% significance level). Given this qualitative picture, two metrics for resolving power will b
21、e described in this section, each one useful in a different context. The metrics are described in 2.3 and 2.4. Also, in 2.5, a method is described for evaluating the frequencies of different kinds of errors made by the VQM. As an example of implementation of all the methods, a computer source code i
22、n MATLAB (The Mathworks, Inc., Natick, MA) is provided in Appendix 2. 2.1 Nomenclature and coordinate scales Let each SRC/HRC combination in a data set be called a “situation”, and let N be the number of situations in this data set. A subjective score for situation i and viewer l will be denoted as
23、Sil, and an objective score for situation i will be denoted as Oi. Averaging over a variable such as viewer will be denoted with a dot in that variable location. For instance, the mean opinion score of a situation will be denoted as .iS The subjective-score statistics from each pair (i, j) of these
24、situations are to be assessed for significance of VQM difference, and then used to arrive at a resolving power for the VQM difference, as a function of the VQM value. Prior to any statistical analysis, the original subjective mean opinion scores iS are linearly transformed to the interval 0, 1, defi
25、ned as the Common Scale, where 0 represents no impairment and 1 represents most impairment. If best represents the no-impairment value of the original subjective score and worst represents the maximum impairment of the original subjective scale, then the scaled scores iSare given by: bestwordbestSSi
26、i= Next, the VQM scores are transformed to this common scale as a by-product of the process of fitting the VQM scores to the subjective data, which will be discussed in the following section. 4 Rec. ITU-R BT.1676 2.2 Fitting VQM values to subjective data Fitting removes systematic differences betwee
27、n the VQM and the subjective data (e.g. d.c. shift) that do not provide any useful quality discrimination information. In addition, fitting all VQMs to one common scale will provide a method for cross-calibration of those VQMs. The simplest method of data fitting is linear correlation and regression
28、. For subjective video quality scores, this may not be the best method. Experience with other video quality data sets indicates chronically poor fits of VQM to subjective scores at the extremes of the ranges. This problem can be ameliorated by allowing the fitting algorithm to use non-linear, but st
29、ill monotonic (order-preserving), methods. If a good non-linear model is used, the objective-to-subjective errors will be smaller and have a central tendency closer to zero. Non-linear methods can be constrained to effectively transform the VQM scale to the 0, 1 common scale. Besides improving the f
30、it of data with a VQM, a fitting curve also offers an additional advantage over the straight-line fit implied by the Native Scale (i.e. the original scale of the VQM): the distribution of objective-to-subjective errors around the fitted model curve is less dependent on the VQM score. Of course, the
31、non-linear transformation may not remove all the score dependency of objective-to-subjective errors. To capture the residual dependence, it would ideally have been useful to record objective-to-subjective error as a function of VQM value. However, typical data sets are too small to divide among VQM
32、bins in a statistically robust way. Therefore, as will be clear in 2.3, a sort of average measure over the VQM range is computed. Figure 1 shows the improved fit of model to data incurred by transforming the objective scores using a fitting function. It can be seen that, besides improving the fit of
33、 data with VQM, the curve also offers an additional advantage over the straight-line fit implied by the native scale: the distri-bution of model-to-data errors around the fitted model curve is less dependent on the VQM score. We denote the original (native scale) objective scores Oi, and the common
34、scale objective scores as iO. A fitting function F (depending on some fitting parameters), connects the two. The function used to fit the objective VQM data (Oi) to the scaled subjective data )(iS must have the following three attributes: a specified domain of validity, which should include the rang
35、e of VQM data for all the situations used to define the accuracy metric; a specified range of validity, defined as the range of common scale scores (a sub-range of 0, 1) to which the function maps; monotonicity (the property of being either strictly increasing or strictly decreasing) over the specif
36、ied domain of validity. Of course, the fitting function would be most useful as a cross-calibration tool if it were monotonic over the entire theoretical domain of VQM scores, covered the entire subjective common scale from 0 to 1, and mapped to zero the VQM score that corresponds to a perfect video
37、 sequence Rec. ITU-R BT.1676 5 (no degradations, hence a null distortion). However, this ideal may not be attainable for certain VQMs and function families used to perform the fit. 1676-01FIGURE 1Improved fit of data to VQM by mapping VQM to common scaleVQM native scaleCommonscalesubjectivescore (10
38、0)DataLogistic fitOne possible family of fitting functions is the set of polynomials of order M. Another is a logistic function with the form: )(1/eiidOcbaO += where a, b, c, d and e are fitting parameters. A third possibility is a logistic function with the form: )(exp1)(/dOcabaOii+= where a, b, c,
39、 d are fitting parameters and c 0. For convenience, we call these logistic forms Logistic I and Logistic II, respectively. The MATLAB code in Appendix 2 instantiates only a polynomial fit. Appendix 3 discusses possible methods of data fitting using the logistic functions. 6 Rec. ITU-R BT.1676 The se
40、lection of a fitting-function family (including a priori setting of some of the parameters) depends on the asymptotic (best and worst) scores of the particular VQM. The number of degrees of freedom used up by the fitting process is denoted by D. For example, if a linear fit is used, D = 2 since two
41、free parameters are estimated in the fitting procedure. The fitting function that transforms objective VQM to the common scale is reported to facilitate industry comparison of two VQMs. Once transformed to the common scale, any VQM can be cross-calibrated to any other VQM through the common scale. R
42、epresenting the accuracy of a VQM in common scale facilitates comparisons between VQMs. Also, assuming the resolving power in the common scale does not vary much with the VQM score at which the resolving power is evaluated, the resolving power can be mapped through the inverse of the logistic functi
43、on to the native scale. In the native scale, the VQM from the common scale generates a VQM-score-dependent resolving power. A table or equation that provides such resolving powers (one at each VQM score in native scale) will have immediate meaning for users of the native scale. 2.3 METRIC 1: VQM acc
44、uracy based on statistical significance We define a new quantitative measure of VQM accuracy, called resolving power, defined as the VQM value above which the conditional subjective-score distributions have means that are statistically different from each other (typically at the 0.95 significance le
45、vel). Such an “error bar” measure is needed in order for video service operators to judge the significance of VQM fluctuations. Of several possible approaches to assessing a VQMs resolving power, the Students t-test was chosen. This test was applied to the measurements in all pairs i and j of situat
46、ions. Emerging from the test are the VQM (i.e. the difference between the greater and lesser VQM score of i and j) and the significance from the t-test. This significance is the probability p that, given i and j, the greater VQM score is associated with the situation that has the greater true underl
47、ying mean subjective score. Thus, p is the probability that the observed difference in sample means of the subjective scores from i and j did not come from a single population mean, nor from population means that were ordered oppositely to the associated VQM scores. To capture this ordering requirem
48、ent, the t-test must be one-tailed. For simplicity, the t-test was approximated by a z-test. This approximation is a close one when the number of viewers is large, as was the case for the Video Quality Experts Group (VQEG) data set (ITU-T Document COM 9-80-E). An analysis of variance (ANOVA) test mi
49、ght seem better than the t-test method. However, although a single application of ANOVA will determine whether a statistical separation exists among a set of categories, further paired comparisons are needed to determine the magnitudes and conditions of the statistically significant differences. Also, ANOVA assumes equal category-data variances (which may not be true). Finally, although ANOVA resides in many software packages, finding the right software package may not be easy (e.g. not all ANOVA routines will accept different quant
copyright@ 2008-2019 麦多课文库（www.mydoc123.com）网站版权所有
备案/许可证编号：苏ICP备17064731号-1