1、 TECHNICAL REPORT T1.TR.72-2003 Technical Report on Methodological Framework for Specifying Accuracy and Cross-Calibration of Video Quality Metrics Prepared by T1A1.3 Working Group on Performance of Networks and Services Problem Solvers to the Telecommunications Industry A Word from ATIS and Committ
2、ee T1 Established in February 1984, Committee T1 develops technical standards, reports and requirements regarding interoperability of telecommunications networks at interfaces with end-user systems, carriers, information and enhanced-service providers, and customer premises equipment (CPE). Committe
3、e T1 is sponsored by ATIS and is accredited by ANSI. T1.TR.82-2003 Published by Alliance for Telecommunications Industry Solutions 1200 G Street, NW, Suite 500 Washington, DC 20005 Committee T1 is sponsored by the Alliance for Telecommunications Industry Solutions (ATIS) and accredited by the Americ
4、an National Standards Institute (ANSI). Copyright 2004 by Alliance for Telecommunications Industry Solutions All rights reserved. No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission of the publisher. For info
5、rmation contact ATIS at 202.628.6380. ATIS is online at . Printed in the United States of America. T1.TR.72-2003 (Revision of T1.TR.72-2001) Technical Report Methodological Framework for Specifying Accuracy and Cross-Calibration of Video Quality Metrics Alliance for Telecommunications Industry Solut
6、ions Approved December 2003 Abstract There is an urgent industry need for documentation of video quality metrics (VQM). To meet that need T1A1 has developed a series of Technical Reports specifying accuracy and cross-calibration of video quality metrics (T1.TR.72-2001), normalization algorithms usef
7、ul in implementing Full Reference (FR) methods (T1.TR.73-2001), and VQM algorithms that are suitable for application in design, manufacturing, installation and servicing of video transmission systems incorporating compression techniques. Peak Signal to Noise Ratio (PSNR) is defined in T1.TR.74-2001.
8、 A Just-Noticeable Difference (JND) based VQM utilizing the FR technique is defined in T1.TR.75-2001. T1.TR.77-2002 defines data and sample program code to be used with the methods specified in T1.TR.74-2001 and T1.TR.75-2001. This Technical Report is the framework document (T1.TR.72-2003). T1.TR.72
9、-2003 Foreword Over the past ten years, the transmission of video using digital compression methods has progressed from limited video conferencing applications to widespread use in applications from high definition television to personal desktop computer communications. During this period, there hav
10、e been continuing efforts by laboratories and standards organizations to develop objective measurement methods to be used for quality of service (QoS) testing. In the mid 1990s a series of three standards - T1.801.01-1995 (R2001), T1.801.02-1996 (R2001), and T1.801.03-1996 - were issued by T1 that p
11、rovide background information and an extensive list of parametric calculations to be used in video performance assessment. T1.801.03-1996 provided the basis for further research, as well as the commonly used definition of peak signal to noise ratio (PSNR) for the luminance (Y) signal of component vi
12、deo that is sampled according to ITU-R Recommendation BT.601. As part of the industry-wide effort to develop video QoS measurements, three methodological approaches have been defined: 1. Full Reference (FR) - A method applicable when the full reference video signal is available. This is a double-end
13、ed method and is the subject of this report. 2. Reduced Reference (RR) - A method applicable when only reduced video reference information is available. This is also a double-ended method. 3. No Reference (NR) - A method applicable when no reference video signal or information is available. This is
14、a single-ended method. To address the validation and comparison of video-quality models, the Video Quality Experts Group (VQEG) was formed in 1997 as an informal subgroup of the ITU-T and ITU-R. VQEG members are experts from various backgrounds and affiliations, including participants from several i
15、nternationally recognized organizations working in video quality assessment. Over a two-year period, VQEG designed and implemented extensive subjective and objective test plans to evaluate a number of proponent algorithms for the FR method. PSNR was also included in these evaluation tests. Results o
16、f those phase-1 tests have been widely publicized (see ITU-T COM 9-80-E, June 2000). An excerpt is quoted here: The VQEG phase-1 test results based on the analysis obtained for the four individual subjective test quadrants essentially showed the following: No objective measurement system in the test
17、 was able to replace subjective testing. No objective model statistically outperformed the others in all reference conditions. No objective model statistically outperformed PSNR in all reference conditions. As a result, no single method was recommended to the ITU. Following the VQEG phase-1 test, Co
18、mmittee T1 published a series of technical reports that documented several phase-1 methods and their application to QoS testing. This series of TRs provided an extensible framework into which any documented video quality metric could be incorporated and quantitatively related to other previously dis
19、closed algorithms. T1.TR.72-2001, the first TR in the framework, covered methods for specifying the accuracy and cross-calibration of the video quality metrics. The second TR in the framework - T1.TR.73-2001 - covered normalization methods (e.g., spatial registration, temporal registration, and gain
20、 / level offset calibration). The third report, T1.TR.74-2001, covered specification of one video quality metric that is commonly used by industry, namely PSNR. The fourth report, T1.TR.75-2001, defined the JND-based PQR method. The fifth report, T1.TR.77-2002, defined data and sample program code t
21、o be used with the methods specified in T1.TR.74-2001 and T1.TR.75-2001. Many FR proponents made improvements to their original algorithms, and VQEG executed a new series of tests for the validation of these improved methods (i.e., phase 2). The results of the VQEG phase-2 tests (documented in ITU-T
22、 COM 9-C60-E, September 2003) demonstrated that several of the phase-2 methods significantly outperformed PSNR in all reference conditions and produced quality estimates that closely tracked the corresponding subjective quality ratings. Only one phase-2 method (method H in ITU-T COM 9-C60-E) was in
23、the top performing group for both the 525-line and 625-line video tests. As a result, method H was approved as a revision to T1.801.03 in July of 2003 (published as T1.801.03-2003) and it is anticipated that method H, as well as several other phase-2 methods, will be included in future ITU-R and ITU
24、-T Recommendations. The methods for specifying the accuracy of video quality metrics found in T1.TR.72-2001 were used by VQEG in their phase-2 tests. This application of T1.TR.72-2001 revealed a number of inconsistencies between the normative text and the non-normative sample program code. Hence, Co
25、mmittee T1A1 decided to revise the text of T1.TR.72-2001. The present document (T1.TR.72-2003) is the result of this effort. ii T1.TR.72-2003 Suggestions for improving this technical report are welcome and should be sent to the Alliance for Telecommunications Industry Solutions - Committee T1 Secret
26、ariat, 1200 G Street N.W., Suite 500, Washington, D.C. 20005. At the time this report was completed, the following were the contributors: C. Underkoffler, ATIS Chief Editor S. Wolf, T1A1 Technical Editor NAME AFFILIATION David Fibush Tektronix Dick Streeter CBS Alexander Woerner Rohde (b) An algorit
27、hm (based on statistical analysis relative to subjective data) to quantify the accuracy of a given VQM; (c) An algorithm for quantifying a transformation between the output of any new VQM and the output of any existing VQM; and (d) Guidelines for documenting and specifying the limitations of a VQM;
28、and (e) A list of the necessary procedural steps for incorporating a new VQM into the family of TRs. 1.2 Purpose This TR describes the basic framework to be used in conjunction with video quality measurements utilizing the Full Reference (FR) technique. FR video quality measurement methods are neces
29、sary to support the interconnection and interoperability of telecommunications networks at interfaces with end-user systems, carriers, information and enhanced-service providers, and customer premise equipment. 1.3 Application The methods specified in this TR are based on objective and subjective ev
30、aluation of component video such as defined by Recommendation ITU-R BT.601, using methods such as described in ITU-R BT.500. A data set for a VQM will consist of objective values and mean subjective scores for a variety of motion video sources (SRC) processed by a variety of hypothetical reference c
31、ircuits (HRC). An example of such a data set is the VQEG phase-1 data given in ITU-T COM9-80-E, Final report from the video quality experts group on the validation of objective models of video quality assessment (see Annex A). The methods specified in this TR are directly applicable to a defined dat
32、a set as described above. For measurements not specifically part of the data set, the methods specified in this TR provide a reasonable estimate of accuracy and cross-calibration for applications that can be considered to be similar to and within the scope of the defined data set. Although the metho
33、ds specified in this TR are based on processing component video such as defined by ITU-R Recommendation BT.601, this does not preclude implementation of measurement methods including composite video inputs and outputs. The conversion between composite and component domains is not part of this TR. 1
34、T1.TR.72-2003 2 Introduction Video Quality Metrics (VQMs) are intended to provide calculated values that are strongly correlated with viewer subjective assessments. Initial efforts - for example, by the Video Quality Experts Group (VQEG) - have shown that developing and validating a VQM standard is
35、a daunting task. Industry needs guidance to ensure specified levels of quality in networked video transport. For example, for the US telecommunication industry, end-to-end Quality of Service (QoS) across the networks of multiple companies requires agreement on quality at the transfer points. Without
36、 standards, different companies are likely to adopt different VQMs, and the communication gap between them will persist. Site of Video Origination(e.g., Denver)Transfer Between Network A (b) Validation/accuracy data, including subjective ratings and model outputs (spanning enough quality range to be
37、 representative of typical transmitted videos); and 2 T1.TR.72-2003 (c) Data for model-to-model VQM cross-calibration (a subset of the validation dataset). Finally, there should be descriptions of scope and limitations, accuracy, and model cross-calibration as described in subsequent clauses of this
38、 TR. 4 Scope/Limitations of a VQM The scope of a VQM can include the following elements (an illustrative list, intended neither to be prescriptive nor exhaustive): (a) The type of scene content (“signal”) - e.g., high/low motion, color versus black-and-white, interlaced versus progressive. (b) The t
39、ype and severity of artifacts (“noise”), driven by encoding techniques and bit rates - e.g., blurring, blockiness. (c) The viewing conditions (including viewing distance, ambient illumination, and display parameters such as gamma, brightness, and phosphor types). Each VQM should be qualitatively ass
40、essed as to the type of scene content, type and severity of artifacts, and viewing conditions under which the VQM can and cannot operate effectively. It is important to list known problem areas (such as video distortions that include dropped frames) that would otherwise not be obvious, but the Scope
41、/Limitations clause is not intended to be an exhaustive list. A set of four tables should be included in the description of the VQMs scope and limitations. The first three of these tables should enumerate all the distortions - Hypothetical Reference Circuits (HRCs) - of the data set of the Video Qua
42、lity Experts Group, and optionally others, as follows: Table a) A table of test factors, coding technologies, and applications for which the VQM has shown accuracy as specified in clause 4; Table b) A table of test factors, coding technologies, and applications for which the VQM has been tested but
43、not shown the accuracy specified in clause 5; and Table c) A table of known test factors, coding technologies, and applications for which the VQM has not been tested, or where the VQM is not recommended. In addition, there should be: Table d) A table of test sequences used to determine test factors,
44、 coding technologies and applications for which the VQM has shown the accuracy specified in clause 5. For the VQEG phase-1 full reference television (FR-TV) tests, sample Table 1 below contains the values for Table a), Table 2 contains the values for Table b), and Table 3 contains the values for Tab
45、le d). Since these three tables exhaust the VQEG phase-1 data set, the sample Table for c) would not contain any entries. The VQEG phase-2 data (ITU-T COM 9-C60-E, September 2003) has not been included since copyright restrictions prevent this data set from being generally available for VQM validati
46、on testing. 3 T1.TR.72-2003 Table 1 - Test factors, coding technologies and applications for which the candidate VQM method has shown the specified accuracy BIT RATE RES METHOD COMMENTS 2 Mb/s resolution mpml This is horizontal resolution reduction only. 2 Mb/s resolution spml 4.5 Mb/s mpml 3 Mb/s m
47、pml 1.5 Mb/s CIF H.263 768 kb/s CIF H.263 4.5 Mb/s mpml Composite NTSC and/or PAL. 6 Mb/s mpml 8 Mb/s mpml Composite NTSC and/or PAL. 8 (b) A specified range of validity, defined as the range of Common Scale scores (a sub-range of 0,1) to which the function maps; and (c) Monotonicity (the property o
48、f being either strictly increasing or strictly decreasing) over the specified domain of validity. Of course, the fitting function would be most useful as a cross-calibration tool if it were monotonic over the entire theoretical domain of VQM scores, covered the entire subjective Common Scale from 0
49、to 1, and mapped to zero the VQM score that corresponds to a perfect video sequence (no degradations, hence a null distortion). However, this ideal may not be attainable for certain VQMs and function families used to perform the fit. One possible family of fitting functions is the set of polynomials of order M. Another is a logistic function with the form: iO= a + b/1 + c(O +d)e, iwhere a, b, c, d, and e are fitting parameters2. A third possibility is a logistic function with the form: iO= a + (b-a)/1 + exp-c( - d), iOwhere a, b, c, d are fitt
