ANSI ATIS T1.TR.74-2001 Objective Video Quality Measurement Using a Peak- Signal-to-Noise-Ratio (PSNR) Full Reference Technique.pdf

1、 TECHNICAL REPORT T1.TR.74-2001 Technical Report on Objective Video Quality Measurement Using a Peak-Signal-to-Noise-Ratio (PSNR) Full Reference Technique Prepared by T1A1.1 Working Group on Multimedia Communications Coding and Performance Problem Solvers to the Telecommunications Industry A Word fr

2、om ATIS and Committee T1 Established in February 1984, Committee T1 develops technical standards, reports and requirements regarding interoperability of telecommunications networks at interfaces with end-user systems, carriers, information and enhanced-service providers, and customer premises equipm

3、ent (CPE). Committee T1 is sponsored by ATIS and is accredited by ANSI. T1.TR.74-2001 Published by Alliance for Telecommunications Industry Solutions 1200 G Street, NW, Suite 500 Washington, DC 20005 Committee T1 is sponsored by the Alliance for Telecommunications Industry Solutions (ATIS) and accre

4、dited by the American National Standards Institute (ANSI). Copyright 2002 by Alliance for Telecommunications Industry Solutions All rights reserved. No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission of the

5、publisher. For information contact ATIS at 202.628.6380. ATIS is online at . Printed in the United States of America. T1.TR.74-2001 Technical Report Objective Video Quality Measurement Using a Peak-Signal-to-Noise-Ratio (PSNR) Full Reference Technique Alliance for Telecommunications Industry Solutio

6、ns Approved October 2001 Abstract This document provides a draft of the technical report for measuring the peak-signal-to-noise-ratio (PSNR) of one-way video signals. This draft technical report is one of a series of four that are being developed by the ad hoc group on video quality metrics. T1.TR.7

7、4-2001 ii Foreword Over the past ten years, the transmission of video using digital compression methods has progressed from limited video conferencing applications to widespread use in applications from high definition television to personal desktop computer communications. During this period there

8、have been continuing efforts by laboratories and standards organizations to develop objective measurement methods to be used for video quality of service (VQoS) testing. In the mid 1990s a series of three standards - T1.801.01-1995 (R2001), T1.801.02-1996 (R2001), and T1.801.03-1996 - were issued by

9、 T1 that provide background information and an extensive list of parametric calculations to be used in video performance assessment. While many of the parametric calculations in T1.801.03-1996 have not, to date, been implemented in commercial measurement instruments, this standard provided the basis

10、 for further research as well as the commonly used definition of peak signal to noise ratio (PSNR) for the luminance (Y) signal of component video that is sampled according to ITU-R Recommendation BT.601. As part of the industry-wide effort to develop VQoS measurements, three methodological approach

11、es have been defined. Full Reference (FR) - A method applicable when the full reference video signal is available. This is a double-ended method and is the subject of this report. Reduced Reference (RR) - A method applicable when only reduced video reference information is available. This is also a

12、double-ended method. No Reference (NR) - A method applicable when no reference video signal or information is available. This is a single-ended method. To address the validation and comparison of video-quality models, the Video Quality Experts Group (VQEG) was formed in 1997 as an informal subgroup

13、of the ITU-T and ITU-R. VQEG members are experts from various backgrounds and affiliations, including participants from several internationally recognized organizations working in video quality assessment. Over a two-year period, VQEG designed and implemented extensive subjective and objective test

14、plans to evaluate a number of proponent algorithms for the FR method. PSNR was also included in these evaluation tests. Results of those tests have been widely publicized (see ITU-T COM 9-80). An excerpt is quoted here: The VQEG test results based on the analysis obtained for the four individual sub

15、jective test quadrants essentially show the following: No objective measurement system in the test is able to replace subjective testing. No objective model statistically outperforms the others in all reference conditions. No objective model statistically outperforms PSNR in all reference conditions

16、. Based on present evidence, no single method can be recommended to ITU at this time. Many FR proponents have made improvements to their original algorithms and VQEG is expected to execute a new series of tests for the validation of these improved methods. Committee T1 is awaiting the results of the

17、se new VQEG validation tests before proceeding with the standardization of any one FR method. To provide immediate guidance for use by industry, Committee T1 has instead decided to publish a series of technical reports that will document existing FR methods and their application to VQoS testing. The

18、 series of TRs provide an extensible framework into which any documented video quality metric can be incorporated and quantitatively related to other previously disclosed algorithms. The first TR in the framework (T1.TR.72-2001) covers methods for specifying the accuracy and cross calibration of the

19、 video quality metrics. The second TR in the framework (T1.TR.73-2001) covers normalization methods (e.g., spatial registration, temporal registration, and gain / level offset calibration). The third and present T1.TR.74-2001 covers specification of one video quality metric that is commonly used by

20、industry, namely peak-signal-to-noise-ratio (PSNR). Suggestions for improving this technical report are welcome and should be sent to the Alliance for Telecommunications Industry Solutions - Committee T1 Secretariat, 1200 G Street N.W., Suite 500, Washington, D.C. 20005. At the time this report was

21、completed, T1A1.1 had the following contributors: T1.TR.74-2001 iii Name Affiliation Ahmad Ansari SBC Michael H. Brill Sarnoff Corporation Greg Cermak Verizon Phil Corriveau CRC Pierre Costa SBC David Fibush Tektronix John Grigg Qwest Jeffrey Lubin Sarnoff Corporation Harley Myler University of Cent

22、ral Florida John Pearson Sarnoff Corporation Debra Phillips SBC Margaret Pinson NTIA Dick Streeter CBS Stephen Voran NTIA A. B. Watson NASA Ames Research Center Alexander Woerner Rohde it may not necessarily denote actual time. T1.TR.74-2001 9 Figure 4 - Coordinate system used for Y frames, ITU R Re

23、commendation BT.601 5 PSNR method of measurement The peak signal to noise ratio is the ratio of the peak signal to root mean squared (RMS) noise observed between the processed video signal and the reference video signal. Computation of PSNR involves the following steps: 1. Selection of the processed

24、 video sequence (see 5.1). 2. Normalization of the processed video sequence (see 5.2). This step includes finding the matching set of reference video frames corresponding to the processed video sequence that was selected. 3. Selection of the spatial region of interest (SROI) over which to compute PS

25、NR (see 5.3). 4. Computation of frame-by-frame PSNR values (see 5.4). 5. Computation of the sequence PSNR (see 5.5). 5.1 Selection of the processed video sequence Select a sequence of M processed video frames (see Figure 1) over which to compute PSNR. Using the ITU R Recommendation BT.601 coordinate

26、 system and up sampling techniques described in clause 4, the processed Y sequence will be denoted as ()10=MprocnntY . Y(0, 0, tn) jiY(tn)Y(tn-1)Y(i, j, tn)hvtT1.TR.74-2001 10 5.2 Normalization of processed video sequence The normalization step includes the following sub-steps: 1. Finding the matchi

27、ng set of reference video frames corresponding to the processed video sequence (i.e., temporal registration or one-way video delay measurement). 2. Performing horizontal and vertical spatial registration of the processed video sequence. 3. Performing gain and level offset correction of the processed

28、 video sequence. Whatever normalization procedures are used to perform sub-steps 1 to 3, the tolerances given in Table 1 should be achieved. The recommended normalization procedures are specified in T1.TR.73-2001. Table 1 - Normalization Requirements for PSNR Parameter Normalization Tolerance Lumina

29、nce gain 0.2 dB Luminance DC level 0.5 % of signal max Horizontal pixel shift 0.1 pixel Vertical line shift 0.1 line This tolerance implies field-accurate temporal registration. The matching sequence of M reference video frames (to the M processed video frames selected in 5.1), using the ITU R Recom

30、mendation BT.601 coordinate system will be denoted as ()10=Mref nndtY . Here d is the delay of the one-way video transmission system determined during temporal registration sub-step 1. The final normalized sequence of M processed video frames will be denoted as ()10=MprocnndtY . 5.3 Selection of the

31、 spatial region of interest (SROI) over which to compute PSNR The recommended spatial region of interest (SROI) for computation of PSNR is the clean aperture defined in SMPTE Recommended Practice 187. Some one-way video transmission systems may not produce a valid picture throughout this entire clea

32、n aperture. For these systems, the SROI may be reduced provided it is reported together with the measured PSNR values. Figure 5 defines the specification for SROI for the luminance video frame Y(tn) depicted in Figure 4. Here, Ohand Ovgive the offset (in pixels and lines, respectively) of the rectan

33、gular sub-region from the upper left hand corner of the image (0, 0). The rectangular sub-region contains Nhhorizontal pixels and Nvvertical lines. T1.TR.74-2001 11 (0, 0)OhOvhvY(tn)NhNvFigure 5 - Rectangular sub-region used to specify SROI for Y frames, ITU R Recommendation BT.601 5.4 Computation o

34、f frame-by-frame PSNR values The frame-by-frame PSNR values8are computed for the ITU R Recommendation BT.601 Y component signal as Eq. (4) ()()()()=+=+=21110,1log20hhhvvvNOONOOprocrefvhpeakNNYjinnntjiYdtjiYtPSNR , where Ypeakis defined to be 235 and Nv, Nh, Ov, and Ohare defined in 5.3. _ 8If higher

35、 PSNR temporal sampling rates are desired (e.g., field-by-field rather than frame-by-frame), one can compute PSNR values using appropriate sub-portions of the ITU R Recommendation BT.601 video frame. The equations given in 5.4 must be suitably modified. T1.TR.74-2001 12 5.5 Computation of the sequen

36、ce PSNR The M-frame sequence peak signal to noise ratio values are computed for the ITU R Recommendation BT.601 Y component signal as Eq. (5) ()()()=+=+=2111010,1log20hhhvvvNOONOOprocrefMvhpeakNNMYjinnntjiYdtjiYPSNR , where Ypeakis defined to be 235 and Nv, Nh, Ov, and Ohare defined in 5.3. M is the

37、 number of frames in the sequence. T1.TR.74-2001 13 Annex A (Informative) A Bibliography T1.801.01-1995, “Digital Transport of Video Teleconferencing/Video Telephony Signals - Video Test Scenes for Subjective and Objective Performance Assessment”6T1.801.02-1995, “Digital Transport of Video Teleconfe

38、rencing/Video Telephony Signals - Performance Terms, Definitions and Examples”6T1.801.03-1996, “Digital Transport of One-Way Digital Signals Parameters for Objective Performance Assessment”6ITU-T COM 9-80-E, Rapporteur Q11/12 (VQEG), “Final report from the video quality experts group on the validati

39、on of objective models of video quality assessment”4ITU-R Recommendation BT.500 (03/2000), “Methodology for the Subjective Assessment of the Quality of Television Pictures”3SMPTE 170M-1994, “SMPTE Standard for Television - Composite Analog Video Signal - NTSC for Studio Applications”5T1.TR.74-2001 1

40、4 Annex B (Informative) B VQEG PSNR Calculation The PSNR calculation given by Eq. (5) in 5.5 was applied to the VQEG 525-line subjective data with HRCs 15 and 16 removed (see ITU-T COM 9-80-E reference in Annex A for a description of the VQEG data). The spatial region of interest (SROI, see 5.3) for

41、 the PSNR calculation was: Ov= 20, Oh= 24, Nv= 448, Nh= 672 except for the scenes “Autumn Leaves” (scene number 18 in ITU-T COM 9-80-E) and “Football” (scene number 19 in ITU-T COM 9-80-E), whose SROI was reduced slightly as follows to eliminate the black border that was present in these scenes: Aut

42、umn Leaves (525-line high-quality and low-quality): Ov= 20, Oh= 24, Nv= 448, Nh= 666 Football (525-line high-quality): Ov= 20, Oh= 24, Nv= 448, Nh= 668 Football (525-line low-quality): Ov= 20, Oh= 24, Nv= 448, Nh= 666 Using the above spatial regions of interest with the VQEG 525-line data produced t

43、he following PSNR values: T1.TR.74-2001 15 Scene HRC PSNR 13 1 28.81 14 1 28.13 15 1 21.55 16 1 28.22 17 1 26.44 18 1 29.78 19 1 30.25 20 1 23.18 21 1 30.28 22 1 26.5 13 2 36.51 14 2 39.22 15 2 29.94 16 2 40.97 17 2 35.29 18 2 37.44 19 2 36.22 20 2 35.87 21 2 39.74 22 2 33.41 13 3 33.28 14 3 35.69 1

44、5 3 27.32 16 3 39.2 17 3 33.5 18 3 33.32 19 3 33.82 20 3 30.6 21 3 36.79 22 3 30.62 13 4 35.36 14 4 37.66 15 4 26.67 16 4 37.47 17 4 32.5618 4 35.4319 4 36.1220 4 30.7521 4 39.4922 4 31.6413 5 32.3414 5 37.7415 5 27 16 5 40.4 17 5 31.6918 5 36.4619 5 32.8520 5 34.1221 5 39.4122 5 30.8613 6 35.5414 6

45、 34.4615 6 25.5816 6 37.8317 6 32.5118 6 36.6519 6 34.5720 6 31.8221 6 39.6622 6 31.8313 7 34.0614 7 40.0715 7 29 16 7 43.7617 7 33.9418 7 38.2319 7 34.9620 7 34.6121 7 40.9522 7 32.8913 8 33.3714 8 34.0915 8 24.8916 8 37.2617 8 31.1418 8 35.9519 8 32.5820 8 31.2521 8 40.9522 8 30.4313 9 30.6 14 9 3

46、7.5515 9 25.6516 9 40.4917 9 29.7518 9 36.2219 9 30.7420 9 34.1221 9 39 22 9 29.9913 8 33.3714 8 34.0915 8 24.8916 8 37.2617 8 31.1418 8 35.9519 8 32.5820 8 31.2521 8 38.8722 8 30.4313 9 30.6 14 9 37.5515 9 25.6516 9 40.49T1.TR.74-2001 16 17 9 29.75 18 9 36.22 19 9 30.74 20 9 34.12 21 9 39 22 9 29.9

47、9 13 10 32.87 14 10 38.72 15 10 27.48 16 10 42.97 17 10 32.64 18 10 37.17 19 10 33.46 20 10 34.47 21 10 40.27 22 10 31.56 13 11 29.34 14 11 36.34 15 11 25.14 16 11 38.05 17 11 27.7 18 11 35.75 19 11 29.55 20 11 32.79 21 11 38.52 22 11 28.61 13 12 31.12 14 12 37.99 15 12 26.91 16 12 39.29 17 12 28.99

48、 18 12 36.96 19 12 31.99 20 12 34.64 21 12 39.6722 12 30.4813 13 31.3 14 13 33.2 15 13 22.5716 13 36.8417 13 25.5818 13 33.5119 13 29.8320 13 28.9621 13 37.5522 13 26.8713 14 29.0214 14 34.6115 14 23.2816 14 37.0317 14 28.0418 14 34.4 19 14 29.4620 14 29.9121 14 38.1422 14 27.88T1.TR.74-2001 1 Annex

49、 C (Informative) C Test Factors, Coding Technologies, and Applications See the VQEG final report (ITU-T COM 9-80-E) for further details regarding the data in these tables. All data is for the 525-line system. Table C 1 - Test factors, coding technologies, and applications for which the PSNR method has shown the accuracy specified in 1.4 Bit Rate Resolution Coding Comments 2 Mb/s resolution mpml This is horizontal resolution reduction only 2 Mb/s resolution spml 4.5 Mb/s mpml With errors 3 Mb/