ANSI ATIS T1.TR.77-2002 Data and sample program code to be used with the method specified in T1.TR.72-2001 for the calculation of resolving power of the video quality metrics in T1.pdf

1、 TECHNICAL REPORT T1.TR.77-2002 Technical Report on Data and sample program code to be used with the method specified in T1.TR.72-2001 for the calculation of resolving power of the video quality metrics in T1.TR.74-2001 and T1.TR.75-2001 Prepared by T1A1.1 Working Group on Multimedia Communications

2、Coding and Performance Problem Solvers to the Telecommunications Industry A Word from ATIS and Committee T1 Established in February 1984, Committee T1 develops technical standards, reports and requirements regarding interoperability of telecommunications networks at interfaces with end-user systems,

3、 carriers, information and enhanced-service providers, and customer premises equipment (CPE). Committee T1 is sponsored by ATIS and is accredited by ANSI. T1.TR.77-2002 Published by Alliance for Telecommunications Industry Solutions 1200 G Street, NW, Suite 500 Washington, DC 20005 Committee T1 is s

4、ponsored by the Alliance for Telecommunications Industry Solutions (ATIS) and accredited by the American National Standards Institute (ANSI). Copyright 2002 by Alliance for Telecommunications Industry Solutions All rights reserved. No part of this publication may be reproduced in any form, in an ele

5、ctronic retrieval system or otherwise, without the prior written permission of the publisher. For information contact ATIS at 202.628.6380. ATIS is online at . Printed in the United States of America. T1.TR.77-2002 Technical Report Data and sample program code to be used with the method specified in

6、 T1.TR.72-2001 for the calculation of resolving power of the video quality metrics in T1.TR.74-2001 and T1.TR.75-2001 Alliance for Telecommunications Industry Solutions Approved January 2002 Abstract This Technical Report contains a listing of the VQEG source (SRC) and hypothetical reference circuit

7、 (HRC) combinations and the subset used in Technical Reports T1.TR.74-2001 and T1.TR.75-2001. It also provides the data and sample Matlab m-files required to apply the analysis of Technical Report T1.TR.72-2001. T1.TR.77-2002 Foreword There is an urgent industry need for documentation of video quali

8、ty metrics (VQM). To meet that need Committee T1A1 has developed a series of Technical Reports specifying accuracy and cross calibration of video quality metrics, T1.TR.72-2001, normalization algorithms useful in implementing full reference VQMs, T1.TR.73-2001, and VQM algorithms that are suitable f

9、or application in design, manufacturing, installation and servicing of video transmission systems incorporating compression techniques. PSNR is defined in Technical Report T1.TR.74-2001 and Technical Report T1.TR.75-2001 specifies a JND-based VQM utilizing the full reference technique. This draft Te

10、chnical Report contains the data and sample Matlab m-files required for implementation of the analysis in T1.TR.72-2001 for the VQMs specified in T1.TR.74-2001 and T1.TR.75-2001. Suggestions for improvement of this standard will be welcome. These should be sent to the Alliance for Telecommunications

11、 Industry Solutions, T1 Secretariat, 1200 G Street, NW, Suite 500, Washington DC 20005. This standard was processed and approved for submittal to ANSI by the Accredited Standards Committee Telecommunications, T1. Committee approval of this standard does not necessarily imply that all committee membe

12、rs voted for its approval. Working Group T1A1.1 developed this standard. Over the course of its development, the following individuals participated in the Working Groups discussions and made significant contributions to the standard: Name Affiliation David Fibush Tektronix Dick Streeter CBS Alexande

13、r Woerner Rohde HRC; VQM (PSNR or PQR); number of viewers; MDOS (mean difference opinion score); Variance of the MDOS. Convert the tables to text files for use with the Matlab code. Note - Soft copies of the data files are available from ATIS headquarters in Washington DC. They are also available as

14、 attachments to contribution T1A1.1/2001-46R3 (1A110463). 5 Abbreviations, Acronyms, and Symbols ANSI American National Standards Institute ATIS Alliance for Telecommunications Industry Solutions DSCQS Double Stimulus Continuous Quality Scale HRC Hypothetical Reference Circuit Hz Hertz 3 T1.TR.77-20

15、02 ITU-R International Telecommunications Union - Radio ITU-T International Telecommunications Union - TelecommunicatonsJND Just-Noticeable Difference kb/s Kilo bits per second Mb/s Mega bits per second MDOS Mean Difference Opinion Score PQR Picture Quality Rating PSNR Peak-Signal-To-Noise-Ratio SRC

16、 Source Sequence VQEG Video Quality Experts Group VQM Video Quality Metrics 4 T1.TR.77-2002 Annex A (informative) A Bibliography ITU-T COM9-80-E, Rapporteur Q11/12 (VQEG): Final report from the video quality experts group on the validation of objective models of video quality assessment, June 2000.1

17、ITU-R Recommendation BT.500-10 (03/00)BT.500, Methodology for the Subjective Assessment of the Quality of Television Pictures.2T1.TR.72-2001, Methods to Specify Accuracy and Cross Calibration of Video Quality Metrics.3T1.TR.74-2001, Objective Perceptual Video Quality Measurement Using Peak Signal-to

18、-Noise Ratio Full.3T1.TR.75-2001, Objective Perceptual Video Quality Measurement Using a JND-Based Full Reference Technique.3T1A1.1/2001-46R3 (1A110463) 09/25/2001, Data and sample program code to be used with the method specified in Technical Report T1.TR.72-2001 for the calculation of resolving po

19、wer of the video quality metrics in Technical Reports T1.TR.74-2001 and T1.TR.75-2001.3_ 1This document is available from the International Telecommunications Union. 2This document is available from the International Telecommunications Union. 3This document is available from the Alliance for Telecom

20、munications Industry Solutions, 1200 G Street N.W., Suite 500, Washington, DC 20005. 5 T1.TR.77-2002 Annex B B Matlab m-file for PSNR, Technical Report T1.TR.74-2001 function vqm_accuracy_psnr2 (data_in, vqm_sign, best, worst, order) % MATLAB function vqm_accuracy_psnr2 (data_in, vqm_sign, best, wor

21、st, order) % % Each row of the input data matrix data_in must be organized as % src_id hrc_id vqm num_view mos variance, where % % src_id is the scene number % hrc_id is the hypothetical reference circuit number % vqm is the video quality metric score for this src_id x hrc_id % num_view is the numbe

22、r of viewers that rated this src_id x hrc_id % mos is the mean opinion score of this src_id x hrc_id % variance is the variance of this src_id x hrc_id % % The total number of src x hrc combinations is size(data_in,1). % % vqm_sign = 1 or -1 and gives the direction of vqm with respect to % the commo

23、n subjective scale. For instance, since “0“ is % no impairment and “1“ is maximum impairment on the common % scale, vqm_sign would be -1 for PSNR since higher values % of PSNR imply better quality (i.e., this is opposite to % the common subjective scale). % % mos and variance will be linarly scaled

24、such that % best is scaled to zero (i.e., the best subjective rating) % worst is scaled to one (i.e., the worst subjective rating) % % order is the order of the polynomial fit used to map the objective data % to the scaled subjective data (e.g., order = 1 is a linear fit). If % order = -2, a logisti

25、cs 2 function is used with 2 degrees of freedom. % % Number of src x hrc combinations num_comb = size(data_in,1); % Pick off the vectors we will use from data_in vqm = data_in(:,3); num_view = data_in(:,4); mos = data_in(:,5); variance = data_in(:,6); % Scale the subjective data for 0,1 mos = (mos-b

26、est)./(worst-best); variance = variance./(worst-best)2); % Fit the objective data to the scaled subjective data. format(long); if (order = 1) % use constrained polynomial fit with this order % Following code implements monotonic polynomial fitting using optimization % toolbox routine lsqlin. 6 T1.TR

27、.77-2002 % % Create x and dx arrays. For the dx slope array (holds the derivatives of % mos with respect to vqm), the vqm_sign specifies the direction of the slope % that must not change over the vqm range. x = ones(num_comb,1); dx = zeros(num_comb,1); for col = 1:order x = x vqm.col; dx = dx col*vq

28、m.(col-1); end % The lsqlin routine uses = subj_th); end rel_hat_freqs(i,3) = length(find(vqm_hat_th = 1) % use constrained polynomial fit with this order % Following code implements monotonic polynomial fitting using optimization % toolbox routine lsqlin. 13 T1.TR.77-2002 % % Create x and dx arrays

29、. For the dx slope array (holds the derivatives of % mos with respect to vqm), the vqm_sign specifies the direction of the slope % that must not change over the vqm range. x = ones(num_comb,1); dx = zeros(num_comb,1); for col = 1:order x = x vqm.col; dx = dx col*vqm.(col-1); end % The lsqlin routine

30、 uses = subj_th); end rel_hat_freqs(i,3) = length(find(vqm_hat_th = delta_vqm_hat) end % Normalize counts by total number of points to get relative frequencies rel_freqs = rel_freqs/length(z_vqm); rel_hat_freqs = rel_hat_freqs/length(z_vqm_hat); % Calculate relative frequency of correctness rel_freq

31、s(:,4) = (1-sum(rel_freqs(:,1:3); rel_hat_freqs(:,4) = (1-sum(rel_hat_freqs(:,1:3); % Figure 5 is plot for vqm and figure 6 is plot for vqm_hat. figure(5) plot(vqm_th_list,rel_freqs(:,1),m-., vqm_th_list,rel_freqs(:,2),r:, . vqm_th_list,rel_freqs(:,3),k-,vqm_th_list,rel_freqs(:,4),b-); grid set(gca,

32、LineWidth,1) set(gca,FontName,Ariel) set(gca,fontsize,12) %xlabel(Delta VQM Significance Threshold) %xlabel(Delta PSNR Significance Threshold) xlabel(Delta PQR Significance Threshold) ylabel(Relative Frequencies) legend(False Tie,False Differentiation,False Ranking,Correct Decision) %title(VQM Subje

33、ctive Classification Errors) print -dpng figure5 figure(6) plot(vqm_hat_th_list,rel_hat_freqs(:,1),m-., . vqm_hat_th_list,rel_hat_freqs(:,2),r:, . vqm_hat_th_list,rel_hat_freqs(:,3),k-, . vqm_hat_th_list,rel_hat_freqs(:,4),b-); grid set(gca,LineWidth,1) set(gca,FontName,Ariel) 18 T1.TR.77-2002 set(g

34、ca,fontsize,12) %xlabel(Delta VQM Hat Significance Threshold) xlabel(Delta VQM Significance Threshold) ylabel(Relative Frequencies) legend(False Tie,False Differentiation,False Ranking,Correct Decision) %title(VQM Hat Subjective Classification Errors) print -dpng figure6 19 T1.TR.77-2002 Annex D D r

35、0.dat for PSNR, Technical Report T1.TR.74-2001 13 1 29.5104 67 12.8 273.65 14 1 28.8391 67 25.4896 436.86 15 1 22.2518 67 33.8627 555.76 16 1 28.918 67 32.1881 588.75 17 1 27.1466 67 7.5955 186.44 18 1 30.4894 67 29.6313 498.06 19 1 30.9623 67 19.903 381.56 20 1 23.86 67 35.7239 622.99 21 1 30.9557

36、67 29.6164 512 22 1 27.208 67 26.8851 623.37 13 2 37.2004 67 5.691 190.98 14 2 39.9515 67 2.1597 128.49 15 2 30.6426 67 17.7627 334.32 16 2 41.6895 67 2.3418 135.73 17 2 35.9675 67 1.9866 137.92 18 2 38.1353 67 5.9567 205.75 19 2 36.923 67 4.3821 115.64 20 2 36.5667 67 -0.5015 155.89 21 2 40.3947 67

37、 6.4075 242.64 22 2 34.1257 67 4.3119 179.68 13 3 33.9842 67 4.803 133.66 14 3 36.4141 67 11.891 258.42 15 3 28.0379 67 22.0642 389.22 16 3 39.8783 67 3.903 133.27 17 3 34.2064 67 4.1373 155.7 18 3 34.0422 67 13.5463 473.9 19 3 34.5368 67 2.5791 50.87 20 3 31.2913 67 15.0239 256.09 21 3 37.4167 67 5

38、.9716 181.52 22 3 31.3359 67 9.3478 164.4 13 4 36.068 67 11.0746 371.64 14 4 38.4125 67 6.309 200.59 20 T1.TR.77-2002 15 4 27.3777 61 24.541 361.8 16 4 38.1829 67 4.6313 128.47 17 4 33.2461 67 5.103 207.03 18 4 36.1527 67 11.791 317.87 19 4 36.8385 67 7.4582 162.35 20 4 31.4572 67 2.4403 181.35 21 4

39、 40.1555 67 9.4104 253.87 22 4 32.3654 67 7.7388 182.4 13 5 33.0558 67 11.0567 238.62 14 5 38.4956 67 7.9746 108.49 15 5 27.7067 67 21.3597 411.86 16 5 41.1231 67 3.903 114.15 17 5 32.3807 67 10.7119 279.49 18 5 37.1835 67 12.5836 179.95 19 5 33.57 67 11.4 314.57 20 5 34.8106 67 4.294 110.07 21 5 40

40、.0707 67 -0.6642 192.18 22 5 31.5915 67 8.7418 254.47 13 6 36.2428 67 10.4119 227.22 14 6 35.1934 67 12.8776 343.23 15 6 26.2817 67 32.2627 374.81 16 6 38.5591 67 4.9194 183.06 17 6 33.2017 67 3.5134 134.23 18 6 37.3673 67 6.5537 177.59 19 6 35.2885 67 10.6612 122.45 20 6 32.5246 67 2.1343 112.5 21

41、6 40.3182 67 1.4791 333.32 22 6 32.55 67 6.8119 240.24 13 7 34.7733 67 8.1224 135.91 14 7 40.8342 67 4.1567 142.32 15 7 29.7059 67 13.4433 302.97 16 7 44.4825 67 4.3866 125.89 17 7 34.6334 67 7.3224 133.91 18 7 38.9453 67 2.8537 159.15 19 7 35.6653 67 2.691 107.48 20 7 35.3041 67 4.8582 161.42 21 7

42、41.609 67 -2.9836 111.28 22 7 33.6126 67 3.4821 145.53 13 8 34.0686 67 13.7955 289.96 21 T1.TR.77-2002 14 8 34.8244 67 19.2254 235.71 15 8 25.5987 67 34.7209 341.11 16 8 37.9883 67 2.209 188.79 17 8 31.8261 67 6.891 213.1 18 8 36.6593 67 8.3194 188.31 19 8 33.2996 67 11.7552 318.21 20 8 31.9548 67 2

43、.4493 154.93 21 8 39.5209 67 2.2179 290.3 22 8 31.1535 67 7.7224 214.48 13 9 31.3002 67 23.9612 418.48 14 9 38.3112 67 7.1134 156.37 15 9 26.3524 67 23.4716 310.98 16 9 41.2101 67 6.5224 172.16 17 9 30.4348 67 18.2806 416.44 18 9 36.9227 67 8.8224 124.78 19 9 31.4578 67 24.9672 545.01 20 9 34.8229 6

44、7 2.6358 102.95 21 9 39.6416 67 0.1716 107.88 22 9 30.708 67 7.9119 206.57 13 8 34.0686 80 19.79 294.37 14 8 34.8244 80 20.4287 371.82 15 8 25.5987 80 24.9538 445.36 16 8 37.9883 80 9.6938 237.57 17 8 31.8261 80 9.1662 348.02 18 8 36.6593 80 12.1437 254.77 19 8 33.2996 80 15.0825 214.47 20 8 31.9548

45、 80 7.8688 262.96 21 8 39.5209 80 -2.0925 156.01 22 8 31.1535 80 9.3912 218.81 13 9 31.3002 80 28.65 537.09 14 9 38.3112 80 11.395 272.71 15 9 26.3524 80 28.4187 283.73 16 9 41.2101 79 4.6266 111.64 17 9 30.4348 80 12.8713 350.67 18 9 36.9227 80 8.265 194.05 19 9 31.4578 80 33.2438 869.72 20 9 34.82

46、29 80 -2.1988 126.58 21 9 39.6416 80 5.3013 135.09 22 9 30.708 80 5.58 327.93 22 T1.TR.77-2002 13 10 33.5802 80 16.795 221.83 14 10 39.48 80 5.8162 175.29 15 10 28.184 80 18.5688 346.11 16 10 43.6867 80 19.4725 987.11 17 10 33.3315 80 13.625 281.31 18 10 37.8866 80 7.635 125.38 19 10 34.1819 80 9.79

47、75 231.11 20 10 35.1505 80 5.355 203.84 21 10 40.9326 80 -1.0612 91.51 22 10 32.2814 80 7.5175 175.11 13 11 30.0427 80 38.7312 881 14 11 37.0832 80 17.76 391.62 15 11 25.8181 80 28.5888 455.84 16 11 38.7674 80 14.04 535.16 17 11 28.3915 80 23.3838 709.84 18 11 36.4454 80 3.54 277.43 19 11 30.2697 80

48、 50.9388 761.88 20 11 33.4905 80 4.3838 215.96 21 11 39.1459 80 12.2338 356.81 22 11 29.3278 80 12.7575 251.53 13 12 31.8525 80 21.5588 615.16 14 12 38.7302 80 16.4662 400.12 15 12 27.6056 80 19.3937 332.54 16 12 40.0117 80 6.1888 161.42 17 12 29.6922 80 10.6062 206.61 18 12 37.6418 80 6.2475 215.21

49、 19 12 32.7116 80 28.6438 612.54 20 12 35.3361 80 8.7987 246.87 21 12 40.3082 80 8.055 585.07 22 12 31.1973 79 12.4354 396.95 13 13 32.0118 80 32.1938 584.77 14 13 33.9625 80 26.3675 529.74 15 13 23.2768 80 55.2925 537.89 16 13 37.5519 80 13.74 337.35 17 13 26.2755 80 50.1575 715.39 18 13 34.2312 80 20.8038 398.73 19 13 30.5484 80 41.2075 551.97 20 13 29.6731 80 11.17 261.08 21 13 38.193 80 3.3 248.93 23 T1.TR.77-2002 22 13 27.6073 80 25.1938 403.