ITU-R BT 1866-2010 Objective perpetual video quality measurement techniques for broadcasting applications using low definition television in the presence of a full reference signal.pdf

1、 Recommendation ITU-R BT.1866(03/2010)Objective perpetual video quality measurement techniques for broadcasting applications using low definition television in the presence of a full reference signalBT SeriesBroadcasting service(television)ii Rec. ITU-R BT.1866 Foreword The role of the Radiocommunic

2、ation Sector is to ensure the rational, equitable, efficient and economical use of the radio-frequency spectrum by all radiocommunication services, including satellite services, and carry out studies without limit of frequency range on the basis of which Recommendations are adopted. The regulatory a

3、nd policy functions of the Radiocommunication Sector are performed by World and Regional Radiocommunication Conferences and Radiocommunication Assemblies supported by Study Groups. Policy on Intellectual Property Right (IPR) ITU-R policy on IPR is described in the Common Patent Policy for ITU-T/ITU-

4、R/ISO/IEC referenced in Annex 1 of Resolution ITU-R 1. Forms to be used for the submission of patent statements and licensing declarations by patent holders are available from http:/www.itu.int/ITU-R/go/patents/en where the Guidelines for Implementation of the Common Patent Policy for ITU-T/ITU-R/IS

5、O/IEC and the ITU-R patent information database can also be found. Series of ITU-R Recommendations (Also available online at http:/www.itu.int/publ/R-REC/en) Series Title BO Satellite delivery BR Recording for production, archival and play-out; film for television BS Broadcasting service (sound) BT

6、Broadcasting service (television) F Fixed service M Mobile, radiodetermination, amateur and related satellite services P Radiowave propagation RA Radio astronomy RS Remote sensing systems S Fixed-satellite service SA Space applications and meteorology SF Frequency sharing and coordination between fi

7、xed-satellite and fixed service systems SM Spectrum management SNG Satellite news gathering TF Time signals and frequency standards emissions V Vocabulary and related subjects Note: This ITU-R Recommendation was approved in English under the procedure detailed in Resolution ITU-R 1. Electronic Publi

8、cation Geneva, 2010 ITU 2010 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without written permission of ITU. Rec. ITU-R BT.1866 1 RECOMMENDATION ITU-R BT.1866 Objective perceptual video quality measurement techniques for broadcasting applications using

9、 low definition television*in the presence of a full reference signal (2010) Scope This Recommendation specifies methods for estimating the perceived video quality of broadcasting applications using low definition television (LDTV) when a full reference signal is available. The ITU Radiocommunicatio

10、n Assembly, considering a) that the ability to automatically measure the quality of broadcast video has long been recognized as a valuable asset to the industry; b) that Recommendation ITU-R BT.1683 describes objective methods for measuring the perceived video quality of standard definition digital

11、broadcast television in the presence of a full reference; c) that Recommendation ITU-R BT.1833 describes multimedia systems for the broadcasting of multimedia and data applications for mobile reception by handheld receivers; d) that low definition television (LDTV) is becoming widely used in the bro

12、adcasting of multimedia and data applications for mobile reception; e) that ITU-T Recommendation J.2471specifies objective measurement techniques of perceptual video quality applicable to LDTV applications in the presence of a full reference; f) that objective measurement of the perceived video qual

13、ity of LDTV may complement subjective assessment methods, recognizing a) that use of LDTV is mainly intended for viewing on small screens such as those available on handheld and mobile receivers, recommends 1 that the guidelines, scope, and limitations given in Annex 1 should be used in the applicat

14、ion of the objective video quality measurement models identified in recommends 2; 2 that the objective perceptual video quality measurement models given in ITU-T Recommendation J.247 should be used for broadcasting applications using LDTV when a full reference signal is available. NOTE 1 Summaries o

15、f the measurement models are given in Annexes 2 to 5 for information. For more detail see ITU-T Recommendation J.247. *Low definition television (LDTV) refers to video resolutions having lesser number of pixels than the ones defined in Recommendation ITU-R BT.601. A pertinent ITU-R Recommendation on

16、 LDTV is under consideration. 1ITU-T Recommendation J.247 is available at . 2 Rec. ITU-R BT.1866 Annex 1 1 Introduction This Recommendation specifies methods for estimating the perceived video quality of broadcasting applications using LDTV when a full reference signal is available. The estimation m

17、ethods in this Recommendation are applicable to: codec evaluation, specification, and acceptance testing; potentially real-time, in-service quality monitoring at the source; remote destination quality monitoring when a copy of the source is available; quality measurement for monitoring of a storage

18、or transmission system that uses either a single pass or concatenation of video compression and decompression techniques; lab testing of video systems. The full reference measurement method can be used when the unimpaired reference video signal is readily available at the measurement point, as may b

19、e the case of measurements on individual equipment or a chain of processes in a laboratory or in a closed environment. The estimation methods are based on processing low definition video in VGA, CIF, and QCIF resolution. The validation test material contained both multiple coding degradations and va

20、rious transmission error conditions (e.g. bit errors and dropped packets). In a case where coding distortions are considered in the video signals, the encoder can use various compression methods (e.g. MPEG-2, H.264, etc.). The models described in this Recommendation may be used to monitor the qualit

21、y of deployed networks to ensure their operational readiness. The visual effects of the degradations may include spatial as well as temporal degradations (e.g. frame repeats, frame skips, and frame rate reduction). The models in this Recommendation can also be used for lab testing of video systems.

22、This Recommendation is deemed appropriate for services delivered at 4 Mbit/s or less presented on mobile receivers. The following conditions were used in the validation test for each resolution and found to be suitable: QCIF (quarter common intermediate format (176 144 pixels): 16 to 320 kbit/s. CIF

23、 (common intermediate format (352 288 pixels): 64 kbit/s to 2 Mbit/s. VGA (video graphics array (640 480 pixels): 128 kbit/s to 4 Mbit/s. TABLE 1 Factors used in evaluation of models Test factors Transmission errors with packet loss Video resolution QCIF, CIF and VGA Video bitrates QCIF: 16 to 320 k

24、bit/s CIF: 64 kbit/s to 2 Mbit/s VGA: 128 kbit/s to 4 Mbit/s Temporal errors (pausing with skipping) of maximum 2 s Video frame rates from 5 to 30 fps Rec. ITU-R BT.1866 3 TABLE 1 (end) Coding schemes H.264/AVC (MPEG-4 Part 10), MPEG-4 Part 2, and three other proprietary coding schemes. (See Note 1

25、below.) NOTE 1 The validation testing of models included video sequences encoded using 15 different video codecs. The five codecs listed in Table 1 were most commonly applied to encode test sequences, and any recommended models may be considered appropriate for evaluating these codecs. In addition t

26、o these five codecs, a smaller proportion of test sequences were created using the following codecs: H.261, H.263, H.263+2, JPEG-2000, MPEG-1, MPEG-2, H.264 SVC, and other proprietary systems. Note that some of these codecs were used only for CIF and QCIF resolutions because they are expected to be

27、used in the field mostly for these resolutions. Before applying a model to sequences encoded using one of these codecs, the user should carefully examine its predictive performance to determine whether the model reaches acceptable predictive performance. 2 Application Applications for the estimation

28、 models described in this Recommendation include but are not limited to: 1 codec evaluation, specification, and acceptance testing; 2 potentially real-time, in-service quality monitoring at the source; 3 remote destination quality monitoring when a copy of the source is available; 4 quality measurem

29、ent for monitoring of a storage or transmission system that uses either a single pass or concatenation of video compression and decompression techniques; and 5 lab testing of video systems. 3 Model usage This Recommendation includes the objective computational models shown in Table 2. An overview of

30、 the model performance is shown in Table 3. Further information is given in Appendix 1. TABLE 2 Objective computational models Model id Proponent Country Annex A NTT Japan 2 B OPTICOM Germany 3 C Psytechnics United Kingdom 4 D Yonsei University Korea (Rep. of) 5 All four models significantly outperf

31、orm the peak signal to noise ratio (PSNR). 2H.263+ is a particular configuration of H.263 (1998). 4 Rec. ITU-R BT.1866 Models B and C tend to perform slightly better than Models A and D in some resolutions. Models B and C usually produce statistically equivalent results. For QCIF, Model A is often s

32、tatistically equivalent to Models B and C. For VGA, Model D is typically statistically equivalent to Models B and C. The tables below provide an overview of the models performances. Although all four models can be used to adequately meet different industries needs, for VGA, it is highly advised that

33、 Models B, C, or D be used to obtain slightly better performance in most cases. For the same reason, it is highly advised that Models B or C be used for CIF and that Models A, B, or C be used for QCIF. Model B shows the best overall minimum correlation. The minimum correlation coefficients of Models

34、 B, A, D and C are 0.68, 0.60, 0.59 and 0.57, respectively. Model C obtained the highest number of occurrences of being in the top performing group. The total number of occurrences in the top group was 37 for Model C, 34 for Model B, 25 for Model A, and 24 for Model D. TABLE 3 Model performance over

35、view VGA Model A Model B Model C Model D PSNR Avg. Correlation 0.786 0.825 0.822 0.805 0.713 Min. Correlation 0.598 0.685 0.565 0.612 0.499 Occurrences at Rank 1 8 10 11 10 3 Ranking analysis Second Best Best Best CIF Model A Model B Model C Model D PSNR Avg. Correlation 0.777 0.808 0.836 0.785 0.65

36、6 Min. Correlation 0.675 0.695 0.769 0.712 0.440 Occurrences at Rank 1 8 13 14 10 0 Ranking analysis Second Best Best Second QCIF Model A Model B Model C Model D PSNR Avg. Correlation 0.819 0.841 0.830 0.756 0.662 Min. Correlation 0.711 0.724 0.664 0.587 0.540 Occurrences at Rank 1 9 11 12 4 1 Ranki

37、ng analysis Best Best Best Second 4 Limitations The estimation models described in this Recommendation cannot be used to fully replace subjective testing. Correlation values between two carefully designed and executed subjective tests (in two different laboratories) normally fall between 0.95 to 0.9

38、8. The models in this Recommendation were validated by measuring video that exhibits frame freezes of up to 2 s. Rec. ITU-R BT.1866 5 The models in this Recommendation were not validated for measuring video that has a steadily increasing delay (e.g. video that does not discard missing frames after a

39、 frame freeze). Note that in the case of new coding and transmission technologies producing artefacts that were not included in this evaluation, the objective models may produce erroneous results. Here a subjective evaluation is required. Appendix 1 to Annex 1 Findings of Video Quality Experts Group

40、 Studies of perceptual video quality measurements are conducted in an informal group, called the Video Quality Experts Group (VQEG), which reports to ITU-T Study Groups 9 and 12 and Radiocommunication Study Group 6. The recently completed Multimedia Phase I test of VQEG assessed the performance of p

41、roposed full reference perceptual video quality measurement algorithms for QCIF, CIF, and VGA formats. Based on present evidence, four methods can be recommended by ITU-R at this time. These are: Model A (Annex 2) VQEG Proponent NTT, Japan Model B (Annex 3) VQEG Proponent OPTICOM, Germany Model C (A

42、nnex 4) VQEG Proponent Psytechnics, UK Model D (Annex 5) VQEG Proponent Yonsei University, Korea (Republic of). The technical descriptions of these models can be found in Annexes 2 through 5 respectively. Note that the ordering of Annexes is purely arbitrary and provides no indication of quality pre

43、diction performance. Table 4 provides details of the models performances in the VQEG Multimedia Phase I test. TABLE 4 a) VGA resolution: Models performances in VQEG Multimedia Phase I test Averages over 14 subjective tests Metric Model A Model B Model C Model D PSNR(1)Annex 2 3 4 5 Pearson correlati

44、on 0.786 0.825 0.822 0.805 0.713 RMS error 0.621 0.571 0.566 0.593 0.714 Outlier ration 0.523 0.502 0.524 0.542 0.615 6 Rec. ITU-R BT.1866 TABLE 4 (end) b) CIF resolution: Models performances in VQEG Multimedia Phase I test Averages over 14 subjective tests Metric Model A Model B Model C Model D PSN

45、R(1)Annex 2 3 4 5 Pearson correlation 0.777 0.808 0.836 0.785 0.656 RMS error 0.604 0.562 0.526 0.594 0.720 Outlier ration 0.538 0.513 0.507 0.522 0.632 c) QCIF resolution: Models performances in VQEG Multimedia Phase I test Averages over 14 subjective tests Metric Model A Model B Model C Model D PS

46、NR(1)Annex 2 3 4 5 Pearson correlation 0.819 0.841 0.830 0.756 0.662 RMS error 0.551 0.516 0.517 0.617 0.721 Outlier ration 0.497 0.461 0.458 0.523 0.596 (1)The PSNR values reported here are taken from the VQEG Multimedia Phase I final report (see: http:/www.its.bldrdoc.gov/vqeg/projects/multimedia/

47、). These values were calculated by NTIA/ITS. Based on each metric, each FR VGA model was in the group of top performing models the following number of times: Statistic Model A Model B Model C Model D PSNR Correlation 8 10 11 10 3 RMSE(1)4 8 10 6 0 Outlier ratio 9 11 12 8 4 (1)RMSE: root mean square

48、error. Based on each metric, each FR CIF model was in the group of top performing models the following number of times: Statistic Model A Model B Model C Model D PSNR Correlation 8 13 14 10 0 RMSE(1)6 10 13 9 Outlier ratio 10 13 12 11 1 (1)RMSE: root mean square error. Rec. ITU-R BT.1866 7 Based on

49、each metric, each FR QCIF model was in the group of top performing models the following number of times: Statistic Model A Model B Model C Model D PSNR Correlation 9 11 12 4 1 RMSE(1)7 10 11 2 Outlier ratio 10 11 12 8 4 (1)RMSE: root mean square error. NOTE 1 As a general guideline, small differences in these totals do not indicate an overall difference in performance. Secondary analysis The secondary analysis averages all video sequences associated with each video system (or condition) and thus reflects how well the model tracks th