ITU-T P 910-2008 Subjective video quality assessment methods for multimedia applications (Study Group 9)《多媒体设备用主观视频质量评估方法》.pdf

1、 International Telecommunication Union ITU-T P.910TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (04/2008) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Audiovisual quality in multimedia services Subjective video quality assessment methods for multimedia app

2、lications Recommendation ITU-T P.910 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Subscribers lines and sets Series P.30 P.300 Tran

3、smission standards Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of quality Series P.80 P.800Audiovisual quality in multimedia servi

4、ces Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications between cars Series P.1100 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P.910 (04/2008) i Recommendation ITU-T P.910 Subjective video quality assessment metho

5、ds for multimedia applications Summary Recommendation ITU-T P.910 describes non-interactive subjective assessment methods for evaluating the one-way overall video quality for multimedia applications such as videoconferencing, storage and retrieval applications, telemedical applications, etc. These m

6、ethods can be used for several different purposes including, but not limited to, selection of algorithms, ranking of audiovisual system performance and evaluation of the quality level during an audiovisual connection. This Recommendation also outlines the characteristics of the source sequences to b

7、e used, like duration, kind of content, number of sequences, etc. Source Recommendation ITU-T P.910 was approved on 6 April 2008 by ITU-T Study Group 9 (2005-2008) under Recommendation ITU-T A.8 procedure. ii Rec. ITU-T P.910 (04/2008) FOREWORD The International Telecommunication Union (ITU) is the

8、United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recom

9、mendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The

10、approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Adminis

11、tration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure e.g. interoperability or applicability) and compli

12、ance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is

13、 required of any party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Int

14、ellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendatio

15、n. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2009 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without

16、 the prior written permission of ITU. Rec. ITU-T P.910 (04/2008) iii CONTENTS Page 1 Scope 1 2 References. 1 3 Terms and definitions . 1 4 Abbreviations 3 5 Source signal. 3 5.1 Recording environment 3 5.2 Recording system . 4 5.3 Scene characteristics. 4 6 Test methods and experimental design. 5 6.

17、1 Absolute category rating (ACR). 6 6.2 Absolute category rating with hidden reference (ACR-HR) 6 6.3 Degradation category rating (DCR) . 7 6.4 Pair comparison method (PC) 8 6.5 Comparison of the methods 9 6.6 Reference conditions 10 6.7 Experimental design . 10 7 Evaluation procedures 10 7.1 Viewin

18、g conditions. 11 7.2 Processing and playback system 11 7.3 Viewers. 12 7.4 Instructions to viewers and training session. 12 8 Statistical analysis and reporting of results 12 Annex A Details related to the characterization of the test sequences . 14 A.1 Sobel filter 14 A.2 How to use SI and TI for t

19、est sequence selection. 15 A.3 Examples 15 Annex B Additional evaluative scales 17 B.1 Rating scales. 17 B.2 Additional rating dimensions . 18 Annex C Simultaneous presentation of sequence pairs 20 C.1 Introduction 20 C.2 Synchronization 20 C.3 Viewing conditions. 20 C.4 Presentations. 20 Annex D Vi

20、deo classes and their attributes 21 iv Rec. ITU-T P.910 (04/2008) Page Appendix I Test sequences . 22 Appendix II Instructions for viewing tests 23 II.1 ACR and ACR-HR. 23 II.2 DCR 23 II.3 PC . 23 Appendix III The simultaneous double stimulus for a continuous evaluation . 25 III.1 Test procedure 25

21、III.2 The training phase 25 III.3 Test protocol features . 25 III.4 Data processing 26 III.5 Reliability of the subjects . 29 Appendix IV Object-based evaluation 31 Appendix V An additional evaluative scale for DCR. 33 Bibliography. 34 Rec. ITU-T P.910 (04/2008) 1 Recommendation ITU-T P.910 Subjecti

22、ve video quality assessment methods for multimedia applications 1 Scope This Recommendation is intended to define non-interactive subjective assessment methods for evaluating the quality of digital video images coded at bit rates specified in classes for TV3, MM4, MM5 and MM6, as specified in Table

23、D.2 for applications such as videotelephony, videoconferencing and storage and retrieval applications. The methods can be used for several different purposes including, but not limited to, selection of algorithms, ranking of video system performance and evaluation of the quality level during a video

24、 connection. 2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subje

25、ct to revision; users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document w

26、ithin this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T J.61 Recommendation ITU-T J.61 (1988), Transmission performance of television circuits designed for use in international connections. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods fo

27、r subjective determination of transmission quality. ITU-T P.930 Recommendation ITU-T P.930 (1996), Principles of a reference impairment system for video. ITU-R BT.500-9 Recommendation ITU-R BT.500-9 (1998), Methodology for the subjective assessment of the quality of television pictures. ITU-R BT.601

28、-4 Recommendation ITU-R BT.601-4 (1994), Encoding parameters of digital television for studios. ITU-R BT.814-1 Recommendation ITU-R BT.814-1 (1994), Specifications and alignment procedures for setting of brightness and contrast of displays. IEC/TR 60268-13 IEC/TR 60268-13 (1998), Sound system equipm

29、ent Part 13: Listening tests on loudspeakers . 3 Terms and definitions This Recommendation defines the following terms: 3.1 gamma: A parameter that describes the discrimination between the grey level steps on a visual display. The relation between the screen luminance and the input signal voltage is

30、 non-linear, with the voltage raised to an exponent gamma. To compensate for this non-linearity, a correction factor that is an inverse function of gamma is generally applied in the camera. Gamma also has an impact on colour rendition. 2 Rec. ITU-T P.910 (04/2008) 3.2 optimization tests: Subjective

31、tests that are typically carried out during either the development or the standardization of a new algorithm or system. The goal of these tests is to evaluate the performance of new tools in order to optimize the algorithms or the systems that are under study. 3.3 qualification tests: Subjective tes

32、ts that are typically carried out in order to compare the performance of commercial systems or equipment. These tests must be carried out under test conditions that are as much representative as possible of the real conditions of use. 3.4 spatial perceptual information (SI): A measure that generally

33、 indicates the amount of spatial detail of a picture. It is usually higher for more spatially complex scenes. It is not meant to be a measure of entropy nor associated with the information defined in communication theory. See clause 5.3.1 for the equation for SI. 3.5 temporal perceptual information

34、(TI): A measure that generally indicates the amount of temporal changes of a video sequence. It is usually higher for high motion sequences. It is not meant to be a measure of entropy nor associated with the information defined in communication theory. See clause 5.3.2 for the equation for TI. 3.6 t

35、ransparency (fidelity): A concept describing the performance of a codec or a system in relation to an ideal transmission system without any degradation. Two types of transparency can be defined: The first type describes how well the processed signal conforms to the input signal, or ideal signal, usi

36、ng a mathematical criterion. If there is no difference, the system is fully transparent. The second type describes how well the processed signal conforms to the input signal, or ideal signal, for a human observer. If no difference can be perceived under any experimental condition, the system is perc

37、eptually transparent. The term “transparent“ without explicit reference to a criterion will be used for systems that are perceptually transparent. 3.7 replication: Repetition of the same circuit condition (with the same source material) for the same subject. 3.8 reliability of a subjective test: a)

38、intra-individual (“within subject“) reliability refers to the agreement between a certain subjects repeated ratings of the same test condition; b) inter-individual (“between subjects“) reliability refers to the agreement between different subjects ratings of the same test condition. 3.9 validity of

39、a subjective test: Agreement between the mean value of ratings obtained in a test and the true value which the test purports to measure. 3.10 reference conditions: Dummy conditions added to the test conditions in order to anchor the evaluations coming from different experiments. 3.11 explicit refere

40、nce (source reference): The condition used by the assessors as reference to express their opinion, when the DCR method is used. This reference is displayed first within each pair of sequences. Usually the format of the explicit reference is the format used at the input of the codecs under test (e.g.

41、, ITU-R BT.601-4, CIF, QCIF, SIF, etc.). In the body of this Recommendation, the words “explicit“ and “source“ will be omitted whenever the context will make clear the meaning of “reference“. 3.12 implicit reference: The condition used by the assessors as reference to express their opinion on the te

42、st material, when the ACR method is used. If the implicit reference is suggested by the experimenter, it must be well known to all the assessors (e.g., conventional TV systems, reality). Rec. ITU-T P.910 (04/2008) 3 4 Abbreviations This Recommendation uses the following abbreviations: ACR Absolute C

43、ategory Rating ACR-HR Absolute Category Rating with Hidden Reference CCD Charge Coupled Device CI Confidence Interval CIF Common Intermediate Format NOTE A picture format defined in b-ITU-T H.261 for video phone: 352 lines 288 pixels. CRT Cathode Ray Tube DCR Degradation Category Rating DV Different

44、ial viewer %GOB Percent of Good or Better (proportion of Good and Excellent) LCD Liquid Crystal Display MOS Mean Opinion Score PC Pair Comparison %POW Percent of Poor or Worse (proportion of Poor and Bad votes) PVS Processed Video Sequence QCIF Quarter CIF NOTE A picture format defined in b-ITU-T H.

45、261 for video phone: 176 lines 144 pixels. S/N Signal-to-Noise ratio SI Spatial Information SIF Standard Intermediate Format NOTE A picture format defined in b-ISO/IEC 11172 (MPEG-1): 352 lines 288 pixels 25 frames/s and 352 lines 240 pixels 30 frames/s. SP Simultaneous Presentation std Standard Dev

46、iation TI Temporal Information VTR Video Tape Recorder 5 Source signal In order to control the characteristics of the source signal, the test sequences should be defined according to the goal of the test and recorded on a digital storage system. When the experimenter is interested in comparing resul

47、ts from different laboratories, it is necessary to use a common set of source sequences to eliminate a further source of variation. 5.1 Recording environment Lighting source(s) (bulbs or fluorescent lamps) can be placed above or on the side of the camera. When placing the lights, recognize that over

48、head is more typical of office lighting, and should be used with scenes that portray the business environment. Studio lights and other non-typical sources should be avoided. 4 Rec. ITU-T P.910 (04/2008) The lighting conditions of the room in the field of view could vary from 100 lux to about 10 000

49、lux for indoor use. The variation (AC frequency) of the light (fluorescent lighting) must be taken into account because this may cause a flicker in the recorded video sequence. Lighting conditions, wall colours, surface reflectance, etc., should be carefully controlled and reported. 5.2 Recording system 5.2.1 Camera Picture sequences should be recorded by a high quality CCD camera. The signal-to-noise ratio of the input video signal can strongly affect the performance of the codec. To define the video input, the following point