ITU-T P 911-1998 Subjective Audiovisual Quality Assessment Methods for Multimedia Applications - Series P Telephone Transmission Quality Telephone Installations Local Line NetworksS.pdf

1、INTERNATIONAL TELECOMMUNICAT ITU-T TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU ION UNION P.91 I (I 2/98) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Auc iovisual quality in multimedia services Subjective audiovisual quality assessment methods for multim

2、edia applications ITU-T Recommendation P.911 (Previously CCITT Recommendation) COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling Services. STDOITU-T RECMN POSIL-ENGL 1998 48b259L Ob574b0 387 9 ITU-T P-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION

3、 QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Subscribers lines and sets Transmission standards Objective measuring apparatus Objective electro-acoustical measurements Measurements related to speec

4、h loudness Methods for objective and subjective assessment of quality Series P. 1 O Series P.30 P.300 Series P.40 Series PS0 P.500 Series P.60 Series P.70 Series P.80 P.800 Series P.900 I For further details, please refer to ITU-T List of Recommendations. COPYRIGHT International Telecommunications U

5、nion/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T RECOMMENDATION P.911 SUBJECTIVE AUDIOVISUAL QUALITY ASSESSMENT METHODS FOR MULTIMEDIA APPLICATIONS Summary This Recommendation describes non-interactive subjective assessment methods for evaluating the one-way overall audiovis

6、ual quality for multimedia applications such as videoconferencing, storage and retrieval applications, telemedical applications, etc. These methods can be used for several different purposes including, but not limited to, selection of algorithms, ranking of audiovisual system performance and evaluat

7、ion of the quality level during an audiovisual connection. When interactive aspects are to be assessed, conversation test methods described in Recommendation P.920 should be used. This Recommendation also outlines the characteristics of the source sequences to be used, like duration, kind of content

8、, number of sequences, etc. Finally it provides indications about relation between audio, video and audiovisual quality, as they were derived from results of tests carried out independently in different laboratories. Source ITU-T Recommendation P.911 was prepared by ITU-T Study Group 12 (1997-2000)

9、and was approved under the WTSC Resolution No. 1 procedure on the 3rd of December 1998. Recommendation P.911 (12/98) I COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesSTD-ITU-T RECMN P-qll-ENGL FOREWORD ITU (International Telecommunicat

10、ion Union) is the United Nations Specialized Agency in the field of telecommunications. The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of the ITU. The ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a

11、 view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Conference (WTSC), which meets every four years, establishes the topics for study by the ITU-T Study Groups which, in their turn, produce Recommendations on these topics. The approval of Recom

12、mendations by the Members of the ITU-T is covered by the procedure laid down in WTSC Resolution No. l. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with IS0 and IEC. NOTE In this Recommendation the term recogn

13、ized operating agency (ROA) includes any individual, company, corporation or governmental organization that operates a public correspondence service. The terms Administration, ROA and public correspondence are defined in the Constitution of the ITU (Geneva, 1992). INTELLECTUAL PROPERTY RIGHTS The IT

14、U draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. The ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU

15、members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, the ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementors are cautioned that thi

16、s may not represent the latest information and are therefore strongly urged to consult the TSB patent database. O ITU 1999 All rights reserved. No part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, witho

17、ut permission in writing from the ITU. 11 Recommendation P.911 (12/98) COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling Services1 2 3 4 5 6 6.1 6.2 6.3 6.4 6.5 6.6 6.7 7 7.1 7.2 7.3 7.4 8 Scope References Terms and definitions . Abbreviations .

18、Source signal Test methods and experimental design . Absolute Category Rating (ACR) Degradation Category Rating (DCR) . Pair Comparison method (PC) Single Stimulus Continuous Quality Evaluation (SSCQE) Comparison of the methods Reference conditions Experimental design . Evaluation procedures Viewing

19、 and listening conditions . Processing and playback system Subjects . Instructions to subjects and training session Statistical analysis and reporting of results Annex A . Details related to the characterization of the test sequences . Annex B . Video and audio classes and their attributes B . 1 Vid

20、eo classes and their attributes . B.2 Audio classes and their attributes . Annex C . Discussion about the Relationship between audio, video and audiovisual quality Appendix I . Test audiovisual sequences Appendix II - Instructions for audiovisual tests II . 1 ACR 11.2 DCR 11.3 PC . Appendix III . Bi

21、bliography . Recommendation P.911 (12/98) Page 1 1 2 3 4 5 5 6 7 8 8 9 9 10 10 11 11 12 12 13 14 14 15 17 18 18 19 19 19 20 . III COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesSTDvITU-T RECMN P-SLZ-ENGL 1998 4862593 0b574bq T22 II Rec

22、ommendation P.911 SUBJECTIVE AUDIOVISUAL QUALITY ASSESSMENT METHODS FOR MULTIMEDIA APPLICATIONS (Geneva, 1998) 1 Scope This Recommendation is intended to define non-interactive subjective assessment methods for evaluating the one-way overall audiovisual quality for bit rates specified in classes TV

23、3, MM 4, MM 5 and MM 6, as specified in Tables B.2 and B.4, for applications such as videoconferencing, storage and retrieval applications, telemedical applications, etc. The methods can be used for several different purposes including, but not limited to, selection of algorithms, ranking of audiovi

24、sual system performance and evaluation of the quality level during an audiovisual connection. When interactive aspects are to be assessed, Recommendation P.920 should be used. 2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this te

25、xt, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; all users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edit

26、ion of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. l ITU-T Recommendation P.930 (1996), Principles of a reference impairment system for video. 2 ITU-T Recommendation P.920 (1996), Interactive test methods for audi

27、ovisual communications. 3 ITU-R Recommendation BT.601-5 (1995), Studio encoding parameters of digital television for standard 4:3 and wide-screen I6:9 aspect ratios. 4 ITU-R Recommendation BT.500-8 (1998), Methodology for the subjective assessment of the Quality of Television Pictures. 5 IEC/TR3 Pub

28、lication 60268-13 (1998), Sound system equipment - Part 13: Listening tests on loudspeakers. 6 ITU-T Recommendation P.800 (1 996), Methods for subjective determination of transmission quality. 7 ITU-R Recommendation BT.8 14-1 (1993), Specifications and alignment procedures for setting of brightness

29、and contrast of displays. 8 ITU-R Recommendation BT. 1 128-2 (1 997), Subjective assessment of conventional television systems. 191 CCITT Recommendation J.61 (1990), Transmission performance of television circuits designed for use in international connections. l O ITU-T Recommendation P.810 (1996),

30、Modulated Noise Reference Unit (MNRU). Recommendation P.911 (12/98) 1 COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesITU-T Recommendation P.910 (1996), Subjective video quality assessment methods for multimedia applications. CCITT Reco

31、mmendation G.722 (1988), 7 kHz audio-coding within 64 kbit/s. CCITT Recommendation G.711 (1988), Pulse Code Modulation (PCM) of voice Pequencies. CCITT Recommendation G.728 (1992), Coding of speech at 16 kbit/s using low-delay code excited linear prediction. ITU-T Recommendation G. 1 14 (1 996), One

32、-way transmission time. ITU-R Recommendation BS.775-1 (1 993), Multi-channel stereophonic sound system with and without accompanyingpicture. IEC Publication 6065 1 (1979), Sound level meters. ITU-T Recommendation P.93 1 (1998), Subjective audiovisual quality assessment methods for multimedia applica

33、tions. Terms and definitions This Recommendation defines the following terms: 3.1 explicit reference (source reference): The condition used by the assessors as reference to express their opinion, when the DCR method is used. This reference is displayed first within each pair of sequences. Usually th

34、e format of the explicit reference is the format used at the input of the codecs under test (e.g.: ITU-R BT.601, CIF, QCIF, SIF, etc.). In the body of this Recommendation, the words “explicit“ and “source“ will be omitted whenever the context will make clear the meaning of “reference“. 3.2 gamma: Th

35、e relationship between the screen luminance and the input signal voltage is non- linear, with the voltage raised to an exponent gamma. To compensate for this non-linearity, a correction factor that is an inverse function of gamma is generally applied in the camera. Gamma also has an impact on colour

36、 rendition. 3.3 optimization tests: Subjective tests that are typically carried out during either the development or the standardization of a new algorithm or system. The goal of these tests is to evaluate the performance of new tools in order to optimize the algorithms or the systems that are under

37、 study. 3.4 qualification tests: Subjective tests that are typically carried out in order to compare the performance of commercial systems or equipment. These tests must be carried out under test conditions that are as representative as possible of the real conditions of use. 3.5 reference condition

38、s: Dummy conditions added to the test conditions in order to better compare the evaluations coming from different experiments. 3.6 reliability of a subjective test: a) intra-individual (“within subject“) reliability refers to the agreement between a certain b) inter-individual (“between subjects“) r

39、eliability refers to the agreement between different subjects repeated ratings of the same test condition; subjects ratings of the same test condition. 3.7 replication: Repetition of the same circuit condition (with the same source material) for the same subject. 2 Recommendation P.911 (12/98) COPYR

40、IGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling Services3.8 Spatial perceptual Information (SI): A measure that generally indicates the amount of spatial detail of a picture. It is usually higher for more spatially complex scenes. It is not meant to

41、 be a measure of entropy nor associated with information as defined in communication theory. The Spatial perceptual Information, SI, is based on the Sobel filter. Each video frame (luminance plane) at time n (F,) is first filtered with the Sobel filter Sobel(F,). The standard deviation over the pixe

42、ls (stdspace) in each Sobel-filtered frame is then computed. This operation is repeated for each frame in the video sequence and results in a time series of spatial information of the scene. The maximum value in the time series (mqime) is chosen to represent the spatial information content of the sc

43、ene. This process can be represented in equation form as: SI = matime Stdspace Sobel(Fn)I 1 3.9 Temporal perceptual Information (TI): A measure that generally indicates the amount of temporal changes of a video sequence. It is usually higher for high motion sequences. It is not meant to be a measure

44、 of entropy nor associated with information as defined in communication theory. The measure of Temporal Information, TI, is computed as the maximum over time (maxti,) of the standard deviation over space (stdspace) of M,(i,j) over all i and j. TI = maxtime Stdspace P“,jl where M,(ij) is the differen

45、ce between pixels at the same position in the frame, but belonging to two subsequent frames, that is: M,(i,j) = F,(ij) - F,-,W where F,(i,j) is the pixel at the ith row and jt“ column of nth frame in time. 3.10 transparency (fidelity): A concept describing the performance of a codec or a system in r

46、elation to an ideal transmission system without any degradation. Two types of transparency can be defined: The first type describes how well the processed signal conforms to the input signal, or ideal signal, using a mathematical distance criterion. If there is no difference (i.e. distance = O), the

47、 system is fully transparent. The second type describes how well the processed signal conforms to the input signal, or ideal signal, for a human observer. If no difference can be perceived under any experimental condition, the system is perceptually transparent. The term transparent without explicit

48、 reference to a criterion will be used for systems that are perceptually transparent. 3.11 validity of a subjective test: Agreement between the mean value of ratings obtained in a test and the true value which the test purports to measure. 4 Abbreviations This Recommendation uses the following abbre

49、viations: ACR Absolute Category Rating CCD Charge Coupled Device CI Confidence Interval CIF Common Intermediate Format (picture format defined in Recommendation H.261 for videophone: 352 lines x 288 pixels) CRT Cathode Ray Tube DCR Degradation Category Rating %GOB Percent of Good or Better (proportion of Good and Excellent) Recommendation P.911 (12/98) 3 COPYRIGHT International Telecommunications Union/ITU TelecommunicationsLicensed by Information Handling ServicesLCD MOS PC %POW QCIF I SI SIF S/N SP std TI VTR Liquid Crystal Display Mean Opinion Score