1、 ATIS-0800025 TEST PLAN FOR EVALUATION OF QUALITY MODELS FOR IPTV SERVICES ATIS is the leading technical planning and standards development organization committed to the rapid development of global, market-driven standards for the information, entertainment and communications industry. More than 250
2、 companies actively formulate standards in ATIS 20 Committees, covering issues including: IPTV, Service Oriented Networks, Home Networking, Energy Efficiency, IP-Based and Wireless Technologies, Quality of Service, Billing and Operational Support. In addition, numerous Incubators, Focus and Explorat
3、ory Groups address emerging industry priorities including “Green”, IP Downloadable Security, Next Generation Carrier Interconnect, IPv6 and Convergence. ATIS is the North American Organizational Partner for the 3rd Generation Partnership Project (3GPP), a member and major U.S. contributor to the Int
4、ernational Telecommunication Union (ITU) Radio and Telecommunications Sectors, and a member of the Inter-American Telecommunication Commission (CITEL). For more information, please visit . Notice of Disclaimer however, test plans are not standardized. Typically, each Standards Development Organizati
5、on (SDO) creates its own test plan that is, very likely, slightly different than another SDOs test plan. However, utilizing a single standardized test plan for IPTV services allows diverse algorithms to be tested in the same manner, and allows cross comparisons. This document also presents the test
6、conditions of the ATIS IIF-defined IPTV system conveniently in a single document. 1.3 Application The following describes a list of uses for this document: The criteria described in this document may be used as input to a test process (e.g., as used in Video Quality Experts Group VQEG, International
7、 Telecommunications Union-Telecommunication Standardization Sector ITU-T Study Group SG-9, SG-12, Video Service Forum VSF, or 3GPP Service and System Aspects SA Working Group WG 4). Note that this document does not specify a test process, only a test plan (see ATIS-0800035, Technical Report on a Val
8、idation Process for IPTV Perceptual Quality Measurements). ATIS-0800025 3 This document provides a reporting format that allows users of the validated models to compare “like”-models (see explanation in section 4). This includes computational complexity as one of the factors. This document can be us
9、ed in conjunction with a “test process” (e.g., ATIS-0800035, Technical Report on a Validation Process for IPTV Perceptual Quality Measurements, which describes a proposed test process for model validation). NOTE This test plan does not negate the standardization of recommended models. The ATIS IIF p
10、rovides this test plan as the industry standard for the evaluation of IPTV quality models. 2 REFERENCES 2.1 Normative References The following standards contain provisions which, through reference in this text, constitute provisions of this ATIS Standard. At the time of publication, the editions ind
11、icated were valid. All standards are subject to revision, and parties to agreements based on this ATIS Standard are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. 1 ANSI S3.6-2004, American National Standard Specification for Audiomet
12、ers, May 2004.12 ATIS-0800002, IPTV Architecture Requirements, May 2006.23 ATIS-0800013, Media Formats and Protocols for IPTV Services, January 2009.24 ATIS-0800018, Linear IPTV Service, January 2009.25 ATIS-0800031, IPTV QoE Requirements, not yet published.26 ITU-R Recommendation BS.1116, Methods f
13、or the subjective assessment of small impairments in audio systems including multichannel sounds systems, October 1997.37 ITU-R Recommendation BT.500-11, Methodology for the subjective assessment of the quality of television pictures, June 2002.38 ITU-T Recommendation G.1050, Network model for evalu
14、ating multimedia transmission performance over Internet protocol, 2007.49 ITU-T Recommendation P.800, Methods for subjective determination of transmission quality, August 1996.410 ITU-T Recommendation P.910, Subjective video quality assessment methods for multimedia applications, April 2008.411 ITU-
15、T Recommendation P.911, Subjective audiovisual quality assessment methods for multimedia applications, September 1999.41This document is available from the American National Standards Institute. 2This document is available from the Alliance for Telecommunications Industry Solutions (ATIS), 1200 G St
16、reet N.W., Suite 500, Washington, DC 20005. 3This document is available from the International Telecommunications Union. 4This document is available from the International Telecommunications Union. ATIS-0800025 4 2.2 Non-Normative References The following references are provided for informative purp
17、oses. At the time of publication, the editions indicated were valid. All standards are subject to revision: 12 ATIS-0800004, A Framework for QoS Metrics and Measurements Supporting IPTV Services, December 2006.213 ATIS-0800005, IPTV Packet Loss Issue Report, November 2007.214 ATIS-0800008, QoS Metri
18、cs for Linear Broadcast IPTV, August 2007.215 ATIS-0800021 trial-use, EPSNR Trial Use Standard, July 2008.2 516 ATIS-0800035, Technical Report on a Validation Process for IPTV Perceptual Quality Measurements, not yet published.217 ITU-R Recommendation BT.1683, Objective perceptual video quality meas
19、urement techniques for standard definition digital broadcast television in the presence of a full reference, February 2004.318 ITU-T Recommendation G.8261/Y.1361, Timing and synchronization aspects in packet networks, 2008.419 ITU-T Recommendation J.144, Objective perceptual video quality measuremen
20、t techniques for digital cable television in the presence of a full reference, March 2004.420 ITU-T Recommendation P.564, Conformance testing for narrowband voice over IP transmission quality assessment models, November 2007.421 ITU-T Recommendation Y.1540, Internet protocol data communication servi
21、ce IP packet transfer and availability performance parameters, November 2007.422 ITU-T Recommendation Y.1541, Network performance objectives for IP-based services, February 2006.423 VQEG, Final report from the video quality experts group on the validation of objective models of video quality assessm
22、ent, March 2000.624 VQEG, Final report from the video quality experts group on the validation of objective models of video quality assessment, Phase II, August 2003.625 VQEG, Multimedia Group Testplan, Draft Version 1.25, March, 2008.626 VQEG, Test Plan for Evaluation of Video Quality Models for Use
23、 with High Definition TV Content, Draft Version 2.7, January 2009.627 M. Spiegel, Theory and problems of statistics, McGraw Hill, 1998. 28 Gersho and R. M. Gray, Vector quantization and signal compression, Kluwer Academic Publisher, SECS159, ISBN 0-7923-9181-0. 29 S. Winkler, Digital Video Quality,
24、John Wiley these both have an absolute frame rate of 25 fps. One example of a variable absolute frame rate is a computer file containing only new frames; in this case the intended frame rate exactly matches the effective frame rate. The content of video frames is not considered when determining inte
25、nded frame rate. Interlacing Interlacing refers to the alternating display of fields comprising the even or odd lines of a frame. The field rate is twice the frame rate. Common in Cathode Ray Tube (CRT) displays. Live Network Conditions Errors imposed upon the digital video bit stream as a result of
26、 live network conditions. Examples of error sources include packet loss due to heavy network traffic, increased delay due to transmission route changes, and multi-path on a broadcast signal. Live network conditions tend to be unpredictable and unrepeatable. Non-expert viewer Viewers occupation (or h
27、obby) does not involve video picture quality or audio-video quality and they are not experienced assessors. These viewers must not have participated in a subjective quality test over a period of six months. They are meant to represent a “typical” consumer of IPTV. Picture Height The height of the di
28、splayed video (expressed as H). Progressive Display format where all lines of a frame are displayed at the same time instant. Common in Liquid Crystal Displays (LCD). ATIS-0800025 6 Term Definition Refresh rate The rate at which the display is updated. Rescaling Signal processing involved in changin
29、g the video resolution. Simulated transmission errors Errors imposed upon the digital video bit stream in a highly controlled environment. Examples include simulated packet loss rates and simulated bit errors. Parameters used to control simulated transmission errors are well defined. Single stimulus
30、 method Method for collecting subjective ratings from a one-time exposure to the PVS. Transmission errors Any error imposed on the video transmission. Example types of errors include simulated transmission errors and live network conditions. Viewing distance Multiple of the displayed display/picture
31、 height. 3.2 Acronyms AAC Advanced Audio Coding AC Alternating Current AC-3 Audio Coding 3 (Dolby Digital) ADSL Asymmetric Digital Subscriber Line ACR Absolute Category Rating AL-FEC Application Layer FEC ARQ Automatic Repeat reQuest ASCII ANSI Standard Code for Information Interchange ATIS Alliance
32、 for Telecommunications Industry Solutions AVC Advanced Video Coding CBR Constant Bit Rate CI Confidence Interval CIF Common Intermediate Format (352 x 288 pixels) CODEC COder-DECoder CPU Central Processing Unit CRT Cathode Ray Tube DCT Discrete Cosine Transform DMOS Difference Mean Opinion Score DM
33、T Discrete Multi Tone DSL Digital Subscriber Line E-AC-3 Enhanced AC-3 EPSNR Estimated Peak Signal to Noise Ratio FEC Forward Error Correction FPS Frames Per Second FR Full Reference GOP Group Of Pictures GUI Graphical User Interface HD High Definition (HDTV: High Definition Television) HE-AAC High-
34、Efficiency AAC ATIS-0800025 7 HRC Hypothetical Reference Circuit HRR Hidden Reference Removal HW Hardware IIF IPTV Interoperability Forum INP Impulse Noise Protection ITF IPTV Terminal Function ITL Independent Test Laboratory ITU International Telecommunication Union ITU-R ITU Radio-communication Se
35、ction ITU-T ITU-Telecommunication Standard Section LCD Liquid Crystal Display MCT Maximum Correlation Threshold MIPS Million Instructions Per Second MOS Mean Opinion Score MOSp Mean Opinion Score, predicted MPEG Motion Pictures Expert Group MSE Mean Squared-Error MTBF Mean Time Between Failures NR N
36、o Reference OEM Original Equipment Manufacturer PC Personal Computer PIP Picture In Picture PLR Packet Loss Ratio also Packet Loss Rate PON Passive Optical Network POS Packet over SONET (Packet over SDH) PS Program Segment PSNR Peak Signal to Noise Ratio PVS Processed Video Sequence PVSa Annotated P
37、VS QCIF Quarter Common Intermediate Format (176 x 144 pixels) QoE Quality of Experience REIN Regular Electric Impulse Noise RMS Root Mean Square RS Reed-Solomon RR Reduced Reference SD Standard Definition (SDTV: Standard Definition Television) SDH Synchronous Digital Hierarchy SFR Source Frame Rate
38、SONET Synchronous Optical Network SRC Source Reference Channel/Circuit STB Set Top Box; see ITF TCO Tjnstemnnens Centralorganisation. Swedish Confederation of Professional Employees ATIS-0800025 8 TE Transmission Error UDP User Datagram Protocol VBR Variable Bit Rate VDSL Very-High-Speed Digital Sub
39、scriber Line VGA Video Graphics Array (640 x 480 pixels) VoIP Voice over Internet Protocol VQEG Video Quality Experts Group VSF Video Services Forum WAN Wide Area Network 3.3 Procedures 3.3.1 Subjective Test Procedure For subjective testing, the following procedures are followed: Creation of or acce
40、ss to a complete library of original media content. Note that access to content may mean that the content can come from one or more external sources. This original content is referred to as Source Reference Channel/Circuit (SRC) material/sequences. Impair the original media content in various manner
41、s. The SRC will be processed through a number of Hypothetical Reference Circuits (HRCs) to generate the Processed Video and audio-video Sequences (PVS). Perform subjective tests with actual non-expert viewers, by showing the PVS and documenting the average of all the viewers ratings i.e., Mean Opini
42、on Scores (MOS) of PVS in Annotated PVS (PVSa). Annotated PVSs comprise the PVS itself and the associated MOS. This process is depicted in Figure 1. Figure 1: Subjective Test Process 3.3.2 Objective Test Procedure For objective testing, the following procedures are followed as depicted in Figure 2:
43、ATIS-0800025 9 Figure 2: Objective Test Process Objective tests are comprised of running the model on the appropriate subset of PVS. The quality predictions (MOSp) of the model are compared with PVSa. (Not shown in picture.) A report is generated. The recommended format is provided in Appendix B. 3.
44、4 Note on Mean Opinion Scores (MOS) The ATIS IIF considers MOS or the measurements that predict MOS one component of QoE, but does not equate MOS to QoE, nor implies that QoE only involves video quality and/or audio quality. From a practical perspective, an objective measurement test that produces a
45、 MOSp with a value on a scale from, (for example) 1 to 5, still needs to be interpreted by an operator of IPTV services. Related to this, while subjective tests involve MOS scores, the ATIS IIF is also interested in more elementary visual and audio impairments such as blockiness and blurriness. The
46、ATIS IIF defined these impairments as QoE indicators in ATIS-0800004 12. These areas are for further study and the subject of ATIS-0800031 5. It is expected that test plans for measuring such impairments will be somewhat different. As progress is made in these areas, it may result in an update to th
47、is specification or the creation of different test plans. 4 REPORTS Specifically, this Test Plan recommends that the following be documented: A. Details of the Subjective Test Process (see Figure 1). The details include: 1. Description of the displays used for the subjective tests. (See section 5.3.
48、) 2. A file that provides results of the subjective test. (See section 5.6.) 3. What processing has been applied to the SRC. (See section 6.6.) 4. Text descriptions of the content used for subjective testing. Providing thumbnails is recommended. (See section 6.7.) 5. Description of all HRCs. (See se
49、ction 7.2.) B. A report with the detailed test results. (See section 7.1.) C. A summary report that describes the performance of the algorithm. (See Appendix B.) D. The identity of the testing organization shall be provided. This will identify whether the results of the test are an exercise in self-validation or formal validation by an Independent Test Laboratory (ITL). ATIS-0800025 10 5 SUBJECTIVE EVALUATION PROCEDURE Beyond the processes and options defined in the r
