1、 ETSI TR 101 565 V1.1.1 (2011-09) Speech and multimedia Transmission Quality (STQ); Guidelines and results of video quality analysis in the context of Benchmark and Plugtests for multiplay services Technical Report ETSI ETSI TR 101 565 V1.1.1 (2011-09) 2Reference DTR/STQ-00185 Keywords analysis, qua
2、lity, video ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N 348 623 562 00017 - NAF 742 C Association but non lucratif enregistre la Sous-Prfecture de Grasse (06) N 7803/88 Important notice Individual copies of the present do
3、cument can be downloaded from: http:/www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dis
4、pute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other
5、ETSI documents is available at http:/portal.etsi.org/tb/status/status.asp If you find errors in the present document, please send your comment to one of the following services: http:/portal.etsi.org/chaircor/ETSI_support.asp Copyright Notification No part may be reproduced except as authorized by wr
6、itten permission. The copyright and the foregoing restriction extend to reproduction in all media. European Telecommunications Standards Institute 2011. All rights reserved. DECTTM, PLUGTESTSTM, UMTSTMand the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members. 3GPPTM and LTE
7、 are Trade Marks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. GSM and the GSM logo are Trade Marks registered and owned by the GSM Association. ETSI ETSI TR 101 565 V1.1.1 (2011-09) 3Contents Intellectual Property Rights 4g3Foreword . 4g31 Scope 5g32 Ref
8、erences 5g32.1 Normative references . 5g32.2 Informative references 5g33 Definitions and abbreviations . 6g33.1 Definitions 6g33.2 Abbreviations . 6g34 Context . 6g35 Platform presentation . 7g36 Presentation of test conditions 7g36.1 Indicator description . 8g36.1.1 “Black Screen“ Occurrences . 8g3
9、6.1.2 Blockiness Occurrences 9g36.1.3 Frozen Picture Occurrences 9g36.2 Result presentations 9g36.2.1 “Sector“ presentation 10g36.2.2 Pie diagram presentation . 10g3Annex A: Video quality performance evaluated on different IPTV services - Overview of results obtained at end 2010 . 12g3A.1 Platform p
10、resentation . 12g3A.2 Description of the methodology . 12g3A.2.1 General Principles 12g3A.2.2 Technical criteria 14g3A.2.3 Scheduling of analyses . 15g3A.3 Results obtained by IPTV service of ISP1 . 15g3A.4 Results obtained by IPTV service of ISP2 . 17g3A.5 Results obtained by IPTV service of ISP3 .
11、 18g3A.6 Results obtained by IPTV service of ISP4 . 20g3A.7 Results obtained by IPTV service of ISP5 . 21g3A.8 Results obtained by TV service of DVB-T 22g3A.9 Result discussion 23g3History 24g3ETSI ETSI TR 101 565 V1.1.1 (2011-09) 4Intellectual Property Rights IPRs essential or potentially essential
12、 to the present document may have been declared to ETSI. The information pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found in ETSI SR 000 314: “Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified t
13、o ETSI in respect of ETSI standards“, which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web server (http:/ipr.etsi.org). Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the e
14、xistence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document. Foreword This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality (STQ).
15、 ETSI ETSI TR 101 565 V1.1.1 (2011-09) 51 Scope The present document presents the results obtained on technological platform where Triple Play offers are available. These results concern the quality evaluation of IPTV video streams produce by the offers available on the platform. The determinate ind
16、icators are presented in the main part of the document. The results were obtained during a specific test campaign for SD streams analysis. Note that determinate indicators are presented in ES 202 765-4 i.1. So the present document represents an implementation report for some metrics and associated m
17、ethods defined in ES 202 765-4 i.1. The main part of the present document presents the performed indicators and charts used for results presentation. Annex A presents the methodology implemented in a first series of tests and the results. 2 References References are either specific (identified by da
18、te of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies. Referenced documents which are not found to be publicly
19、available in the expected location might be found at http:/docbox.etsi.org/Reference. NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee their long term validity. 2.1 Normative references The following referenced documents are necessary for
20、 the application of the present document. Not applicable. 2.2 Informative references The following referenced documents are not necessary for the application of the present document but they assist the user with regard to a particular subject area. i.1 ETSI ES 202 765-4: “Speech and multimedia Trans
21、mission Quality (STQ); QoS and network performance metrics and measurement methods; Part 4: Indicators for supervision of Multiplay services“. i.2 ITU-T Recommendation P.505: “One-view visualization of speech quality measurement results“. i.3 ITU-T Recommendation J.247: “Objective perceptual multime
22、dia video quality measurement in the presence of a full reference“. i.4 ITU-T Recommendation J.144: “Objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference“. ETSI ETSI TR 101 565 V1.1.1 (2011-09) 63 Definitions and abbreviations 3.1
23、 Definitions For the purposes of the present document, the following terms and definitions apply: platform: premise installed in residential environment where the accesses to different Multi Play offers proposed by ISP on the same country are made available NOTE: This platform is generally installed
24、 in the centre of a city. 3.2 Abbreviations For the purposes of the present document, the following abbreviations apply: ADSL Asymmetric Digital Subscriber Line DVB-T Digital Video Broadcasting - Terrestrial HGW Home GateWay NOTE: Referenced also as Residential Gateway. IP Internet Protocol IPTV IP
25、TeleVisionNOTE: System where a digital television service is delivered using Internet Protocol. ISP Internet Service Provider ITU-T International Telecommunication Union - Telecommunication standardization sector MOS Mean Opinion Score VoIP Voice over Internet Protocol 4 Context The deployment of mu
26、ltiplay offers is increasing, so it is important to measure quality performances of services proposed via these multiplay offers. Concerning performance evaluation of multiplay offers, ES 202 765-4 i.1 lists indicators and presents methodologies for quality characterization in a context of end-user.
27、 The present document gives practical requirements of use in the context of service verification and benchmark. Multiplay offers developed on IP technology give access to IPTV services. These IPTV services develop in an important way in the residential context. But contrary to the VoIP services, per
28、formances of IPTV services of are not really evaluated. Because there is a necessity of having a performance overview of the service brought to the users, the actors of the domain (operators, benchmakers) began to perform evaluations of IPTV services. In these conditions, the present document presen
29、ts a benchmark of the video quality evaluated on different TV services. The present document is intended to provide an overview of the performance of IPTV offers deployed in France (and used by customers). The core of the report presents determined metrics and results presentation. The results prese
30、nted in annex A of the present document, concern video quality of IPTV service associated to Triple Play offers implemented on the technological platform installed in a city in France. The interest of these results is to give an overview of video quality of deployed IPTV services compared to a kind
31、of reference which is the video quality of DVB-T (Digital Video Broadcasting - Terrestrial) service. ETSI ETSI TR 101 565 V1.1.1 (2011-09) 75 Platform presentation The platform is a premise installed in residential environment and in which there are the accesses to different Multi Play offers propos
32、ed by ISP on the same country. The technological platform used for performing video quality assessment is installed in a premise in the city centre of a town counting more than 210 000 residents. During the second semester of 2010, this platform was characterized by: Implementation of 6 offers conce
33、rning 6 different ISPs. Each offer proposes 3 services: Internet access, VoIP and IPTV. Access to services is obtained through a Home GateWay (HGW). Access to the network can be ADSL or cable type depending on the ISP. Distance between HGW and first digital equipment is about 350 meters (Length of t
34、he ADSL line). A DVB-T (Digital Video Broadcasting - Terrestrial) access is also available. So in the platform, there are 7 different TV accesses: 6 IPTV accesses and 1 DVB-T access. Figure 1 shows an overview of the architecture and a synopsis of the platform in the context of TV services. Figure 1
35、: Architecture overview of TV service broadcasting 6 Presentation of test conditions To assess the video quality of the different TV services, the principles are to analyze several video sequences belonging to different channels. TVTVTVSTBHGWSTBHGWDVB-T ReceiverAccess NodeBackhaulIP BackboneTV produ
36、ction platformPOPAccessResidential domainArchitecture OverviewBroadcast station1212Classical TV serviceIPTV servicesETSI ETSI TR 101 565 V1.1.1 (2011-09) 86.1 Indicator description To take into account the end user context, only objective measurements models based on No Reference approach correspond
37、 to the need. The current issue is that there is no standardized algorithm in that area. The ITU-T Recommendation J.247 i.3 is not applicable in this context because J.247 is a model functioning with reference. J.144 i.4 is also not applicable in this context due to these restrictions (synchronisati
38、on issues, not application to MPEG4). For the time being, it is proposed to qualify video quality by the occurrence of particular degradations like “black screen“, blockiness and frozen picture. So in this context, the indicators are the following ones. 6.1.1 “Black Screen“ Occurrences Definition Cu
39、rrently, a major trouble of IPTV service is the fact of having displayed on the TV set a “black screen“. Black screen can outcome from: encoder / decoder implementation when no video stream is present a major loss of video packet during a long period of time This metric correspond to the number of b
40、lack screen sequence during a time period (24 hours, 1 week). “Black screen“ is one of the cases of “isochrominance“. Assessment method This default is detected mainly by using robots or probes implementing objective video signal measurement algorithms that are able to detect an image fully coloured
41、 in black. Currently, this is the most suitable approach so as to perform a consistent signal-based analysis. Unit Number or Ratio (number of occurrence by time unit). Standardization reference ES 202 765-4 i.1. Comment The duration of the measurement should be greater than the “inter-advertisements
42、“ duration, because sometimes the broadcasters insert “black screen“ sequences not visible by the users between advertising. ETSI ETSI TR 101 565 V1.1.1 (2011-09) 96.1.2 Blockiness Occurrences Definition In video and image compression, a common artifact called “Blockiness“ comes firstly from low-qua
43、lity compression when too few bits are used. This artifact can be appeared when packet loss ratio is too high packet loss ratio on the transmission link (operator network, user installation). Blockiness is an obviously perceptible contrast of color at the boundaries of the encoding blocks with a cod
44、ec like JPEG or MPEG video. This metric correspond to the number of blockiness sequence during a time period (24 hours, 1 week). B-3 block Group of pels. For example, a block of 8xuniF0208 pels is the smallest coding block used in MPEG-1 algorithms. There are 1 320 blocks in a CIF image, 44 in the h
45、orizontal direction (352 pels/8) and 36 in the vertical direction (288 lines/8). B-4 block distortion Distortion of the image characterized by the appearance of an underlying block encoding structure, also called tiling. E-22 error blocks A form of block distortion where one or more blocks in the im
46、age bear no resemblance to the current or previous scene and often contrast greatly with adjacent blocks. Assessment method This default is mainly detected by using robots or probes that implement objective video signal measurement algorithms that are able to detect it. Currently, this is the most s
47、uitable approach so as to perform a consistent signal-based analysis. These measurements may be done by taking into account the STB integrating error recovery. Unit Number or Ratio (number of occurrence by time unit). Standardization reference ES 202 765-4 i.1 Comment It is often referred to macro b
48、locking, this occurs when a certain amount of the IPTV streams are unavailable to the Set Top Box at playout time. This is most commonly due to packet loss at some point in the network but could be due to everything from content encoding issues to delay-variations (jitter) as packets arrive too late
49、 to the STB. 6.1.3 Frozen Picture Occurrences Definition The frozen picture phenomena can be expressed through some pictures appearing as stopped / freezed from time to time on the end-user screen. These freezes may be issued by the decoder or the network. There are usually very annoying for the end-user. This metric correspond to the number of freezed picture sequence during a time period (24 hours, 1 week). Assessment method In practical way, this indicator can be measured by verifying on adjacent image the stability of the luminance an
