1、 ATIS-0800035.v002 TECHNICAL REPORT VALIDATION PROCESS FOR IPTV PERCEPTUAL QUALITY MEASUREMENTS As a leading technology and solutions development organization, ATIS brings together the top global ICT companies to advance the industrys most-pressing business priorities. Through ATIS committees and fo
2、rums, nearly 200 companies address cloud services, device solutions, M2M communications, cyber security, ehealth, network evolution, quality of service, billing support, operations, and more. These priorities follow a fast-track development lifecyclefrom design and innovation through solutions that
3、include standards, specifications, requirements, business use cases, software toolkits, and interoperability testing. ATIS is accredited by the American National Standards Institute (ANSI). ATIS is the North American Organizational Partner for the 3rd Generation Partnership Project (3GPP), a foundin
4、g Partner of oneM2M, a member and major U.S. contributor to the International Telecommunication Union (ITU) Radio and Telecommunications sectors, and a member of the Inter-American Telecommunication Commission (CITEL). For more information, visit . Notice of Disclaimer by standardizing the test plan
5、, it encourages industry developments where multiple organizations could develop QoE models, and these models can be validated based on the same basic test plan. A basic test plan allows for “delta” documents to be developed for extensions that adopt the same fundamentals as this basic test plan but
6、 go into more detail for particular types of QoE models and applications. ATIS-0800035 2 ATIS-0800031 3, IPTV QoE Measurement Recommendations and Framework, identifies some of the aspects in points 2 and 3 in section 5 and in its informative appendices. The document reviews a variety of IPTV QoE mea
7、surement algorithms (e.g., parametric and bit-stream approaches). It also provides some guidance where full-reference, reduced-reference, and no-reference models may be employed in an IPTV network. The present document, ATIS-0800035, Validation Process for IPTV Perceptual Quality Measurements, addre
8、sses the fourth point. Today, standards groups combine the test process and test plan activities with the eventual goal of a standardized PQM solution. The ATIS IIF separates these two processes. The test plan is described in 2, and this document describes a recommended test process. This separation
9、 provides for greater flexibility. This document describes the industry standards test process currently followed, indicates the shortcomings, and then suggests solutions for these weaknesses. The proposed process is flexible and is believed to address market needs better than the current process. 1
10、.3 Application The formalization of a test process is discussed in this document at a high level. The ATIS IIF has identified a streamlined process for validating the quality of proposed measurement solutions. The primary intent of this Technical Report is to advance the state of the art in this imp
11、ortant aspect of the IPTV industry. The ATIS IIF understands and expects the process to evolve and improve over time and therefore solicits comments towards constructing a coordinated industry consensus position on a standardized test process for validating measurements, algorithms, and the related
12、aspects needed to support such efforts. 2 References The following standards contain provisions which, through reference in this text, constitute provisions of this ATIS Standard. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agr
13、eements based on this ATIS Standard are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. 2.1 Normative References 1 ATIS-0800008, QoS Metrics for Linear/Broadcast IPTV, December 2007.12 ATIS-0800025, Test Plan for Evaluation of Quality
14、Models for IPTV Services, October 2009.13 ATIS-0800031, IPTV QoE Measurement Recommendations and Framework, 2012.12.2 Informative References 4 ITU-T Recommendation P.564, Conformance testing for narrowband voice over IP transmission quality assessment models, November 2007.25 ITU-T Recommendation P.
15、862, Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs, February 2001.21This document is available from the Alliance for Telecommunications Industry Solutions, 1200 G Street N.W., Suite 500
16、, Washington, DC 20005. 2This document is available from the International Telecommunications Union. ATIS-0800035 3 6 CableLabs, Certification Wave Requirements and Guidelines, Certification Wave 68-70, May 2009, Revision 31.37 ISO/IEC 17025:2005, General requirements for the competence of testing a
17、nd calibration laboratories, 2005, Edition 2.48 ISO/IEC 17011, Conformity assessment - General requirements for accreditation bodies accrediting conformity assessment bodies, 2004, Edition 1.49 ISO/IEC Guide 65:1996, General requirements for bodies operating product certification systems, 1996, Edit
18、ion 1.410 ITU-T Recommendation P.563, Single-ended method for objective speech quality assessment in narrow-band telephony applications, May 2004.211 ITU-T Recommendation P.861, Objective quality measurement of telephone-band (300-3400 Hz) speech codecs, February 1998.212 ITU-T Recommendation P.862,
19、 Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs, February 2001.213 ITU-T Recommendation P.862.1, Mapping function for transforming P.862 raw result scores to MOS-LQO, November 2003.214 A
20、TIS Technical Report T1.TR.72-2001, Methodological framework for specifying accuracy and cross-calibration of video quality metrics, October 2001.115 ITU-T Recommendation J.149, Method for specifying accuracy and cross-calibration of Video Quality Metrics (VQM), March 2004.23 Definitions, Acronyms,
21、2) For vendors to implement the specification; 3) (optional) To perform independent validation of the implementation, possibly with certification, to see if the vendors product conforms to the specification. This process is followed by a number of industry forums and standards organizations, includi
22、ng the MultiService Forum (MSF), the International Standards Organization (ISO), and CableLabs (see Appendix A). Certification provides a level of comfort to the purchasers of products that the product performs according to specification (standards compliance) and has been proven to interface with o
23、ther vendors products (interoperability). The certification is meaningful when standards are to be supported by two or more vendors. Most notably, and especially in the case of communications, this is beneficial in multi-vendor environments where the interoperability of communication interfaces or p
24、rotocols is involved. ATIS-0800035 6 4.2 Validation Process for QoE Models Differences and Characteristics While there are similarities, the process for QoE models is different from the interoperability and/or compliance process (described in section 4.1). First, there is no strict interoperability
25、requirement between vendor products for QoE models; interoperability testing is not meaningful for QoE models. Instead, a certain level of accuracy is expected from a QoE model, and as a result, accuracy is used as the main criterion in QoE model testing to test whether on its own it results in a cl
26、ose correlation with the equivalent subjective test results. Furthermore, models that have shown a close correlation with subjective tests can form the basis of ITU-T Recommendations that document the results as well as the QoE model algorithms. This does not occur for a process where a vendor imple
27、ments a product, demonstrates interoperability, and is possibly certified for it. Secondly, the standards “process” followed by the ITU-T (and the Video Quality Experts Group VQEG), as observed with the standardization of models for VoIP and digital video quality models, is slightly different. We ca
28、n see the following characteristics: Competition. Collaboration in part (see ITU-R Rec. P.862 5) or in whole (see the component model in the Non-intrusive parametric model for Assessment of performance of Multimedia Streaming P.NAMS). Self-certification (see ITU-R Rec. P.564 4). The interoperability
29、/compliance process does not have these characteristics. After the specification of requirements, the implementation (underlying algorithms) of a vendors product and how the product competes in the market place is a private matter. Certification in the interoperability/compliance process is for that
30、 vendors product alone, not for a collection of products as is the case for QoE models according to the standards process (e.g., ITU-T). The process currently followed for validating QoE models has various characteristics (in no particular order of appearance): QoE is subjective by nature and is dif
31、ficult to capture in detailed requirements. Requirements documents for QoE models are relatively high-level compared to requirements documents related to, for example, transmission gear, for which traditional interoperability/compliance processes are suitable. QoE models are not very well specified
32、in terms of the input and output parameters of the models and where and how these models may be used. The number of vendors that choose to openly test their QoE models is (at the time that this report was created) very small compared to the vendors certifying their equipment following traditional in
33、teroperability/compliance processes. Validating QoE models requires the acquisition of suitable multimedia content. To date, such source content is selected upon the “judgment of experts” (see section 7.3 for more detail). Furthermore, once test material has been made available to model proponents i
34、t cannot be re-used in future validation tests, requiring the selection and preparation of new content that must be used in a new round of subjective testing. Given the current approach (competition, cooperation, etc.), QoE models can only be considered if they are available at a specific cut-off da
35、te, because the validation is scheduled by consensus with multiple models from multiple developers being evaluated in the same exercise. Model developers must reveal the details of their algorithms if they want to be included in the standard. They must also agree to the licensing conditions imposed
36、by the standards body. ITLs carry a large responsibility and a large part of the workload, yet they do not get fully compensated for their work, and they do not stand to benefit financially from the resulting model standards. ATIS-0800035 7 Model developers are sometimes involved in the preparation
37、of a Processed Video Sequence (PVS) or in conducting subjective experiments due to ITL budget and time constraints, which is not ideal for an independent evaluation. The process for validating QoE models is very lengthy. From start to finish, a process can take a number of years. This time is spent
38、on the creation of a unique test plan for the particular type of test (and also indicates who does what and many other process related aspects), collecting source material, impairing the source material, performing subjective tests, performing objective tests, and analysis of results. Successful QoE
39、 models are often documented in a standards document (i.e., a fourth step based on the three steps in section 4.1), such as an ITU-T recommendation (this does not happen with the traditional conformance testing process). This leads to the possibility that successful QoE models can be licensed to oth
40、er vendors who wish to implement the standard. Once a standard has been defined and approved, it is very difficult to change, as the database cannot be re-used. Consequently, the validation process must be repeated with the associated elapsed time to completion. This means that standardized models c
41、an quickly become outdated, and there is no process for the models or the standards to be updated in a prompt fashion. This situation is acceptable only for applications that are somewhat static in nature. 4.3 Proposed Alternative Process The following points summarize the main benefits of the propo
42、sed process for QoE model validation. The various items and issues are discussed in more detail in the remainder of this Technical Report. 1. The most critical component of the entire process is the validation and that there is an open process for approving QoE models. This open process allows for r
43、apid, reliable, and inexpensive validation of QoE models. In such a process, anyone (e.g., industrial, academic, or governmental organizations) who creates a model will have access to this process. With the appropriate set of policies in place, such a process is expected to result in a much more dyn
44、amic market place for quality models than the current standards-based process. 2. The inner details of an algorithm are de-emphasized. This can be done only if there is a well articulated definition of the type of model, and its input and output parameters (see also ATIS-0800031 3) 3. When the detai
45、ls of the algorithm are not subject to standardization, the model developer can make routine updates to algorithms to improve performance. Improved algorithms can be (re-) validated easily and quickly. The proposed process thus promotes innovation, improves the time-to-market for solutions, and redu
46、ces development costs. 4. The proposed process allows vendors to keep their intellectual property intact. Validating the results of the algorithm serves to inform the user (e.g., IPTV service provider or test equipment manufacturer) that the algorithm performs well without necessarily revealing the
47、inner workings of the algorithm. 5. The proposed process allows for development and deployment of models that are “fit-for-purpose” by permitting trade-offs between model accuracy and model complexity. The choice of the specific trade-off lies with the model user and not the standardization body. 6.
48、 Creation of reporting templates provides a standard format for vendors to report the performance of their model. This standard report template will help the industry by simplifying interpretation of the performance characteristics of different vendors models. Note that ATIS-0800025 2 specifies a re
49、port template. 5 The Participants The different entities, generically referred to as participants and their roles, are explained in more detail below. ATIS-0800035 8 MODEL DEVELOPER INDEPENDENT TEST LAB (ITL) MODEL USER THIRD PARTY ORGANIZATION models algorithms reports/results reports capabilities reports Figure 1: The Participants The relationship between the ITL and Model User indicates that test reports may be provided by the ITL directly to the Model User. However, it should be noted in the process
