ITU-T P 1501-2014 Subjective testing methodology for web browsing (Study Group 12)《网络浏览的主观测试方法》.pdf

1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T P.1501 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (02/2014) SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Methods for objective and subjective assessment of quality of services other than voice se

2、rvices Subjective testing methodology for web browsing Recommendation ITU-T P.1501 ITU-T P-SERIES RECOMMENDATIONS TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal charact

3、eristics Series P.30 P.300 Reference systems Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 P.800 Audio

4、visual quality in multimedia services Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting assessment Series P.1300 Statistical analysis

5、, evaluation and reporting guidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than voice services Series P.1500 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P.1501 (02/2014) i Recommendat

6、ion ITU-T P.1501 Subjective testing methodology for web browsing Summary Recommendation ITU-T P.1501 describes subjective testing methods for assessing the user perceived quality for web browsing in browser-based applications of different device classes. It gives guidance on the selection of a test

7、environment, equipment and content, and how test procedures should be properly planned in order to facilitate realistic web browsing behaviour by the subjects. The methods described can be used to identify the impact of several different factors that influence the user perceived quality of web brows

8、ing, e.g., identifying the influence of the browsing device on user perceived quality for identical technical access network settings. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T P.1501 2014-02-13 12 11.1002/1000/12126 Keywords Absolute Category Rating, Mean Opinion Scor

9、e, page view cycle, web browsing, web-QoE. _ * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/11830-en. ii Rec. ITU-T P.1501 (02/2014) FOREWORD The In

10、ternational Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical,

11、 operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in tu

12、rn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOT

13、E In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure,

14、e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not

15、 suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the

16、evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, whi

17、ch may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2014 All rights reserved. No part of this publication m

18、ay be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.1501 (02/2014) iii Table of Contents Page 1 Scope . 1 2 References . 1 3 Definitions 1 3.1 Terms defined elsewhere 1 3.2 Terms defined in this Recommendation . 2 4 Abbreviations and acronyms 2 5 Conv

19、entions 2 6 Web browsing test procedure . 2 6.1 Purpose . 2 6.2 Test facilities 3 6.3 End-user devices . 4 6.4 Test content 4 6.5 Test procedure 5 6.6 Test conditions 6 6.7 Subjects . 6 6.8 Tasks . 6 6.9 Questions 7 6.10 Data analysis and report . 7 Annex A Result reporting: mandatory structure and

20、details . 9 A.1 Test set-up 9 A.2 Result presentation . 10 Appendix I Example of instructions for the web browsing test 11 Appendix II Example of task instructions for the free-browsing test . 12 Appendix III Example of instructions for the task-dependent browsing test . 14 Appendix IV Example of qu

21、estions for the web browsing test 15 IV.1 Assessment of media quality 15 IV.2 Assessment of temporal aspects . 15 Appendix V Example test plan . 16 V.1 Test facilities 16 V.2 Test content 16 V.3 Test conditions and test design . 16 V.4 Result presentation . 17 Bibliography. 18 Rec. ITU-T P.1501 (02/

22、2014) 1 Recommendation ITU-T P.1501 Subjective testing methodology for web browsing 1 Scope This Recommendation describes the methods and procedures used for subjective testing of user perceived quality of web browsing. The method generally applies to degradations and characteristics that can be int

23、roduced at the network level (e.g., round-trip time, downlink and uplink bandwidth, packet losses), as well at the application level (e.g., page load-times). Combinations of two or more of these factors also have to be considered. Impacting factors due to webpage design are not in the scope of this

24、Recommendation. 2 References The following ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are su

25、bject to revision; users of this Recommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a documen

26、t within this Recommendation does not give it, as a stand-alone document, the status of a Recommendation. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.800.1 Recommendation ITU-T P.800.1 (2006), Mean Opinion Scores (MOS) terminol

27、ogy. ITU-T P.910 Recommendation ITU-T P.910 (2008), Subjective video quality assessment methods for multimedia applications. ITU-T Y.1540 Recommendation ITU-T Y.1540 (2011), Internet protocol data communication service IP packet transfer and availability performance parameters. ITU-T Y.1541 Recommen

28、dation ITU-T Y.1541 (2011), Network performance objectives for IP-based services. ITU-R BT.500 Recommendation ITU-R BT.500 (2012), Methodology for the subjective assessment of the quality of television pictures. ITU-T HSTP ITU- T Handbook (2011), Practical procedures for subjective testing, ITU, Gen

29、eva. 3 Definitions 3.1 Terms defined elsewhere None. 2 Rec. ITU-T P.1501 (02/2014) 3.2 Terms defined in this Recommendation This Recommendation defines the following terms: 3.2.1 condition: Time period in which a defined quality of service (QoS) setting is applied for a certain webpage. A condition

30、is concluded by retrospective user ratings on one or multiple dimensions of quality. 3.2.2 element: Visual content of a webpage, which is displayed to the user on the rendered webpage, e.g., text, pictures, widgets, videos. 3.2.3 object: HTTP object, which is used for processing and rendering the we

31、bpage and is referenced by the page mark-up or script. These objects are not necessarily visible on the fully rendered page (e.g., elements). 3.2.4 task: A set of instructions given to the user on how to reach certain information within a test condition. 3.2.5 web browsing session: An interactive in

32、formation exchange between a user and one or more websites over a limited period of time, mediated via a web browsing application. The starting point of a session is the first page request initiated by the user, which is followed by a number of request-response interactions between the user and the

33、webhost(s), resulting in a series of page views. See b-Egger-2. A web session is typically terminated when the user exits the browsing application or stops the browsing activity for a certain period of time. 4 Abbreviations and acronyms This Recommendation uses the following abbreviations and acrony

34、ms: DLBW Downlink Bandwidth MOS Mean Opinion Score RTT Round Trip Time ULBW Uplink Bandwidth WBS Web Browsing Session 5 Conventions The methodology discussed in this Recommendation is tailored for web browsing only. Browser-based applications, such as online office applications, webmail, etc., might

35、 be evaluated by the same means, but they have to consider different scenarios depending on the defined tasks during subjective evaluation and the technical implementation of the services. The focus of this Recommendation is the evaluation method of the complete web browsing session (WBS) to achieve

36、 and maintain a flow experience of web browsing for better external validation of the test results. However, this methodologys test set-up with respect to test facilities, test conditions, subject selections and test devices is also applicable to the subjective evaluation of single web page loading

37、times. 6 Web browsing test procedure 6.1 Purpose Subjective quality tests for web browsing aim to assess the impact of transmission impairments on the user-perceived quality of a web browsing session (WBS) under realistic usage conditions. Rec. ITU-T P.1501 (02/2014) 3 The main characteristics of a

38、subjective web browsing test are: To simulate realistic web browsing where users are browsing and interacting with webpages in order to acquire certain information; the procedure they go through within this methodology ensures that users get into a browsing mode rather than a pure page loading mode.

39、 Subjects are exposed to a certain quality of service (QoS) level over a period of time rather than for one event, in order to grasp several request-response cycles for the subjective evaluation. The use of certain tasks to stimulate the interaction between the webpage and the subject for each test

40、condition. The webpage must be interactive and has to provide sufficient content such that the subject can browse through it over several conditions, without getting bored. This test methodology can be adapted to field testing; however, it has to be ensured that the manipulated parameters (e.g., del

41、ay, packet loss, downlink bandwidth) can be set to the desired values and that these settings can be verified by a posteriori analysis (e.g., traffic traces). The results obtained with this methodology can be used by telecommunication providers, vendors and customers to get information on the influe

42、nce of the tested technical parameters on end-user perceived quality. 6.2 Test facilities Typically, test facilities for subjective web browsing evaluations include an end-user device, a network emulator, a router and some optional components as depicted in Figure 6-1 below. Network parameters are s

43、et through a network emulator (EMU) which is placed between the user device (UD) and the router (RT), which ensures the eventual routing between local content servers and the internet. When using live webpages within the evaluation one has to ensure the measurement of the page health, in order to id

44、entify potential influences due to congestion or server malfunctions. This is done through a page health monitor (PHM) connected to the Internet via the router. See Figure 6-1. In the case of multiple subjects participating in parallel, it must be ensured that each subject receives the obliged param

45、eters, hence shared channel settings have to be avoided. If webpages are used, that include video content that is also affected by the test parameters, additional requirements for the lab facilities as specified in ITU-T P.910 apply. P . 1 5 0 1 (1 4 )_ F 0 1Cl i en t w . Bro w s er ,el ec t ro n i

46、c p ro mp t er , et c.U s er d ev i ce N et w o rk em u l at o rEM UA cc es s n et w o rk(L A N / W i fi / 3 G / 4 G )Co n t ro lco n n ec t i o n sL A N / W i fiOP T es t o p er at o r fr o n t -e n dPag e h ea l t h mo n i t o rP HMIn t er n etRo u t erR1 TL o ca l co n t en ts er v erLOCFigure 6-

47、1 Example of test facilities 4 Rec. ITU-T P.1501 (02/2014) 6.3 End-user devices The end-user device has a major impact on the overall performance as perceived by the user. Therefore, detailed reporting about the device characteristics and setting are essential. If the end-user device or only any of

48、its characteristics are considered to be a variable parameter of the quality evaluation, then all applied variations must be reported. 6.3.1 Hardware Due to the great dynamics in web technology developments, it is desirable that modern hardware is used for the tests in order to ensure that unintende

49、d device performance does not dominate the chosen test variables. At the time of drafting this Recommendation, hardware can be roughly classified in four categories: 1) PC/laptop, 2) tablets, 3) smartphones and 4) other devices that are able to connect to the web. Typically, these four categories feature different performance issues. Therefore, one should only use devices from different categories when they are part of the condition variables under test. Displays should be calibrated to the D65 standard illum