1、CCITT RECMN*F=OL 93 m 4862593 0577058 T42 m I NTE R NATI ON AL TELECOM M U N I CATI0 N U N I ON ITU-T TELECOMMUNICATION STANDARD IZATI ON SECTOR OF ITU OPERATIONS AND QUALITY OF SERVICE HUMAN FACTORS - GENERAL (NOT SPECIFIC TO TELEPHONE SERVICE) F.901 (03193) USABILITY EVALUATION OF TELECOMMUNICATIO
2、N SERVICES ITU-T Recommendation F.901 (Previously ?CCIlT Recommendation?) CCITT RECNN*F-SOL 93 48b259L 0577059 989 FOREWORD The ITLJ Telecommunication Standardization Sector (ITU-T) is a permanent organ of the International Telecom- munication Union. The ITU-T is responsible for studying technical,
3、operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Conference (WTSC), which meets every four years, established the topics for study by the ITU-T Study Groups which, in t
4、heir turn, produce Recommendations on these topics. ITU-T Recommendation F.901 was prepared by the ITLJ-T Study Group I (1988-1993) and was approved by the WTSC (Helsinki, March 1-12, 1993). NOTES 1 As a consequence of a reform process within the International Telecommunication Union (IT), the CCITT
5、 ceased to exist as of 28 February 1993. In its place, the ITU Telecommunication Standardization Sector (ITU-T) was created as of 1 March 1993. Similarly, in this reform process, the CCIR and the IFRl3 have been replaced by the Radiocommunication Sector. In order not to delay publication of this Rec
6、ommendation, no change has been made in the text to references containing the acronyms “CCITT, CCIR or IFRB” or their associated entities such as Plenary Assembly, Secretariat, etc. Future editions of this Recommendation will contain the proper terminology related to the new ITU structure. 2 telecom
7、munication administration and a recognized operating agency. In this Recommendation, the expression “Administration” is used for conciseness to indicate both a O IT 1993 All rights reserved. No part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanic
8、al, including photocopying and microfilm, without permission in writing from the ITU. CCITT RECMN+F=SOL 93 m 48b259L 05770b0 bTO = Recommendation F.901 USABILITY EVALUATION OF TELECOMMUNICATION SERVICES (Helsinki, 1993) 1 Introduction This Recommendation defines guidelines for measuring the usabilit
9、y of human-to-system interfaces in telecornmuni- cations. Usability is widely recognized as one of the components of the Quality of Service. It plays a pivotal role in determining the success of new telecommunication services. 2 Definitions The concept of usability and its components has been define
10、d within IS0 and ETSI. A specific interpretation of this definition for the human-to-system interface is as follows: usability: A concept comprising the effectiveness, efficiency and satisfaction with which specified users can achieve specified system tasks in a particular environment; effectiveness
11、: Measures of the accuracy and completeness of the system tasks achieved; efficiency: Measures of the accuracy and completeness of system tasks relative to the resources (e.g. be, human effort) used to achieve the specific system tasks; satisfaction: Measures of the comfort and acceptability of the
12、system to its users and other people affected by its use. 3 Evaluation aspects The current literature and laboratmy reports show that there is a world-wide commonality on the experimentai procedures for usability evaluation. Both the techniques of the Protocol Analysis) and controlled experiments ha
13、ve been used to evaluate the human-to-system interface. These techniques include many of the following considerations: decide which usability dimensions are to be measured; explore methods by which these can be assessed within the system-specific framework; set explicit, quantifiable usability crite
14、ria (i.e. reference values); design experimental tasks by which to assess the degree to which usability criteria have been met by the system; let a sample of experimental subjects complete the set tasks. The subjects can be tested one at a time or together, dependent on experimental design. Subjects
15、 should be given instructional material as demanded by the tasks; analyse the data statistically; modify those areas of the system in which the usability criteria were not met, relying on anecdotal observations of where and when subjects encountered cognitive stumbling blocks; Protocol Analysis: Sta
16、ndard evaluation method in cognitive psychology. Recommendation F.901 (03/93) 1 CCITT RECMN*F*SDL 93 4862591 0577061 537 9 8) 9) re-test as under 5) above, using different subjects belonging to the same class but the same tasks as before in the modified system; upon completion of the new experiment,
17、 analyse data: i) ii) by themselves as before, using the same types of analyses, and comparing data from both experiments; 10) iterate through steps 7) to 9) until usability criteria are clearly met by the system. 3.1 Methods and measures Since usability is a multi-disciplinary concept, no single, u
18、niversally “correct“ usability evaluation method exists. The choice of the most suitable methods for data capture among the many available (such as data log, observation, questionnaires, interviews and so forth) depends on the system under investigation and the usability dimensions deemed important
19、for that system. In a public telecommunication terminal, for example, effectiveness would seem to be the most important dimension of usability. People should be able to approach the telecommunication terminal, make use of the service and pay for the service without any fuss. In the case of a public
20、telecommunication terminal, learnability should not be an issue, as one may assume that people will not use a public telecommunication terminal which they cannot readily use. The use of both performance and attitude measures is recommended because they focus on different aspects of usability. Perfor
21、mance measures, sometimes referred to as objective measures, include performance time, error rate and number of failures to complete the task, whilst attitude (or subjective) measures capture user opinions. Checks should be made to ensure that there are no human factors deficiencies (such as poor li
22、ghting). 3.2 Criteria The usability criteria are quantified reference values set operationally for each usability component. The data captured is compared against the criteria to assess whether these have been reached. 3.3 Tasks Experimental tasks are to be identified by task analysis and appropriat
23、e scenarios made up. The scenarios should cover the most important system functions and be typical of the tasks that potential users will perform in an up-and-running system. 3.4 Users Experimental subjects should be representative of the potential users. A suitable number of subjects should partici
24、pate in the experiment, so as to guarantee the statistical reliability of the results. 3.5 Analysis of dataz) Experimental data is processed to check how well the usability criteria have been met. Problems are identified and solutions proposed. 4 Example An application of the general procedure to a
25、specific instance is provided in the Annex A. 2, In certain situations a statistically significant sample is not necessary to bring to light a design problem. 2 Recommendation F.901 (03/93) Annex A (to Recommendation F.901) Usability evaluation of ISDN videotelephone terminals (This annex forms are
26、integral part of this Recommendation) The following is an example of an approach to evaluating the usability of the ISDN videophone service access procedures according to this Recommendation. This example contains specific numeric values and specific procedures for illustrative purposes. In an actua
27、l evaluation, these values and procedures and other particular details of the test would of course have to be adjusted to suit the situation being tested. A.l The user interface The ISDN videophones are here referred to as terminals consisting of a basic ISDN telephone set and a colour video display
28、 (typically 9” to 127. The telephone set consists of a numeric keypad, a number of additional keys and a display which provides the user with information (status of the line, dialled number, charging, etc.) and prompts. Dialling may be performed according to the “overiap” and “en-bloc” procedure. Th
29、e “overiap” procedure encompasses the following steps: going off-hook, waiting for dial tone, dialling, changing mode from audio to audiovisual communication. The “en-bloc” procedure consists of dialling, going off-hook and changing mode from audio to audiovisual communication. The following functio
30、ns are provided in addition to dialling: terminal Set-up, re-did, supplementary services, suspendresume, diaiied number editing, mute, hands-free, video pause, self-view, mode change control. They are accessed by means of dedicated keys located near the numeric keypad. The self-view facility may be
31、provided through a separate monitor. Instructions of use may be provided in written form or verbally by the personnel who instails the equipment. A.2 Users The videophones have been designed for the general public. A.3 Environment The videophones for home or office applications are designed for an e
32、nvironment with the following viewing conditions: - - environmental lighting: 2 400 lux; viewing distance: 60 to 120 cm. A.4 Tasks When using the videophone the user can set up videophone calls with either the “en-bloc” or “overlap” procedure, edit dialled numbers in the “en-bloc” procedure, change
33、mode (from audio to audiovisual and vice versa), display hisher own picture (unless continuously provided on a separate screen), inhibit the transmission of the outgoing picture, release videophone ds. Recommendation F.901 (03/93) 3 CCITT RECMN*F=SOL 73 4862573 0577063 30T m AS Usability measures To
34、 evaluate the videophone usability both user performance and user attitudes are to be measured. The following data are captured: number of failures to complete a task, error rate, task completion time and satisfaction of user procedures. A.6 Testdesign To check whether the usability criteria are met
35、 by the videophone user interface, laboratory tests are to be performed. In the following an example is provided of how to plan such an experiment for testing the most important aspects of videotelephony . A.6.1 The subjects At least thirty representative users drawn as a random sample (the higher t
36、he number, the more statistically reliable the results) take part in the experiment. A.6.2 The environment The videophone terminal is located in premises with 400 lux lighting and is used at 60 cm viewing distance. A.6.3 The tasks In each experimental session a given subject performs the following e
37、xperimental tasks: - Task 1 consists of setting up a videophone call with the “overlap” procedure. The videophone number to be diailed is a 10-digit one. This task has to be repeated five times. Task 2 consists of setting up a videophone call with the “en-bloc” procedure. The videophone number to be
38、 diailed is a 10-digit one. This task has to be repeated seven times. Task 3 consists of editing one digit in the middle of a 10-digit telephone number dialled with the “en-bloc” procedure. This task has to be repeated five times. - - To assess the effect of learnability a subject will typically ret
39、urn for a number of sessions spread over a period of time. The experimental conditions are administered in a different random order to each subject. A.6.4 The usability criteria They are as follows: A.6.4.1 Task 1 1) 90% of the subjects should be able to set up the five videophone calls within a ses
40、sion using the “overlap” procedure without committing any errors3) or failing to complete a task in the last three attempts; the average performance time of the last (correct) attempt should not exceed 60 seconds; the MOS4) of satisfaction should be higher than 3 and the distribution should not be s
41、ignificantly bimodal. 2) 3) 4 Recommendation F.901 (03D3) CCITT RECMN*F-SOL 93 4862593 0577064 24b A.6.4.2 Task 2 1) 90% of the subjects should be able to set up the seven videophone calls within a session using the “en- bloc” procedure without committing any errors or failing to complete a task in
42、the last three attempts; the average performance time of the last (correct) attempt should not exceed 60 seconds; the MOS of satisfaction should be higher than three and the distribution should not be significantly bimodal. 2) 3) A.6.4.3 Task 3 1) 90% of the subjects sbould be able to complete the f
43、ive wrong digit correction attempts within a session without committing any errors or failing to complete a task in the last three attempts; 2) the average performance time for editing the wrong digit should not exceed 30 seconds; 3) the MOS of satisfaction should be higher than three and the distri
44、bution should not be significantly bimodal. A.6.5 The experimental procedure All the users receive written instructions on how to use the equipment and a description of the tasks to be performed. Training may be provided if the service providers introduction foresees so. Then the subjects carry out
45、the assigned tasks the specified number of times. At each attempt the accuracy (number of errors or failures) and speed (performance the) are recorded. No time-out for task completion is set. In case of failure to complete task, subjects proceed to the next attempt. Possible questions on the experim
46、ent may be answered by the experimenter, but not ones on the tasks and the use of the terminal. At the end of each task the subjects are asked to express their “satisfaction” assessment on a 5-point rating scale. Recommendation F.901 (03/93) 5 CCITT RECMN*F.SOL 73 48b257L 0577065 182 Printed in Switzerland Geneva, 1993
