1、ANWAIAA G -035A-2000 Guide to Human Performance Measurements AN S I/Al AA G-035A-2000 American National Standard Guide to Human Performance Measurements Sponsor American Institute of Aeronautics and Astronautics Approved 9 July 2001 American National Standards Institute Abstract This Guide provides
2、methods for measuring human performance for the purpose of scientific research and system evaluation. The guidelines are intended to assist scientists and systems specialists in selecting human performance measurement methods appropriate to the situation being studied or the system being evaluated.
3、ANSVAIAA G-035A-2000 Approval of an American National Standard requires verification by ANSI that the requirements for due process, consensus, and other criteria have been met by the Consensus is established when, in the judgment of the ANSI Board of Standards Review, substantial agreement has been
4、reached by directly and materially affected interests. Substantial agreement means much more than a simple majority, but not necessarily unanimity. Consensus requires that all views and objections be considered, and that a concerted effort be made toward their resolution. The use of American Nationa
5、l Standards is completely voluntary; their existence does not in any respect preclude anyone, whether he has approved the standards or not, from manufacturing, marketing, purchasing, or using products, processes, or procedures not conforming to the standards. The American National Standards Institut
6、e does not develop standards and will in no circumstances give an interpretation of any American National Standard. Moreover, no person shall have the right or authority to issue an interpretation of an American National Standard in the name of the American National Standards Institute. Requests for
7、 interpretations should be addressed to the secretariat or sponsor whose name appears on the title page of this standard. CAUTION NOTICE: This American National Standard may be revised or withdrawn at any time. The procedures of the American National Standards Institute require that action be taken
8、to affirm, revise, or withdraw this standard no later than five years from the date of approval. Purchasers of American National Standards may receive current information on all standards by calling or writing the American National Standards Institute. American Standard National standards developer.
9、 Library of Congress Cataloging-in-Publication Data Guide to human performance measurements I sponsor, American Institute of Aeronautics and Astronautics ; approved, American National Standards Institute. p. cm. “AIAAIANSI.“ ISBN 1-56347-451-4 (softcover) 1. Human engineering-United States-Measureme
10、nt. 2. Performance standards-United States. I. American Institute of Aeronautics and Astronautics II. American National Standards Institute. TAI 66 .G85 2000 “G-035A-2000 .“ 620,820287-d21 00-0401 75 CIP Published by American Institute of Aeronautics and Astronautics 1801 Alexander Bell Drive, Resto
11、n, VA 22091 Copyright O 2001 American Institute of Aeronautics and Astronautics All rights reserved. No pari of this publication may be reproduced in any form, in an electronic retrieval system or othetwise, without prior written permission of the publisher. Printed in the United States of America I
12、I ANSVAIAA G-035A-2000 Contents Foreword vi 1 1 .I 1.2 2 3 3.1 3.2 3.3 3.4 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.5 4 4.1 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.3 4.4 4.4.1 4.4.2 4.4.3 4.4.4 4.4.5 4.4.6 4.4.7 4.4.8 4.4.9 4.4.1 O Scope and Purpose Purpose . 1 Vocabulary 1 Applications. . 3 General Applications . Bas
13、ic Research . . 4 Applied Research . . 6 System Design and Development . 1 7 The Checklist . 7 Mockup Testing . . 9 Rapid Prototyping . . 9 Verbal Protocol Methodology . System Test and Evaluation . 12 General Issues in Human Performance Measurement 15 Underlying Problems in Human Performance Measur
14、ement 15 Bridging the Gap Between Human and System Performance Measurement . 16 . 16 16 Using Expert Judgments of Task Performance 17 Modeling and Simulation 17 . 17 18 I9 . 20 . 21 Interactions of Human and System Performance and Their Exploration Developing Performance Criteria . Asking the Right
15、Questions Selection Criteria for Human Performance Measures Appropriate Level of Detail Sensitivity . 21 Diagnosticity . .22 No n-i n t rus ive ness 22 Implementation Requirements 23 Ope rato r Acceptance . 23 Fairness. . 23 . 23 iii ANSVAIAA G-035A-2000 4.4.1 1 4.4.12 4.4.13 4.4.14 4.4.15 4.4.16 4.
16、4.17 4.5 4.5.1 4.5.2 4.5.3 4.5.4 4.6 4.7 4.7.1 4.7.2 4.7.3 4.7.4 4.7.5 4.7.6 4.7.7 4.7.8 4.7.9 4.7.1 O 4.7.1 1 4.7.12 4.7.1 3 4.7.14 4.7.15 4.7.1 6 4.7.1 7 4.8 4.8.1 4.8.2 4.8.3 4.8.4 Sim pl icity . 24 Timeliness . 24 Objectivity 24 Quantitativeness/Qualitativeness 24 Cost . 24 Flexibility . 24 Util
17、ity 24 Measurement Uncertainty . 25 Combining Errors . 25 Outliers . 25 Speed/Accuracy Tradeoff 25 Biases in Usability Measurement 25 Experimental Design 28 Instrumentation 28 Accelerometer 29 Anemometer . 30 Anthropometry Instrument Kit 30 Electrogoniometer . 30 Force, Torque, and Dimension Kit 30
18、Gas Tester . 30 Hygrometer or Psychrometer . 30 Motion Measurement Systems . 30 Photometer 31 Sound Level Meter and Analyzer 31 Spot Brightness Meter 31 Thermometer 31 Vibration Meter and Analyzer 31 Video Tape Instrumentation . 31 Spectroradiometer . 32 Video Digitizer 32 Digital Audio Tape (DAT) R
19、ecorder 32 Data Collection and Analysis . 32 General Considerations 32 Data Collection . 32 . Data Analysis 35 Data Storage 35 iv ANSVAIAA G-035A-2000 4.8.5 5 5.1 5.2 6 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 7 Data Reporting 36 Human Performance in the Context of Systems . 36 Types of Testbeds 36 Human Per
20、formance Analysis and Synthesis 36 Performance-Shaping Factors . 37 Identification of Factors and Their Effect 37 Stress 38 Workload . 39 Situation (a I) Aware ness 40 Motivation 41 General State of Health 39 Physiological Capacity . 39 Train i ng . 41 Referenced Publications . 42 Annex A . Taxonomy
21、 of Human Performance Measures 47 A.l Criteria 47 A.2 References 49 List of Figures Figure 1 . Major Steps in Measurement 4 Figure 2 . The Checklist Comparison Process . 8 Figure 3 . Types of HPM Instrumentation 29 Figure 4 . Suggested File Format . 35 List of Tables Table 1 . Recommended Data Colle
22、ction Approach and Archival Media and Media Formats 34 Table 2 . Representative Performance-Shaping Factors . 39 Table 3 . Factors in Workload Assessment 40 Table AI . Taxonomy of Performance Measures . 48 Table A2- Taxonomy of Driver Errors 51 V ANSVAIAA G-035A-2000 Foreword The project on human pe
23、rformance measurement was initiated under the auspices of the Life Sciences and Systems Committee on Standards (LS (2) to promote commonality across research projects and, thus, enable comparison of results across evaluations; and (3) to develop and use common HPM tools for data collection and data
24、processing. Note that this Guide does not include physiological measures. Such measures will be handled in a separate AIAA standard. Until then, the reader is referred to Martin and Venables (1980). 1.2 Purpose The purpose of this Guide is to aid the reader in measuring human performance. The reader
25、 is assumed to have a basic knowledge of experimental design, statistics, and human performance. 2 Vocabulary anemometer An instrument used to measure local air flow. anthropometer An instrument consisting of a ruler with two perpendicular movable legs, used to measure distances such as upper-arm le
26、ngth. asymptotic learning The point at which performance does not improve with increased practice. average dwell time A measure of instrument-scanning behavior: “the total time spent looking at an instrument divided by the total number of individual dwells on that instrument” (Harris, Glover, and Sp
27、ady, 1986, p. 38). consensus A substantial agreement reached by directly and materially affected interests. continuous performance Performance, such as tracking or monitoring, that requires constant attention over a period of time. CRT Cathode-ray tube. DAT Digital Audio Tape. dimensions Units of me
28、asurement, e.g., deviation from glideslope in meters. discrete performance Performance that has a well-defined start and end, such as switch activation or issuance of a voice command. DT a series of lookpoints that stay within a visual radius of 1 degree (Harris, Glover, and Spady, 1986, p. 38). fix
29、ations per dwell A measure of instrument-scanning behavior: “the number of individual fixations during an instrument dwell” (Harris, Glover, and Spady, 1986, p. 38). function A major category of activity associated with a system or subsystem and assigned to a person or a machine or shared between a
30、person and a machine (Berson and Crooks, 1976). gas tester An instrument that measures, detects, and quantifies presence of specific gases goniometer An instrument for measuring angles, including human joint angles such as wrist flexion. HPM Human performance measurement. hygrometer An instrument th
31、at measures relative humidity. LED Light-emitting diode. lookpoint A measure of instrument-scanning behavior, e.g., “the current coordinates of where the pilot is looking during any one thirtieth of a second” (Harris, Glover, and Spady, 1986, p. 38). oculometer An instrument that measures lookpoint.
32、 OJT On-the-job training. one-way transition A measure of instrument-scanning behavior: “the sum of all transitions from one instrument to another (one direction only) in a specific pair” (Harris, Glover, and Spady, 1986, p. 38). OST Operational system testing OT (2) training, in which measurements
33、are made to determine the effects of instruction on personnel performance or to determine the instructional variables affecting performance; and (3) test and evaluation, in which measurements are made to evaluate the capability of the human to operate and maintain the system in its intended environm
34、ent. The measurements described in this Guide can be applied to any technology that is controlled and operated by personnel (system personnel) or that affects humans as clients, even if these are not system personnel, or both. For example, the operators of a computerized production line stamping out
35、 automobile bodies are system personnel; the owner of one of these automobiles is a client of that system. The focus of the measurements is on the humans performance, although it may appropriate to also measure the equipment or system with which he or she interacts. These measurement guidelines can
36、be applied to a wide range of devices, from the individual work station (such as a word processor) to a large system (such as an airliner). They can be applied to the single operator and to teams of varying size. More specific applications are to types of measurement: (1) basic and applied research;
37、 (2) system design and development; and (3) system test and evaluation. Discussions of these applications are provided below. All measurements should include consideration of the system context in which they are made. System context is defined as the next higher level system in which the equipment o
38、r system being tested is embedded. For example, if measurements are made for a word processor, the office in which the word processor is located should be examined before testing to determine if factors present in the larger office could influence operator performance with the word processor. 3 ANSV
39、AIAA G-035A-2000 Because this Guide can be used in any situation in which humans control or are influenced by equipment and systems, it is independent of any particular application. It is, therefore, incumbent on those who use this Guide to ensure that their usage in any specific context is both sci
40、entifically and legally appropriate. Measurements involving human subjects are controlled by ethical standards described in such documents as the Code of Federal Regulations, Part 46, which forbids exposing human subjects to harmful or potentially harmful tests. These documents also require that all
41、 research involving human subjects must meet current ethical standards and be approved by authorized review boards for the protection of human subjects. Before proceeding to a detailed description of various measurement aspects, it will be useful to summarize the major steps in measurement (Figure 1
42、). Note in Figure 1 that there are feed-forward and feed-back loops and, in particular, that it may be necessary to revise measurement objectives and procedures when required resources are not available. 1.0 Define the system to be measured in terms of 1.1 Specific equipment 1.2 Specific tasks 1.3 Q
43、uestions to be answered (see Section 4.3 and Sections 3.2, 3.3, 3.4, and 3.5) Determine measurement objectives 2.1 Basic research (see Section 3.2) 2.2 Applied research (see Section 3.3) 2.3 System design and development (see Section 3.4) 2.4 System test and evaluation (see Section 3.5) Determine th
44、e measurement environment (see Section 5.1) 3.1 Laboratory 3.2 Engineering facility 3.3 Field test site 3.4 Operational environment Determine resources required 4.1 Subjects 4.1.1 Number 4.1.2 Type 4.1.3 4.2 Cost 4.2.1 4.3 Time required for measurement 4.3.1 Determine measurement constraints 5.1 Non
45、e of importance 5.2 Important; re-examine 2.0 Determine criteria and measures Availability; if not available, reconsider 1.0 and 2.0 Money available? If not, reconsider 1 .O and 2.0 Available or not? If not, reconsider 1.0 and 2.0 -6.1 Criteria (see Sections 4.2.2 and 4.4) 6.2.1 6.2.2 Sufficiently s
46、ensitive/objective? 7.1 Examine available procedures (e.g., instrumentation, observation, interviews) (see Sections 3.4.2, 3.4.3, 3.4.4, and 4.7) 7.2 Select procedures in terms of all preceding steps Develop statistical analysis plan (see Section 4.8.5.3) 8.1 If measurement procedures can provide st
47、atistically valid data, accept procedures 8.2 If measurement procedures will not provide statistically valid data, revise procedures Document all procedures by writing test plan (see Section 3.5) Pretest total measurement procedures (see Section 3.5) 10.1 Accept unaltered 10.2 Unacceptable; revise p
48、rocedure 11.0 Collect data (see Section 4.8) 11.1 Applydatato1.3and2.0 6.2 Measures (see Section 5.2) Validate by expert opinion and/or pretest and comparison of pretest data with objectives 7.0 Develop measurement procedures 8.0 9.0 10.0 11 .I .I 11 .I .2 If questions and objectives answered, stop
49、measurement If questions and objectives unanswered, continue measurement 12.0 Analyze data (see Section 4.8.3) 13.0 Write test report Figure 1 - Major Steps in Measurement 4 ANSVAIAA G-035A-2000 3.2 Basic Research Basic behavioral research is performed to develop an understanding of how behavioral variables function, the interrelationships among them, and their effect on the performance of individuals and groups; and, through the latter, their effect on systems. As a means of acquiring knowledge, basic research is often theory-directed, performed for either the development of a theor
