1、_SAE Technical Standards Board Rules provide that: “This report is published by SAE to advance the state of technical and engineering sciences. The use of this report is entirely voluntary, and its applicability and suitability for any particular use, including any patent infringement arising theref
2、rom, is the sole responsibility of the user.” SAE reviews each technical report at least every five years at which time it may be reaffirmed, revised, or cancelled. SAE invites your written comments and suggestions. Copyright 2008 SAE International All rights reserved. No part of this publication ma
3、y be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE. TO PLACE A DOCUMENT ORDER: Tel: 877-606-7323 (inside USA and Canada) Tel: 724-776-4970 (outside USA)
4、Fax: 724-776-0790 Email: CustomerServicesae.org SAE WEB ADDRESS: http:/www.sae.orgJ2830 JUL2008 SURFACEVEHICLEINFORMATIONREPORTIssued 2008-07Process for Comprehension Testing of In-Vehicle Icons RATIONALEIcons and symbols can be used to communicate information to the driver in a manner that is not d
5、ependent on language and can save valuable space on in-vehicle displays. Incomprehensible icons, however, have the potential to negatively affect safety (e.g., if the driver does not understand the icon). Despite the ubiquity of icons and symbols within the in-vehicle environment, few guidelines exi
6、st for testing of icons and symbols. Key shortcomings of existing icon testing procedures include: a lack of contextual information provided to experimental subjects, over-reliance on evaluator judgment regarding how well an experimental subject comprehended an icons meaning, and a lack of prescript
7、ive informationfeedback to icon designers regarding how individual icons could be improved based on the results of comprehension testing. INTRODUCTIONThis information report provides a valid and reliable process for comprehension testing of candidate automotive icons and symbols. The comprehension t
8、esting process was developed though a multi-year effort, supported by the SAE International (SAE) and other organizations, aimed at generating a test methodology that would: yield high-quality comprehension data for new automotive symbols, provide clear and specific guidance back to symbol developer
9、s based on the test results, and could be adopted and performed internationally to support international standards efforts. TABLE OF CONTENTS 1. SCOPE 22. REFERENCES 22.1 Applicable Publications . 22.1.1 SAE Publications. 22.2 Related Publications . 33. PROCESS STEPS 33.1 Step 1: Review Candidate Ic
10、ons for Comprehension Testing 33.2 Step 2: Prepare for Comprehension Testing 43.3 Step 3: Conduct Comprehension Testing. 73.4 Step 4: Analyze Comprehension Data and Summarize Results 73.5 Comparison of SAE and ISO Methods . 104. NOTES 104.1 Marginal Indicia. 10Copyright SAE International Provided by
11、 IHS under license with SAENot for ResaleNo reproduction or networking permitted without license from IHS-,-,-SAE J2830 Issued JUL2008 - 2 -APPENDIX A PROVIDING SUBJECTS WITH CONTEXT DURING ICON EVALUATIONS 11APPENDIX B EXAMPLE AND BLANK SCORE SHEETS 13FIGURE 1 EXAMPLE OF USING A PHOTOGRAPH AS TEST
12、STIMULI, SHOWING THE ICON- TO-BE-TESTED, AS WELL AS SURROUNDING AREAS OF THE VEHICLE INTERIOR . 5FIGURE 2 EXAMPLE OF A COMMON ICON THAT CAN BE PRESENTED TO SUBJECTS PRIOR TO TESTING 6FIGURE 3 THE “TIRE PRESSURE WARNING-LOW” ICON 9FIGURE A1 SAMPLE ICON FOR OCCUPANT PROTECTION (SEAT BELT/AIR BAG) 12FI
13、GURE A2 SAMPLE ICON FOR MOTORIST SERVICES . 12FIGURE B1 ICON GENERATING EXAMPLE SCORE SHEET RESPONSES 13TABLE 1 RATING SCALES FOR CATEGORIZING AND SCORING SUBJECT RESPONSES TO THE ICONS. 8TABLE 2 SAMPLE RESPONSES AND RATINGS FOR A “TIRE PRESSURE WARNING-LOW” ICON . 9TABLE B1 EXAMPLE SCORE SHEET FOR
14、ICON IN FIGURE B1. 13TABLE B2 SCORING CRITERIA IN SCORE SHEET FORMAT 141. SCOPE This document describes a process for testing the comprehension of symbols or icons. Although the process may be used to test any symbols or icons, it has been developed specifically for testing ITS active safety symbols
15、 or icons (e.g., collision avoidance), or other symbols or icons that reflect some in-vehicle ITS message or function (e.g., navigation, motorist services, infotainment). Within the process, well-defined criteria are used to identify the extent to which the perceived meaning matches the intended mea
16、ning for a representative sample of drivers. Though the process described below reflects a paper-and-pencil approach to conducting the testing, electronic means (i.e., conducted using a computer) can be used as well. The data or results from this process are analyzed to assess the drivers comprehens
17、ion of the symbol or icon. These data will be used to provide guidance in the design of in-vehicle symbols or icons. 2. REFERENCES 2.1 Applicable Publications The following publications form a part of this specification to the extent specified herein. Unless otherwise specified, the latest issue of
18、SAE publications shall apply. 2.1.1 SAE Publications Available from SAE International, 400 Commonwealth Drive, Warrendale, PA 15096-0001, Tel: 877-606-7323 (inside USA and Canada) or 724-776-4970 (outside USA), www.sae.org.Campbell, J. L., Kludt, K., and Kiefer, R. J. (2007). Evaluation of In-vehicl
19、e Symbols for an Intersection Crash Avoidance System. Paper presented at the 14th Asia Pacific Automotive Engineering Conference (APAC-14), August 5-8, 2007, Hollywood, CA Campbell, J. L., Hoffmeister, D.H., Kiefer, R. J., Selke, D. J., Green, P. diverse backgrounds and fields of study are preferred
20、. e. Testing is typically done with groups of 10-20 subjects in order to increase the overall efficiency of individual sessions. Copyright SAE International Provided by IHS under license with SAENot for ResaleNo reproduction or networking permitted without license from IHS-,-,-SAE J2830 Issued JUL20
21、08 - 7 -For example, an adequate number of younger subjects can be obtained from local universities. Past studies in this area have placed an advertisement in the daily newspaper at a local university or on craigslist, an on-line classifieds service. The advertisement provides a very general descrip
22、tion of the research being conducted, as well as a telephone number to call for those who are interested in participating. Individuals responding to the ads can be screened and, as appropriate, scheduled for a data collection session. Older subjects can be obtained in a similar fashion using local n
23、ewspapers instead of the university newspaper. It may be more efficient to directly contact local retirement centers or senior centers. Using this method, an initial telephone call can be made to key administrative staff at the retirement or senior center, followed, perhaps, by a direct presentation
24、 to the potential subjects themselves describing the purpose of the study. Once adequate interest and availability of subjects at the retirement or senior center have been established, an on-site data collection session can be scheduled. 3.3 Step 3: Conduct Comprehension Testing 3.3.1 Provide subjec
25、ts with instructions and examples. a. Indicate the context in which the icon will be used (see Appendix A). b. Explain the task that subjects will perform during the testing. 3.3.2 Test candidate symbols with a representative group of subjects. Present test subjects with candidate icons and ask them
26、 to write down the action, condition, activity, location, etc. that they believe is represented by the icon. 3.3.3 If not conducted prior to the comprehension testing, conduct appropriateness ranking testing.1After the comprehension testing is complete, ask subjects to rank order the candidate symbo
27、ls for a particular message in terms of the degree to which they think each symbol communicates the message, in the following manner: a. On a separate page/section of the response booklet, state the true and complete meaning for the symbols, and present the symbols vertically in a column. b. Ask the
28、 subjects to place a “1” next to the symbol that they think is the best symbol for communicating the message, a “2” next to the symbol that they think is the second best symbol for communicating the symbol, and so on, until all the candidate symbols have been ranked according to their perceived appr
29、opriateness. 3.4 Step 4: Analyze Comprehension Data and Summarize Results 3.4.1 Analyze data. 3.4.1.1 Train at least two scorers. Have at least two (2) trained scorers categorize responses along a scale according to well-defined criteria that identify the likelihood that an individual response indic
30、ates correct comprehension of the icon. Adequately trained scorers should, at least: (1) be familiar and comfortable with the criteria presented in Table 1, (2) practice scoring sample data from a representative comprehension test or a pre-test, and (3) have a chance to compare and reconcile these p
31、ractice scoring efforts against the results of other scorers. The “major” and “minor” components of each icons message (as provided by the organization submitting the iconsee Step 1 above) should be used to categorize the subjects responses. That is, the perceived meaning (the subjects response) sho
32、uld be compared to the intended meaning (provided by the organization submitting the icon).1It was noted above thattypicallyproduction tests and appropriateness ranking tests should be conducted prior to conducting comprehensiontesting. However, appropriateness ranking data can be a useful complemen
33、t to comprehension data, and can be collected and scored with very little effort. In particular, the appropriateness ranking test can be a useful way to resolve “ties” between candidate icons that result from the comprehension testing.Copyright SAE International Provided by IHS under license with SA
34、ENot for ResaleNo reproduction or networking permitted without license from IHS-,-,-SAE J2830 Issued JUL2008 - 8 -3.4.1.2 Score responses for each icon. Table 1 should be used to make this comparison between perceived and intended meaning and to categorize the subjects responses. For each icon candi
35、date, convert the total number of responses in each category into percentages. During the scoring process, scorers should maintain a detailed score sheet of the specific subject responses assigned to each of the nine (9) comprehension score categories for each icon tested. These can be provided back
36、 to the organization that originally submitted the icons for comprehension testing. Appendix B provides an actual example of this score sheet (taken from recent icon comprehension tests), as well as a blank score sheet. In order to support the evaluation of the comprehension results, as well as subs
37、equent design decisions arising from the testing, it may also be helpful to note the frequency of individual responses on the score sheet. 3.4.1.3 Get scorer consensus. Once all of the subjects responses for an individual icon have been scored by the two trained scorers, the scorers should compare t
38、heir scores, reconcile any differences, and develop a single “consensus” set of scores for each icon. These consensus scores will represent the final scores for each icon. 3.4.1.4 Diagnose responses. Of the nine (9) scoring categories in Table 1, scores of 1 or 2 are the most important, as theywhen
39、summeddefine overall comprehension rates. The remaining scores (particularly 3, 4, 5, and 9) are most useful for diagnostic purposes; i.e., identifying problems with candidate icons and possible methods for improving comprehension. For safety-critical icons, identify the number and percentage of cri
40、tical confusions or errors (Category 9 in Table 1). Critical confusions or errors reflect responses that indicate that the subject perceived the message to convey a potentially unsafe action. For non-safety-critical icons, such errors will probably fall into Category 5 or 6. 3.4.2 If appropriateness
41、 ranking data were collected, analyze these data. Calculate average rankings for each symbol; lower average rankings correspond to higher levels of perceived appropriateness.TABLE 1 - RATING SCALES FOR CATEGORIZING AND SCORINGSUBJECT RESPONSES TO THE ICONS ComprehensionScore Description 1 The respon
42、se matches the intended meaning of the icon exactly. 2The response captures all major informational elements of the intended meaning of the icon, but is missing one or more minor informational elements.3The response captures some of the intended meaning of the icon, but it is missing one or more maj
43、or informational elements. 4The response does not match the intended meaning of the icon, but it captures some major or minor informational elements. 5The response does not match the intended meaning of the icon, but it is somewhat relevant. 6Participants response is in no way relevant to the intend
44、ed meaning of the icon.7 Participant indicated he/she did not understand the icon. 8 No answer. 9For safety-critical icons, identify the number and percentage of critical confusions or errors. Critical confusions or errors reflect responses that indicate that the participant perceived the message to
45、 convey a potentially unsafe action.Copyright SAE International Provided by IHS under license with SAENot for ResaleNo reproduction or networking permitted without license from IHS-,-,-SAE J2830 Issued JUL2008 - 9 -Table 2 provides sample responses for the “Tire pressure warning-low” icon (shown in
46、Figure 3) and their corresponding ratings, taken from our recent icon comprehension research. FIGURE 3 - THE “TIRE PRESSURE WARNING-LOW” ICON TABLE 2 - SAMPLE RESPONSES AND RATINGS FOR A “TIRE PRESSURE WARNING-LOW” ICON Subject Response Rating Description Warning - low tire pressure1 The response ma
47、tches the intended meaning of the icon exactly. Tire pressure needs attention2The response captures all major informational elements of the intended meaning of the icon, but it is missing one or more minor informational elements. Normal tire pressure 3 The response captures some of the intended mean
48、ing of the icon, but it is missing one or more major informational elements. You have a mechanical problem4The response does not match the intended meaning of the icon, but it captures some major or minor informational elements. Tire symbol 5 The response does not match the intended meaning of the i
49、con, but it is somewhat relevant. Be aware of what is ahead6Participants response is in no way relevant to the intended meaning of the icon. 3.4.3 Interpret and summarize results. Decisions regarding criteria for minimum percent correct comprehension rates for individual icons should reflect designers needs, as well
