1、American National StandardDeveloped byfor Information Technology Biometric Performance Testingand Reporting Part 1: Principles and FrameworkANSI INCITS 409.1-2005ANSIINCITS 409.1-2005Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction
2、or networking permitted without license from IHS-,-,-Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSIINCITS 409.1-2005American National Standardfor Information Technology Bio
3、metric Performance Testingand Reporting Part 1: Principles and FrameworkSecretariatInformation Technology Industry CouncilApproved October 25, 2005 American National Standards Institute, Inc.AbstractThis multipart standard develops a common set of methodologies and procedures to be followed for con-
4、ducting technical performance testing and evaluations. Included are guidelines that address issues re-garding required test sizes, performance statistics, error reporting, and presentation of performanceresults. These procedures can be incorporated in an “end-to-end“ system approach or from an indiv
5、idualtechnical component perspective. Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Approval of an American National Standard requires review by ANSI that therequirements for d
6、ue process, consensus, and other criteria for approval havebeen met by the standards developer.Consensus is established when, in the judgement of the ANSI Board ofStandards Review, substantial agreement has been reached by directly andmaterially affected interests. Substantial agreement means much m
7、ore thana simple majority, but not necessarily unanimity. Consensus requires that allviews and objections be considered, and that a concerted effort be madetowards their resolution.The use of American National Standards is completely voluntary; theirexistence does not in any respect preclude anyone,
8、 whether he has approvedthe standards or not, from manufacturing, marketing, purchasing, or usingproducts, processes, or procedures not conforming to the standards.The American National Standards Institute does not develop standards andwill in no circumstances give an interpretation of any American
9、NationalStandard. Moreover, no person shall have the right or authority to issue aninterpretation of an American National Standard in the name of the AmericanNational Standards Institute. Requests for interpretations should beaddressed to the secretariat or sponsor whose name appears on the titlepag
10、e of this standard.CAUTION NOTICE: This American National Standard may be revised orwithdrawn at any time. The procedures of the American National StandardsInstitute require that action be taken periodically to reaffirm, revise, orwithdraw this standard. Purchasers of American National Standards may
11、receive current information on all standards by calling or writing the AmericanNational Standards Institute.American National StandardPublished byAmerican National Standards Institute, Inc.25 West 43rd Street, New York, NY 10036Copyright 2005 by Information Technology Industry Council (ITI)All right
12、s reserved.No part of this publication may be reproduced in anyform, in an electronic retrieval system or otherwise,without prior written permission of ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of AmericaCAUTION: The developers of this standard have requested that h
13、olders of patents that may berequired for the implementation of the standard disclose such patents to the publisher. However,neither the developers nor the publisher have undertaken a patent search in order to identifywhich, if any, patents may apply to this standard. As of the date of publication o
14、f this standardand following calls for the identification of patents that may be required for the implementation ofthe standard, no such claims have been made. No further patent search is conducted by the de-veloper or publisher in respect to any standard it processes. No representation is made or i
15、mpliedthat licenses are not required to avoid infringement in the use of this standard.Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-iContentsPageForeword ii1 Scope and Conform
16、ance 11.1 Scope . 11.2 Conformance 12 Normative References 13 Terms and Definitions. 24 Biometric Systems 34.1 General. 34.2 Identification and Verification Systems. 44.3 Positive and Negative Recognition Claims. 45 Evaluation Protocols. 55.1 General. 55.2 Technology Evaluation . 55.3 Scenario Evalu
17、ation 55.4 Operational Evaluation . 56 Testing Characteristics . 66.1 General. 66.2 Physical Environment . 66.3 User Population 66.4 Habituated vs Nonhabituated . 66.5 Attended vs Unattended. 76.6 Privacy 77 Evaluation Test Data 77.1 Online / Offline Testing. 77.2 Test Size. 88 Generalized Test Proc
18、edure. 89 Data Analysis Part 2: Technology Testing and Reporting (ANSI INCITS 409.2);Part 3: Scenario Testing and Reporting (ANSI INCITS 409.3);Part 4: Operational Testing and Reporting (ANSI INCITS 403.4 in development);Part 5: Framework for Testing and Evaluation of Biometric Devices for Access Co
19、ntrol (ANSI INCITS 403.5 in development)This document contains two annexes that are informative and are for informationonly. They do not contain information required by this standard.Requests for interpretation, suggestions for improvement or addenda, or defect re-ports are welcome. They should be s
20、ent to InterNational Committee for InformationTechnology Standards (INCITS), ITI, 1250 Eye Street, NW, Suite 200, Washington,DC 20005.This standard was processed and approved for submittal to ANSI by INCITS. Com-mittee approval of this standard does not necessarily imply that all committee mem-bers
21、voted for its approval. At the time it approved this standard, INCITS had thefollowing members:iiCopyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Karen Higginbottom, ChairJennifer
22、 Garner, SecretaryOrganization Represented Name of RepresentativeApple Computer, Inc .David Michael Electronic Industries Alliance Edward Mikoski, Jr. Henry Cuschieri (Alt.)EMC Corporation Gary Robinson Farance, Inc. Frank Farance GS1 US.Frank Sharkey James Chronowski (Alt.)Mary Wilson (Alt.)Hewlett
23、-Packard Company .Karen Higginbottom Steve Mills (Alt.)Scott Jameson (Alt.)IBM CorporationRonald F. Silletti Institute for Certification of Computer Professionals.Kenneth M. Zemrowski Thomas Kurihara (Alt.)IEEEJudith Gorman Richard Holleman (Alt.)Robert Pritchard (Alt.)Intel .Philip Wennblom Dave Th
24、ewlis (Alt.)Jesse Walker (Alt.)Lexmark International .Don Wright Dwight Lewis (Alt.)Paul Menard (Alt.)Microsoft CorporationIsabelle Valet-Harper Don Stanwyck (Alt.)Mike Ksar (Alt.)National Institute of Standards i.e., someone fools the system and an unauthorized person is granted access. A ROC measu
25、res the trade-off between the probability of correct verification and the false accept rate by plotting the true accept rate or the probability of a correct verification on the y-axis against the false accept rate on the x-axis . A Detection Error Trade-off (DET) is also used for this purpose. It is
26、 a modified ROC curve that plots the false rejection rate (rather than the true match rate) along the y-axis. The DET curve plotted on logarithmic axes has the effect of spreading out the plot and distinguishes different well-performing systems more clearly. The ROC and DET plots are also used for t
27、he identification system analysis. Guidelines and details for each evaluation protocol are presented in the other respective parts of this standard. However, there is a special case for identification analysis such that the enrolled population will always contain a mate for the user. This is known a
28、s a closed-set identification. For this situation, a Cumulative Match Characteristic curve (CMC) is used to graphically illustrate the results. This curve plots the rank on the x-axis and the probability of identification at that rank or better on the y-axis. Copyright American National Standards In
29、stitute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
30、ANSI INCITS 409.1-2005 11 Annex A Comparison Table of Test Types (Informative) Table A.1 Comparison of test types Type of Test Technology Scenario Operational Subject of Biometric component Biometric system Biometric system testing (matching or extraction algorithm, sensor) Ground truth Known, subje
31、ct to data collection errors and intersections in merged data sets Known, subject to data collection errors and tester failure to note Unknown unwanted subject behavior User behavior controlled by experimenter Not applicable during testing. May be known to be controlled when biometric data recorded,
32、 otherwise considered to be uncontrolled. Controlled (unless user behavior is an independent variable) Uncontrolled The possibility that the user has real-time feedback of No Yes Yes the result of attempt Repeatability of results Repeatable Quasi-repeatable (if test scenario and population controlle
33、d) Not repeatable Control of May be known to be controlled when Controlled and/or recorded Not controlled, ideally physical biometric data recorded, otherwise recorded environment considered to be uncontrolled. User Not applicable during testing. May be Recorded May be recorded during interaction re
34、corded when biometric data is verification/identification recorded recorded Typical results reported Comparison of biometric components or versions of components (e.g., matching or extraction algorithms or sensors), Determine critical performance factors Compare biometric systems, Determine critical
35、 performance factors. Measure simulated performance Measure performance in an operational environment Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSI INCITS 409.1-2005 12 Ty
36、pical metrics Most performance metrics. Not end-to-end throughput. Most error rates. Good for large-scale identification system performance where difficult to assemble large test crew. Predicted end-to-end throughput, FMR, FNMR, FTA, and FTE. Operational FRR. Constraints Appropriate test database, e
37、.g., gathered with one or more sensors, the identity of which may or may not be known. Operational, instrumented system Operational, instrumented system; typically only decision rates are available Human test population Recorded Live Live NOTE: Although in some cases there may be exceptions to the e
38、ntries in this table, these are the mainstream, fundamental characteristics and distinctions. Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSI INCITS 409.1-2005 13 Annex B Sa
39、mple Size (Informative) The question of how large a biometric test has to be in order to reliably draw conclusions on the error rate of a biometric system is an important issue. It arises in the context of showing that a biometric product is performing at or above some requirement, and in inferring
40、what level of error can be claimed given some observed error rate in so many trials. A trial here refers to a verification or identification attempt. The problem is statistical and is treated by regarding a biometric test as just a sequence of Bernoulli trials, i.e., independent events, each of whic
41、h has a fixed probability of error. The results of this Annex apply separately to both to genuine and impostor attempts. B.1 Sample Size Estimates Table B.1 Application of Sample Size Estimates Kind Metric Number of trials possible given N enrollees with two samples per person and M true impostors w
42、ith one sample per person 1 Genuine (user A vs. enrollee A) FNMR N. The case when more than two samples are available for each user is discussed in the notes following Table B.4. 2 Impostor (user A vs. enrollee B) FMR NM dependent trials but only N independent trials. See notes after Table B.4. The
43、Binomial distribution governs the case of Bernoulli trials. The Normal approximation for large numbers of trials gives the number of trials needed to sustain a claim with a specified level of certainty that the true error rate lies between (p - E) and (p + E): EQUATION 1Copyright American National S
44、tandards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSI INCITS 409.1-2005 14 Here p is the true underlying error rate of each Bernoulli trial, and z/2= 1.96 for a 95% confidence interval and 1.6449 for a 90% in
45、terval, and E, is the desired error bar on the measurement. It is useful to express the value of E as some fraction, c, of p, so. E = cp. By substituting this in Eq. (1), the formula reduces to: EQUATION 2which for small p is approximately a constant divided by p. The rule-of-thumb results (which ar
46、e apparent in Table B.2) are that more trials are needed for small claimed error rates and for small error bars. Indeed the number of trials increases as the square of the tightness of the error bar. Table B.2 Number of Trials Needed to Measure Error Rate (Equation 3) Underlying Error Rate, p Fracti
47、onal Error Bar 0.002 0.004 0.008 0.01 0.02 0.04 0.08 0.1 0.2 95% confidence c = 0.2 47924 23914 11909 9508 4706 2305 1104 864 384 c = 0.3 21300 10628 5293 4226 2092 1024 491 384 171 c = 0.6 5325 2657 1323 1056 523 256 123 96 43 c = 1.0 1917 957 476 380 188 92 44 35 15 90% confidence c = 0.2 33754 16
48、843 8388 6697 3314 1623 778 609 271 c = 0.3 15002 7486 3728 2976 1473 722 346 271 120 c = 0.6 3750 1871 932 744 368 180 86 68 30 c = 1.0 1350 674 336 268 133 65 31 24 11 NOTE: The equations generalize the “Rule of 30“ which applies for c = 0.3 and 90% confidence. That is, at least 30 errors in 3000 trials should be observed if a claim that the error rate is (0.01 +/- 0.003) is to be upheld with 90% confidence. (The corresponding number in Table B.2 is 2976; difference is due to rounding of z/2).Copyright American National Standards Institute Provided by IHS under license with ANSIN
