1、American National StandardDeveloped byfor Information Technology Biometric Performance Testingand Reporting Part 4: Operational TestingMethodologiesANSI INCITS 409.4-2006 (R2011) ANSI INCITS 409.4-2006(R2011)ANSIINCITS 409.4-2006(R2011)American National Standardfor Information Technology Biometric P
2、erformance Testingand Reporting Part 4: Operational TestingMethodologiesSecretariatInformation Technology Industry CouncilApproved September 8, 2006 American National Standards Institute, Inc.Approval of an American National Standard requires review by ANSI that therequirements for due process, cons
3、ensus, and other criteria for approval havebeen met by the standards developer.Consensus is established when, in the judgement of the ANSI Board ofStandards Review, substantial agreement has been reached by directly andmaterially affected interests. Substantial agreement means much more thana simple
4、 majority, but not necessarily unanimity. Consensus requires that allviews and objections be considered, and that a concerted effort be madetowards their resolution.The use of American National Standards is completely voluntary; theirexistence does not in any respect preclude anyone, whether he has
5、approvedthe standards or not, from manufacturing, marketing, purchasing, or usingproducts, processes, or procedures not conforming to the standards.The American National Standards Institute does not develop standards andwill in no circumstances give an interpretation of any American NationalStandard
6、. Moreover, no person shall have the right or authority to issue aninterpretation of an American National Standard in the name of the AmericanNational Standards Institute. Requests for interpretations should beaddressed to the secretariat or sponsor whose name appears on the titlepage of this standa
7、rd.CAUTION NOTICE: This American National Standard may be revised orwithdrawn at any time. The procedures of the American National StandardsInstitute require that action be taken periodically to reaffirm, revise, orwithdraw this standard. Purchasers of American National Standards mayreceive current
8、information on all standards by calling or writing the AmericanNational Standards Institute.American National StandardPublished byAmerican National Standards Institute, Inc.25 West 43rd Street, New York, NY 10036Copyright 2006 by Information Technology Industry Council (ITI)All rights reserved.No pa
9、rt of this publication may be reproduced in anyform, in an electronic retrieval system or otherwise,without prior written permission of ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of AmericaCAUTION: The developers of this standard have requested that holders of patent
10、s that may berequired for the implementation of the standard disclose such patents to the publisher. However,neither the developers nor the publisher have undertaken a patent search in order to identifywhich, if any, patents may apply to this standard. As of the date of publication of this standarda
11、nd following calls for the identification of patents that may be required for the implementation ofthe standard, no such claims have been made. No further patent search is conducted by the de-veloper or publisher in respect to any standard it processes. No representation is made or impliedthat licen
12、ses are not required to avoid infringement in the use of this standard.iContentsPageForeword .iiiIntroduction . viii1 Scope. 12 Conformance . 13 Normative References . 14 Terms and Definitions 25 Operational Testing 45.1 Introduction 45.2 Planning Operational Tests 45.2.1 Matching Functionality . 55
13、.2.2 Performance Measures 55.2.3 Characteristics of Operational System Evaluated 55.2.3.1 Concept of Operations. 65.2.3.2 Environment. 65.2.3.3 Administrator-Test Subject Interaction. 65.2.3.4 Habituation. 75.2.3.5 Operational Habituation Testing 85.2.3.6 Acclimatization. 85.2.3.7 Levels of Effort a
14、nd Decision Policies 95.2.3.8 Multiple-Instance Systems. 105.2.4 Scale 105.2.4.1 Test Population 105.2.4.2 Test Transactions 105.2.5 Criteria for System Selection . 105.2.6 System Implementation and Configuration 115.2.7 Genuine Transactions 115.2.8 Impostor Transactions . 115.3 Specific Test Plan D
15、evelopment 125.3.1 System Information 125.3.1.1 Specifications. 125.3.1.2 Architecture 125.3.1.3 Outputs 135.3.2 Implementation 135.3.3 Physical Layout of Operational Environment. 135.3.4 Subject-System Interaction 145.3.4.1 Enrollment Events 145.3.4.2 Matching Events 145.3.5 Guidance and Instructio
16、n. 155.4 Data Collection Process and Test Subject Management Reporting . 155.4.1 Data Collection Processes. 155.4.2 Test Subject Management. 16iiPage5.5 Analyses 165.5.1 Enrollment Analysis . 165.5.2 Verification System Analysis 175.5.3 Identification System Analysis . 185.6 Record Keeping . 185.7 R
17、eporting Performance Results. 195.7.1 Reporting Requirements 195.7.2 Report Structure 19AnnexesA Environmental Conditions Reporting Requirements 20B Subtransaction Events in Operational Testing. 21C Sample Operational Test Plan Outline 22D Sample Operational Test Report Outline. 24iiiForeword (This
18、foreword is not part of American National Standard ANSI INCITS 409.4-2006 (R2011).)INCITS (The International Committee for Information Technology Standards) is theAmerican National Standards Institute-recognized Standards Development Organi-zation for information technology within the United States
19、of America. Members of IN-CITS are drawn from Government, Corporations, Academia and other organizationswith a material interest in the work of INCITS and its Technical Committees. INCITSdoes not restrict membership and attracts participants in its technical work from 13different countries, and oper
20、ates under the rules of the American National StandardsInstitute.In the field of Biometrics, INCITS has established the Technical Committee M1. Stan-dards developed by this Technical Committee have reached consensus throughoutthe development process and have been thoroughly reviewed through several
21、PublicReview processes. In addition, this American National Standard has been approvedby the INCITS Executive Board and ANSI Board of Standards Review for Publicationas an ANSI INCITS Standard. Karen Higginbottom, ChairJennifer Garner, SecretaryOrganization Represented Name of RepresentativeAIM Glob
22、al Dan MullenCharles Biss (Alt.)Apple Computer, Inc. David MichaelElectronic Industries Alliance Edward Mikoski, Jr.Henry Cuschieri (Alt.)EMC Corporation Gary RobinsonFarance, Inc Frank FaranceGS1 US Frank SharkeyJames Chronowski (Alt.)Mary Wilson (Alt.)Hewlett-Packard Company. Karen HigginbottomSte
23、ve Mills (Alt.)Scott Jameson (Alt.)IBM Corporation . Ronald F. SillettiPeter Schirling (Alt.)IEEE . Judith GormanTerry DeCourcelle (Alt.)Robert Pritchard (Alt.)Jodi Haasz (Alt.)Bob Labelle (Alt.)Intel. Philip WennblomDave Thewlis (Alt.)Jesse Walker (Alt.)Grace Wei (Alt.)Lexmark International . Don W
24、rightDwight Lewis (Alt.)Paul Menard (Alt.)Microsoft Corporation . Jim HughesDon Stanwyck (Alt.)Mike Ksar (Alt.)Isabelle Valet-Harper (Alt.)National Institute of Standards incident (2) means that the data was not acquired and/or processed. It is not feasible to allow a biometric system to attempt to
25、acquire data indefinitely; therefore for systems that do not time out, a time limit shall be established Maximum number or duration of presentations, attempts, and transactions during enrollment are referred to as enrollment presentation limits, enrollment attempt limits, and enrollment transaction
26、limits, respectively The Experimenter shall specify and report levels of effort and decision policies for system(s) tested as follows: Minimum and maximum number of presentations, attempts, and transactions required or permitted to match. A system may allow matching after one attempt, or may require
27、 multiple attempts and transactions Minimum and maximum duration permitted or required to match within a given match presentation, attempt or transaction. A system may terminate a match transaction after a fixed duration. It is not feasible to allow a biometric system to attempt to acquire data inde
28、finitely; therefore for systems that do not time out, a time limit shall be established Maximum number or duration of presentations, attempts, and transactions during matching are referred to as match presentation limits, match attempt limits, and match transaction limits, respectively ANSI INCITS 4
29、09.4-2006 (R2011) 10 5.2.3.8 Multiple-Instance Systems A systems native utilization of multiple instances from the same test subject for the purposes of enrollment and recognition shall be reported. NOTE: In an operational system, an attempt in which the user is rejected may be automatically followe
30、d by a fallback attempt in which the system utilizes a different instance of the same modality from the same subject. For example, a system may utilize the left index finger for matching and prompt for placement of the right index finger if the left index finger fails. This would typically apply to
31、modalities such as fingerprint and iris in which most end users can utilize more than one instance for enrollment and recognition. 5.2.4 Scale The Experimenter shall address Test Population and Test Transactions related to operational test scale. 5.2.4.1 Test Population Operational testing typically
32、 utilizes employees, citizens, customers, or individuals otherwise associated with an organization or entity as its test population. The Administrator shall report the relationship between Test Subjects and the entity in whose environment testing is being conducted. The Administrator shall report wh
33、ether the subjects utilize the biometric system(s) being evaluated in the regular course of their interaction with the organization, or whether utilization takes place specifically for the purposes of the test. If possible, statistical methods should be utilized to establish a minimum Test Populatio
34、n size based upon the level of confidence desired from the test results. The degree to which the Test Subjects are representative of the target population shall be documented. NOTE: An operational test of employees comfortable with biometric technologies may not generate results applicable to a targ
35、et population. 5.2.4.2 Test Transactions The number of transactions executed in an operational test should be commensurate with the normal interaction of end users with the biometric system in the operational environment. For systems previously in use, any modifications to system interaction introdu
36、ced in the course of the test shall be reported. If possible, statistical methods should be utilized to establish minimum numbers of transactions required for the overall test population and/or individual Test Subjects to achieve the desired confidence level from the test results. The number of tran
37、sactions shall be reported. 5.2.5 Criteria for System Selection The Experimenter shall address the criteria by which biometric system(s) are included in an operational test. ANSI INCITS 409.4-2006 (R2011) 11 The Experimenter shall determine and report the rationale for biometric system selection. Bi
38、ometric systems may be included in an operational test due to their having been previously fielded, due to selection on the part of the sponsoring organization, or due to selection on the part of the evaluating entity Operational testing may incorporate one or multiple biometric systems. The Experim
39、enter shall determine and report the rationale for the number of biometric systems to be tested. The number of biometric systems tested may be constrained by budgetary constraints, availability of suitable technologies, or time required to process Test Subjects If multiple biometric systems are test
40、ed for the purpose of comparative evaluation, the Administrator shall report on the degree to which factors such as ordering, environment, population, and throughput requirements may have impacted the comparative nature of the evaluation 5.2.6 System Implementation and Configuration The test system
41、may be configured to maximize the amount of data accessible to the test organization so long as such configuration does not materially impact performance. NOTE: The issue of visibility into device operations is essential to operational testing. Depending on the output of the device, one may not know
42、 whether a rejected transaction was due to an actual rejection, a time-out, or a failure to acquire. 5.2.7 Genuine Transactions Operational tests should incorporate methods by which the identity of Test Subjects can be confirmed without recourse to the biometric system. Use of any such non-biometric
43、 methods used to confirm the identity of Test Subjects executing transactions recorded as genuine shall be reported. The degree of certainty regarding the identity of Test Subjects executing transactions recorded as genuine shall be reported. NOTE: In an operational environment, it may be impossible
44、 to determine whether transactions intended to be genuine are in fact being executed by the correct Test Subject. Further, it may be impossible to determine whether transactions are being executed in good faith. 5.2.8 Impostor Transactions Operational tests shall incorporate impostor trials. Generat
45、ion of impostor trials is a difficult aspect of operational testing. The following are potential methods through which impostor transactions can be generated. In-line with genuine Test Subject system usage. Impostors may execute impostor transactions through the same systems as genuine users, inters
46、persed with genuine users. An impostor may utilize an ID or token against which to execute matching. This provides a direct reflection of the ability of impostors to match in a system. In-line testing may not be viable for reasons of security, throughput, or process flow Through dedicated operationa
47、l systems. Impostors may execute transactions at a dedicated time, or through a dedicated system, utilized solely for the purpose of impostor testing. Such testing would have no impact on primary path operations. An impostor may execute several transactions against one or more gallery records. Admin
48、istrators shall ensure that the method of interaction with dedicated systems is consistent with that of operational devices In a scenario environment, impostors may execute transactions in a nonoperational environment configured with the same thresholds as operational systems. Executing such ANSI IN
49、CITS 409.4-2006 (R2011) 12 testing and synthesizing results with genuine Test Subject data as gathered in operational systems requires that Administrators ensure that the method of interaction with dedicated systems is consistent with that of operational devices NOTE: It may not be viable to maintain zero-effort impostor attempts as in an operational environment a Test Subject will typically know whether genuine or impostor transactions are being executed. If an Administrator wishes to maintain the zero-effort impostor methodology for verification systems, it wi
