1、American National StandardDeveloped byfor Information Technology Biometric Performance Testingand Reporting Part 4: Operational TestingMethodologiesANSI INCITS 409.4-2006ANSIINCITS 409.4-2006Copyright American National Standards Institute Provided by IHS under license with ANSI Not for ResaleNo repr
2、oduction or networking permitted without license from IHS-,-,-Copyright American National Standards Institute Provided by IHS under license with ANSI Not for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSIINCITS 409.4-2006American National Standardfor Information Tech
3、nology Biometric Performance Testingand Reporting Part 4: Operational TestingMethodologiesSecretariatInformation Technology Industry CouncilApproved September 8, 2006 American National Standards Institute, Inc.Copyright American National Standards Institute Provided by IHS under license with ANSI No
4、t for ResaleNo reproduction or networking permitted without license from IHS-,-,-Approval of an American National Standard requires review by ANSI that therequirements for due process, consensus, and other criteria for approval havebeen met by the standards developer.Consensus is established when, i
5、n the judgement of the ANSI Board ofStandards Review, substantial agreement has been reached by directly andmaterially affected interests. Substantial agreement means much more thana simple majority, but not necessarily unanimity. Consensus requires that allviews and objections be considered, and th
6、at a concerted effort be madetowards their resolution.The use of American National Standards is completely voluntary; theirexistence does not in any respect preclude anyone, whether he has approvedthe standards or not, from manufacturing, marketing, purchasing, or usingproducts, processes, or proced
7、ures not conforming to the standards.The American National Standards Institute does not develop standards andwill in no circumstances give an interpretation of any American NationalStandard. Moreover, no person shall have the right or authority to issue aninterpretation of an American National Stand
8、ard in the name of the AmericanNational Standards Institute. Requests for interpretations should beaddressed to the secretariat or sponsor whose name appears on the titlepage of this standard.CAUTION NOTICE: This American National Standard may be revised orwithdrawn at any time. The procedures of th
9、e American National StandardsInstitute require that action be taken periodically to reaffirm, revise, orwithdraw this standard. Purchasers of American National Standards mayreceive current information on all standards by calling or writing the AmericanNational Standards Institute.American National S
10、tandardPublished byAmerican National Standards Institute, Inc.25 West 43rd Street, New York, NY 10036Copyright 2006 by Information Technology Industry Council (ITI)All rights reserved.No part of this publication may be reproduced in anyform, in an electronic retrieval system or otherwise,without pri
11、or written permission of ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of AmericaCAUTION: The developers of this standard have requested that holders of patents that may berequired for the implementation of the standard disclose such patents to the publisher. However,ne
12、ither the developers nor the publisher have undertaken a patent search in order to identifywhich, if any, patents may apply to this standard. As of the date of publication of this standardand following calls for the identification of patents that may be required for the implementation ofthe standard
13、, no such claims have been made. No further patent search is conducted by the de-veloper or publisher in respect to any standard it processes. No representation is made or impliedthat licenses are not required to avoid infringement in the use of this standard.Copyright American National Standards In
14、stitute Provided by IHS under license with ANSI Not for ResaleNo reproduction or networking permitted without license from IHS-,-,-iContentsPageForeword . iiiIntroduction . viii1 Scope. 12 Conformance . 13 Normative References . 14 Terms and Definitions 25 Operational Testing 45.1 Introduction 45.2
15、Planning Operational Tests 45.2.1 Matching Functionality . 55.2.2 Performance Measures 55.2.3 Characteristics of Operational System Evaluated 55.2.3.1 Concept of Operations. 65.2.3.2 Environment. 65.2.3.3 Administrator-Test Subject Interaction. 65.2.3.4 Habituation. 75.2.3.5 Operational Habituation
16、Testing 85.2.3.6 Acclimatization. 85.2.3.7 Levels of Effort and Decision Policies 95.2.3.8 Multiple-Instance Systems. 105.2.4 Scale 105.2.4.1 Test Population 105.2.4.2 Test Transactions 105.2.5 Criteria for System Selection . 105.2.6 System Implementation and Configuration 115.2.7 Genuine Transactio
17、ns 115.2.8 Impostor Transactions . 115.3 Specific Test Plan Development 125.3.1 System Information 125.3.1.1 Specifications. 125.3.1.2 Architecture 125.3.1.3 Outputs 135.3.2 Implementation 135.3.3 Physical Layout of Operational Environment. 135.3.4 Subject-System Interaction 145.3.4.1 Enrollment Eve
18、nts 145.3.4.2 Matching Events 145.3.5 Guidance and Instruction. 155.4 Data Collection Process and Test Subject Management Reporting . 155.4.1 Data Collection Processes. 155.4.2 Test Subject Management. 16Copyright American National Standards Institute Provided by IHS under license with ANSI Not for
19、ResaleNo reproduction or networking permitted without license from IHS-,-,-iiPage5.5 Analyses 165.5.1 Enrollment Analysis . 165.5.2 Verification System Analysis 175.5.3 Identification System Analysis . 185.6 Record Keeping . 185.7 Reporting Performance Results. 195.7.1 Reporting Requirements 195.7.2
20、 Report Structure 19AnnexesA Environmental Conditions Reporting Requirements 20B Subtransaction Events in Operational Testing. 21C Sample Operational Test Plan Outline 22D Sample Operational Test Report Outline. 24Copyright American National Standards Institute Provided by IHS under license with ANS
21、I Not for ResaleNo reproduction or networking permitted without license from IHS-,-,-iiiForeword (This foreword is not part of American National Standard ANSI INCITS 409.4-2006.)INCITS (The International Committee for Information Technology Standards) is theAmerican National Standards Institute-reco
22、gnized Standards Development Organi-zation for information technology within the United States of America. Members of IN-CITS are drawn from Government, Corporations, Academia and other organizationswith a material interest in the work of INCITS and its Technical Committees. INCITSdoes not restrict
23、membership and attracts participants in its technical work from 13different countries, and operates under the rules of the American National StandardsInstitute.In the field of Biometrics, INCITS has established the Technical Committee M1. Stan-dards developed by this Technical Committee have reached
24、 consensus throughoutthe development process and have been thoroughly reviewed through several PublicReview processes. In addition, this American National Standard has been approvedby the INCITS Executive Board and ANSI Board of Standards Review for Publicationas an ANSI INCITS Standard. Karen Higgi
25、nbottom, ChairJennifer Garner, SecretaryOrganization Represented Name of RepresentativeAIM Global Dan MullenCharles Biss (Alt.)Apple Computer, Inc. David MichaelElectronic Industries Alliance Edward Mikoski, Jr.Henry Cuschieri (Alt.)EMC Corporation Gary RobinsonFarance, Inc Frank FaranceGS1 US Frank
26、 SharkeyJames Chronowski (Alt.)Mary Wilson (Alt.)Hewlett-Packard Company. Karen HigginbottomSteve Mills (Alt.)Scott Jameson (Alt.)IBM Corporation . Ronald F. SillettiPeter Schirling (Alt.)IEEE . Judith GormanTerry DeCourcelle (Alt.)Robert Pritchard (Alt.)Jodi Haasz (Alt.)Bob Labelle (Alt.)Intel. Phi
27、lip WennblomDave Thewlis (Alt.)Jesse Walker (Alt.)Grace Wei (Alt.)Lexmark International . Don WrightDwight Lewis (Alt.)Paul Menard (Alt.)Microsoft Corporation . Jim HughesDon Stanwyck (Alt.)Mike Ksar (Alt.)Isabelle Valet-Harper (Alt.)National Institute of Standards incident (2) means that the data w
28、as not acquired and/or processed. It is not feasible to allow a biometric system to attempt to acquire data indefinitely; therefore for systems that do not time out, a time limit shall be established Maximum number or duration of presentations, attempts, and transactions during enrollment are referr
29、ed to as enrollment presentation limits, enrollment attempt limits, and enrollment transaction limits, respectively The Experimenter shall specify and report levels of effort and decision policies for system(s) tested as follows: Minimum and maximum number of presentations, attempts, and transaction
30、s required or permitted to match. A system may allow matching after one attempt, or may require multiple attempts and transactions Minimum and maximum duration permitted or required to match within a given match presentation, attempt or transaction. A system may terminate a match transaction after a
31、 fixed duration. It is not feasible to allow a biometric system to attempt to acquire data indefinitely; therefore for systems that do not time out, a time limit shall be established Maximum number or duration of presentations, attempts, and transactions during matching are referred to as match pres
32、entation limits, match attempt limits, and match transaction limits, respectively Copyright American National Standards Institute Provided by IHS under license with ANSI Not for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSI INCITS 409.4-2006 10 5.2.3.8 Multiple-Inst
33、ance Systems A systems native utilization of multiple instances from the same test subject for the purposes of enrollment and recognition shall be reported. NOTE: In an operational system, an attempt in which the user is rejected may be automatically followed by a fallback attempt in which the syste
34、m utilizes a different instance of the same modality from the same subject. For example, a system may utilize the left index finger for matching and prompt for placement of the right index finger if the left index finger fails. This would typically apply to modalities such as fingerprint and iris in
35、 which most end users can utilize more than one instance for enrollment and recognition. 5.2.4 Scale The Experimenter shall address Test Population and Test Transactions related to operational test scale. 5.2.4.1 Test Population Operational testing typically utilizes employees, citizens, customers,
36、or individuals otherwise associated with an organization or entity as its test population. The Administrator shall report the relationship between Test Subjects and the entity in whose environment testing is being conducted. The Administrator shall report whether the subjects utilize the biometric s
37、ystem(s) being evaluated in the regular course of their interaction with the organization, or whether utilization takes place specifically for the purposes of the test. If possible, statistical methods should be utilized to establish a minimum Test Population size based upon the level of confidence
38、desired from the test results. The degree to which the Test Subjects are representative of the target population shall be documented. NOTE: An operational test of employees comfortable with biometric technologies may not generate results applicable to a target population. 5.2.4.2 Test Transactions T
39、he number of transactions executed in an operational test should be commensurate with the normal interaction of end users with the biometric system in the operational environment. For systems previously in use, any modifications to system interaction introduced in the course of the test shall be rep
40、orted. If possible, statistical methods should be utilized to establish minimum numbers of transactions required for the overall test population and/or individual Test Subjects to achieve the desired confidence level from the test results. The number of transactions shall be reported. 5.2.5 Criteria
41、 for System Selection The Experimenter shall address the criteria by which biometric system(s) are included in an operational test. Copyright American National Standards Institute Provided by IHS under license with ANSI Not for ResaleNo reproduction or networking permitted without license from IHS-,
42、-,-ANSI INCITS 409.4-2006 11 The Experimenter shall determine and report the rationale for biometric system selection. Biometric systems may be included in an operational test due to their having been previously fielded, due to selection on the part of the sponsoring organization, or due to selectio
43、n on the part of the evaluating entity Operational testing may incorporate one or multiple biometric systems. The Experimenter shall determine and report the rationale for the number of biometric systems to be tested. The number of biometric systems tested may be constrained by budgetary constraints
44、, availability of suitable technologies, or time required to process Test Subjects If multiple biometric systems are tested for the purpose of comparative evaluation, the Administrator shall report on the degree to which factors such as ordering, environment, population, and throughput requirements
45、may have impacted the comparative nature of the evaluation 5.2.6 System Implementation and Configuration The test system may be configured to maximize the amount of data accessible to the test organization so long as such configuration does not materially impact performance. NOTE: The issue of visib
46、ility into device operations is essential to operational testing. Depending on the output of the device, one may not know whether a rejected transaction was due to an actual rejection, a time-out, or a failure to acquire. 5.2.7 Genuine Transactions Operational tests should incorporate methods by whi
47、ch the identity of Test Subjects can be confirmed without recourse to the biometric system. Use of any such non-biometric methods used to confirm the identity of Test Subjects executing transactions recorded as genuine shall be reported. The degree of certainty regarding the identity of Test Subject
48、s executing transactions recorded as genuine shall be reported. NOTE: In an operational environment, it may be impossible to determine whether transactions intended to be genuine are in fact being executed by the correct Test Subject. Further, it may be impossible to determine whether transactions a
49、re being executed in good faith. 5.2.8 Impostor Transactions Operational tests shall incorporate impostor trials. Generation of impostor trials is a difficult aspect of operational testing. The following are potential methods through which impostor transactions can be generated. In-line with genuine Test Subject system usage. Impostors may
