1、American National StandardDeveloped byfor Information Technology Biometric Performance Testingand Reporting Part 3: Scenario Testingand ReportingANSI INCITS 409.3-2005ANSIINCITS 409.3-2005Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduc
2、tion or networking permitted without license from IHS-,-,-Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-ANSIINCITS 409.3-2005American National Standardfor Information Technolog
3、y Biometric Performance Testingand Reporting Part 3: Scenario Testing and ReportingSecretariatInformation Technology Industry CouncilApproved October 25, 2005 American National Standards Institute, Inc.AbstractThis standard specifies requirements for scenario-based biometric testing and reporting.Co
4、pyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Approval of an American National Standard requires review by ANSI that therequirements for due process, consensus, and other criteri
5、a for approval havebeen met by the standards developer.Consensus is established when, in the judgement of the ANSI Board ofStandards Review, substantial agreement has been reached by directly andmaterially affected interests. Substantial agreement means much more thana simple majority, but not neces
6、sarily unanimity. Consensus requires that allviews and objections be considered, and that a concerted effort be madetowards their resolution.The use of American National Standards is completely voluntary; theirexistence does not in any respect preclude anyone, whether he has approvedthe standards or
7、 not, from manufacturing, marketing, purchasing, or usingproducts, processes, or procedures not conforming to the standards.The American National Standards Institute does not develop standards andwill in no circumstances give an interpretation of any American NationalStandard. Moreover, no person sh
8、all have the right or authority to issue aninterpretation of an American National Standard in the name of the AmericanNational Standards Institute. Requests for interpretations should beaddressed to the secretariat or sponsor whose name appears on the titlepage of this standard.CAUTION NOTICE: This
9、American National Standard may be revised orwithdrawn at any time. The procedures of the American National StandardsInstitute require that action be taken periodically to reaffirm, revise, orwithdraw this standard. Purchasers of American National Standards mayreceive current information on all stand
10、ards by calling or writing the AmericanNational Standards Institute.American National StandardPublished byAmerican National Standards Institute, Inc.25 West 43rd Street, New York, NY 10036Copyright 2005 by Information Technology Industry Council (ITI)All rights reserved.No part of this publication m
11、ay be reproduced in anyform, in an electronic retrieval system or otherwise,without prior written permission of ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of AmericaCAUTION: The developers of this standard have requested that holders of patents that may berequired fo
12、r the implementation of the standard disclose such patents to the publisher. However,neither the developers nor the publisher have undertaken a patent search in order to identifywhich, if any, patents may apply to this standard. As of the date of publication of this standardand following calls for t
13、he identification of patents that may be required for the implementation ofthe standard, no such claims have been made. No further patent search is conducted by the de-veloper or publisher in respect to any standard it processes. No representation is made or impliedthat licenses are not required to
14、avoid infringement in the use of this standard.Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-iContentsPageForeword iiIntroduction vii1 Scope. 12 Conformance . 13 Terms and defi
15、nitions. 14 Requirements for scenario testing . 44.1 Planning scenario evaluations . 44.1.1 Concept of operations 44.1.2 Matching functionality 44.1.3 Test transactions 54.1.4 Test population 54.1.5 Test system(s) selection 54.1.6 Environmental conditions. 64.1.7 Physical layout of test environment
16、. 64.1.8 Test administration. 64.1.9 Test subject management 74.1.10 Habituation. 84.1.11 Decision policies 84.1.12 Data collection . 94.1.13 Retention of samples . 94.1.14 Test platform 104.1.15 Test system information. 104.2 Analyses 124.2.1 Enrollment analysis 124.2.2 Verification system analyses
17、 124.2.3 Identification system analyses . 134.2.4 Level of effort analyses 144.2.5 Interim analyses. 144.2.6 Record keeping 144.3 Reporting performance results. 154.3.1 Reporting requirements . 154.3.2 Report structure . 15AnnexesA Environmental Conditions Reporting Requirements 16B Subtransaction E
18、vents in Scenario Tests 17C Scenario Test Plan Outline 18D Scenario Test Report Outline. 22E Reporting Effort Levels 25F Reporting Cross-System Results in Multi-System Tests . 27G Bibliography . 28Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo
19、 reproduction or networking permitted without license from IHS-,-,-.3-2005.)Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Foreword (This foreword is not part of American Nation
20、al Standard ANSI INCITS 409This standard specifies the requirements for scenario-based biometric testing and re-portingINCITS (The International Committee for Information Technology Standards) is theANSI recognized Standards Development Organization for information technologywithin the United States
21、 of America. Members of INCITS are drawn from Govern-ment, Corporations, Academia and other organizations with a material interest in thework of INCITS and its Technical Committees. INCITS does not restrict membershipand attracts participants in its technical work from 13 different countries, and op
22、er-ates under the rules of the American National Standards Institute. In the field of Biometrics, INCITS has established the Technical Committee M1. Stan-dards developed by this Technical Committee have reached consensus throughoutthe development process and have been thoroughly reviewed through sev
23、eral PublicReview processes. In addition, this American National Standard has been approvedby the INCITS Executive Board and ANSI Board of Standards Review for Publicationas an ANSI INCITS Standard. The ANSI INCITS 409 series of standards was prepared by Technical CommitteeINCITS/M1, Biometrics, Tas
24、k Group 5, Performance Testing and Reporting. TheANSI INCITS 409 series of standards consists of the following parts, under the gen-eral title Biometric Performance Testing and Reporting:Part 1: Principles and Framework (ANSI INCITS 409.1);Part 2: Technology Testing and Reporting (ANSI INCITS 409.2)
25、;Part 3: Scenario Testing and Reporting (ANSI INCITS 409.3);Part 4: Operational Testing and Reporting (ANSI INCITS 403.4 in development);Part 5: Framework for Testing and Evaluation of Biometric Devices for Access Control (ANSI INCITS 403.5 in development)This document contains seven annexes; four a
26、nnexes that are normative and threeannexes that are informative.Requests for interpretation, suggestions for improvement or addenda, or defect re-ports are welcome. They should be sent to InterNational Committee for InformationTechnology Standards (INCITS), ITI, 1250 Eye Street, NW, Suite 200, Washi
27、ngton,DC 20005.This standard was processed and approved for submittal to ANSI by INCITS. Com-mittee approval of this standard does not necessarily imply that all committee mem-bers voted for its approval. At the time it approved this standard, INCITS had thefollowing members:Karen Higginbottom, Chai
28、rJennifer Garner, SecretaryOrganization Represented Name of RepresentativeApple Computer, Inc. David Michael Electronic Industries Alliance Edward Mikoski, Jr. Henry Cuschieri (Alt.)EMC Corporation Gary Robinson Farance, Inc. Frank Farance GS1 US Frank Sharkey James Chronowski (Alt.)iiMary Wilson (A
29、lt.)Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Organization Represented Name of RepresentativeHewlett-Packard Company .Karen Higginbottom Steve Mills (Alt.)Scott Jameson (Al
30、t.)IBM Corporation Ronald F. Silletti Institute for Certification of Computer Professionals .Kenneth M. Zemrowski Thomas Kurihara (Alt.)IEEE Judith Gorman Richard Holleman (Alt.)Robert Pritchard (Alt.)Intel .Philip Wennblom Dave Thewlis (Alt.)Jesse Walker (Alt.)Lexmark InternationalDon Wright Dwight
31、 Lewis (Alt.)Paul Menard (Alt.)Microsoft CorporationIsabelle Valet-Harper Don Stanwyck (Alt.)Mike Ksar (Alt.)National Institute of Standards if available, such documents shall be utilized and their use noted in the Test Report. If the system lacks a native decision policy (such as if the technology
32、is only implemented via SDK as opposed to application software), then the system shall be implemented in a fashion commensurate with a typical decision policy in that application. The examiner shall document which components of the decision policy could be manipulated. EXAMPLE Access control applica
33、tions often utilize a best-of-three decision policy for matching. Many biometric systems allow for multiple attempts and transactions to enroll due to the necessity for high-quality data collection. Systems may incorporate either timeout or attempt-based failure policies depending on the modality an
34、d application. NOTE 1 See Annex B for discussion of the relationship between presentations, attempts, and sequences. NOTE 2 Defining the point at which enrollment and matching errors have occurred can be problematic in a scenario test. More than one attempt or placement is normally permitted for enr
35、ollment and verification in a real-world application; however, enrollment and matching transactions must be bound in terms of presentations or time. Further, decision policies manifest themselves differently for different modalities. See Annex E for a sample rendering of matching performance for var
36、ying levels of effort. 4.1.12 Data collection The experimenter shall report the following information related to data collection. Methods of recording data for each performance element, including those not logged by the system(s) Processes for auditing and validating performance data collection, inc
37、luding those not logged by the systems(s) The experimenter shall provide examples of data collection elements such as spreadsheets and logs, whether as screenshots or reproduced forms. 4.1.13 Retention of samples Retention of samples acquired during enrollment and matching is strongly recommended fo
38、r the purposes of cross-comparative testing. Such retention may be a native function of the system under evaluation, or may require modification on the part of the technology provider or testing organization. If samples are retained, test design shall incorporate an approach to processing such data.
39、 If samples are not retained, the Test Report shall document why such inclusion was not possible. NOTE 1 Care should be taken to ensure that sample retention does not impact transaction time. It may be necessary to execute some number of transactions both with and without sample retention to gauge i
40、ts impact. NOTE 2 Certain technologies may facilitate cross-comparative matching through retention of templates as opposed to samples. NOTE 3 If the experimenter intends to compare real-time results with results derived from sample cross-comparison, it is necessary to ensure that the samples retaine
41、d represent the same samples used in real-time processing. A system may acquire several frames during a sequence, utilizing one frame for sample retention and another frame for template generation. Copyright American National Standards Institute Provided by IHS under license with ANSINot for ResaleN
42、o reproduction or networking permitted without license from IHS-,-,-ANSI INCITS 409.3-2005 10 4.1.14 Test platform Scenario testing may entail multiple systems being tested on different platforms, ranging from standalone devices to multi-processor workstations. The experimenter shall ensure that sys
43、tems processing power and specifications are commensurate with the scenario being evaluated. The experimenter shall collect information regarding the system(s) tested sufficient to execute testing and report test results. 4.1.15 Test system information 4.1.15.1 Implementation The experimenter shall
44、report whether the system was evaluated through application or demonstration software or through a custom implementation of a software development kit. If application or demonstration software was utilized, the precise nature of this software should be reported. EXAMPLE Vendors may provide an applic
45、ation specifically designed for testing that provides variable thresholds, full event logging, or other functionality not accessible through their standard software. If an SDK was utilized to build a custom test application, the examiner shall specify which partie(s) were responsible for specificati
46、on of functions as well as development. NOTE Scenario testing may incorporate commercial off-the-shelf (COTS) systems, customized systems, or a mixture of both. There are benefits to enforcing COTS-only or customized-only system requirements. By testing COTS systems, a test organization has an incre
47、ased certainty that device performance is reflective of a sensor/algorithm combination as available on the market. By allowing for customization, a test organization has an increased certainty that a sensor/algorithm combination can be modified to meet the requirements of a given test scenario. Cust
48、omization of a biometric system could entail modifying enrollment thresholds to accommodate specific test populations. Customization of systems for a test effort is not generally considered an ideal process, as the results may be more reflective of a vendors ability to customize a system to address
49、a specific scenario than of the biometric systems core abilities to enroll and verify users. However, it should be noted that there may be substantial interest in how well a vendor can customize a system to meet a certain test protocol. The examiner shall report the method of biometric system acquisition. Vendors may loan or donate systems, or sys
