1、 INCITS/ISO/IEC 19795-2:2007 2009 ISO/IEC 19795-2:2007 Information technology Biometric performance testing and reporting Part 2: Testing methodologies for technology and scenario evaluation INCITS/ISO/IEC 19795-2:2007 2009 PDF disclaimer This PDF file may contain embedded typefaces. In accordance w
2、ith Adobes licensing policy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobes licensing
3、 policy. The ISO Central Secretariat accepts no liability in this area. Adobe is a trademark of Adobe Systems Incorporated. Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Ev
4、ery care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below. Adopted by INCITS (InterNational Committee for Information Technology Standards) a
5、s an American National Standard. Date of ANSI Approval: 8/27/2009 Published by American National Standards Institute, 25 West 43rd Street, New York, New York 10036 Copyright 2009 by Information Technology Industry Council (ITI). All rights reserved. These materials are subject to copyright claims of
6、 International Standardization Organization (ISO), International Electrotechnical Commission (IEC), American National Standards Institute (ANSI), and Information Technology Industry Council (ITI). Not for resale. No part of this publication may be reproduced in any form, including an electronic retr
7、ieval system, without the prior written permission of ITI. All requests pertaining to this standard should be submitted to ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of America ii ITIC 2009 All rights reserved ISO/IEC 19795-2:2007(E) ISO/IEC 2007 All rights reserved
8、iiiContents Page Foreword iv Introduction v 1 Scope 1 2 Conformance1 3 Normative references 1 4 Terms and definitions .2 4.1 Biometric data2 4.2 Components of a biometric system.2 4.3 User interaction with a biometric system .2 4.4 Performance measures .3 5 Overview of technology evaluations and sce
9、nario evaluations.3 6 Technology evaluation6 6.1 Test design.6 6.2 Assembling an appropriate test corpus8 6.3 Performance measurement 11 6.4 Reporting 16 7 Scenario evaluation.18 7.1 Test design.18 7.2 Test crew 23 7.3 Performance measurement 24 7.4 Reporting 26 8 Other issues applicable to technolo
10、gy and scenario evaluations .29 8.1 Parties to a test 29 8.2 Fairness 29 8.3 Basis for inclusion of test systems .29 8.4 Use of Frequently Asked Questions30 8.5 Legal issues .30 8.6 Release of test source code .30 8.7 Supplier comment on test report .30 Annex A (informative) Phases and activities fo
11、r primary technology test types .31 Annex B (informative) Relationship between presentations, attempts, and transactions 37 Annex C (informative) Reporting effort levels38 Annex D (informative) Client-server testing .40 Annex E (informative) Comparing results across systems in multi-system tests .41
12、 ISO/IEC 19795-2:2007(E) iv ISO/IEC 2007 All rights reservedForeword ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate
13、in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and n
14、on-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The mai
15、n task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casti
16、ng a vote. Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights. ISO/IEC 19795-2 was prepared by Joint Technical Committee ISO/IEC JTC 1, Informati
17、on technology, Subcommittee SC 37, Biometrics. ISO/IEC 19795 consists of the following parts, under the general title Information technology Biometric performance testing and reporting: Part 1: Principles and framework Part 2: Testing methodologies for technology and scenario evaluation The followin
18、g parts are under preparation: Part 3: Modality-specific testing Technical Report Part 4: Performance and interoperability testing of data interchange formats Part 5: Performance of biometric access control systems ISO/IEC 19795-2:2007(E) ISO/IEC 2007 All rights reserved vIntroduction This part of I
19、SO/IEC 19795 addresses two specific biometric performance testing methodologies: technology and scenario evaluation. The large majority of biometric tests are of one of these two generic evaluation types. Technology evaluations evaluate enrolment and comparison algorithms by means of previously coll
20、ected corpuses, while scenario evaluations evaluate sensors and algorithms by processing of samples collected from Test Subjects in real time. The former is intended for generation of large volumes of comparison scores and candidate lists indicative of the fundamental discriminating power of an algo
21、rithm. The latter is intended for measurement of performance in modeled environments, inclusive of Test Subject-system interactions. This part of ISO/IEC 19795 builds on requirements and best practices specified in ISO/IEC 19795-1, which addresses specific philosophies and principles that can be app
22、lied over a broad range of test conditions. This part of ISO/IEC 19795 is meant to provide biometric system developers, deployers and end users with mechanisms for design, execution and reporting of biometric performance tests in a fashion that allows meaningful benchmarking of biometric performance
23、 within and across technologies, usage scenarios and environments. INTERNATIONAL STANDARD ISO/IEC 19795-2:2007(E) ISO/IEC 2007 All rights reserved 1Information technology Biometric performance testing and reporting Part 2: Testing methodologies for technology and scenario evaluation 1 Scope This par
24、t of ISO/IEC 19795 provides requirements and recommendations on data collection, analysis and reporting specific to two primary types of evaluation: technology evaluation and scenario evaluation. This part of ISO/IEC 19795 specifies requirements in the following areas: development and full descripti
25、on of protocols for technology and scenario evaluations; execution and reporting of biometric evaluations reflective of the parameters associated with biometric evaluation types. 2 Conformance A test shall claim conformance to either the technology evaluation or scenario evaluation clauses of this p
26、art of ISO/IEC 19795. The set of clauses to which a scenario test shall conform differs from the set of clauses to which a technology test shall conform. In addition, the set of clauses to which an identification-system test shall conform differs from the set of clauses to which a verification-syste
27、m test shall conform. To conform to this part of ISO/IEC 19795, an evaluation shall conform to clauses of this part of ISO/IEC 19795 as shown in Table 1. Table 1 Conformance for evaluation methodologies and comparison types Evaluation methodology Comparison type Required clauses Technology or scenar
28、io Identification or verification Clauses 5 and 8 Technology Identification All of Clause 6, except 6.3.3 Technology Verification All of Clause 6, except 6.3.4 Scenario Identification All of Clause 7, except 7.3.4 Scenario Verification All of Clause 7, except 7.3.5 3 Normative references The followi
29、ng referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO/IEC 19795-1, Information technology Biometric performanc
30、e testing and reporting Part 1: Principles and framework ISO/IEC 19795-2:2007(E) 2 ISO/IEC 2007 All rights reserved4 Terms and definitions For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1:2006 and the following apply. 4.1 Biometric data 4.1.1 biometric reference
31、template, model users stored reference measure based on features extracted from enrolment samples 4.2 Components of a biometric system 4.2.1 feature extractor apparatus that extracts features from a sample 4.2.2 biometric reference generator apparatus that transforms a sample into a biometric refere
32、nce 4.3 User interaction with a biometric system 4.3.1 acclimatization reduction, over the course of an evaluation, in a temporal condition of a biometric characteristic that may impact the ability of a sensor to process a sample 4.3.2 effort level number of presentations, attempts or transactions n
33、eeded to successfully enrol or match in a biometric system 4.3.3 enrolment attempt submission of one or more biometric samples for a Test Subject for the purpose of enrolment in a biometric system NOTE 1 One or more enrolment attempts may be permitted or required to constitute an enrolment transacti
34、on. An enrolment attempt may comprise one or more enrolment presentations. NOTE 2 See Annex B for illustration of the relationship between presentation, attempt and transaction. 4.3.4 enrolment attempt limit maximum number of attempts, or the maximum duration, a Test Subject is permitted before an e
35、nrolment transaction is terminated 4.3.5 enrolment presentation submission of an instance of a biometric characteristic for a Test Subject for the purpose of enrolment NOTE One or more enrolment presentations may be permitted or required to constitute an enrolment attempt. An enrolment presentation
36、may or may not result in an enrolment attempt. 4.3.6 enrolment presentation limit maximum number of presentations, or the maximum duration, a Test Subject is permitted before an enrolment attempt is terminated ISO/IEC 19795-2:2007(E) ISO/IEC 2007 All rights reserved 34.3.7 guidance direction provide
37、d by an Administrator to a Test Subject in the course of enrolment or recognition NOTE Guidance is separate from feedback provided by a biometric system or device in the course of enrolment or recognition, such as audible or visual presentation queues. 4.3.8 habituation degree of familiarity a Test
38、Subject has with a device NOTE A Test Subject having substantial familiarity with a biometric device, such as that gained in the course of employment, is referred to as a habituated Test Subject. 4.3.9 comparison attempt submission of one or more biometric samples for a Test Subject for the purpose
39、of comparison in a biometric system 4.3.10 comparison attempt limit maximum number of attempts, or the maximum duration, a Test Subject is permitted before a comparison transaction is terminated 4.3.11 comparison presentation submission of an instance of a single biometric characteristic for a Test
40、Subject for the purpose of comparison NOTE One or more comparison presentations may be permitted or required to constitute a comparison attempt. A comparison presentation may or may not result in a comparison attempt. 4.3.12 comparison presentation limit maximum number of presentations, or the maxim
41、um duration, a Test Subject is permitted before a comparison attempt is terminated 4.4 Performance measures 4.4.1 failure at source rate proportion of samples discarded from the corpus either manually or by use of an automated biometric system prior to use in a technology evaluation EXAMPLE A propor
42、tion of images collected in a face data collection effort may be discarded due to lack of a face in the image. 5 Overview of technology evaluations and scenario evaluations This standard addresses two types of evaluation methodologies: technology evaluations and scenario evaluations. A test report s
43、hall state whether it presents results from a technology evaluation, a scenario evaluation, or an evaluation that combines aspects of both technology and scenario evaluations. Technology evaluation is the offline evaluation of one or more algorithms for the same biometric modality using a pre-existi
44、ng or specially-collected corpus of samples. The utility of technology testing stems from its separation of the human-sensor acquisition interaction and the recognition process, whose benefits include the following: ISO/IEC 19795-2:2007(E) 4 ISO/IEC 2007 All rights reserved Ability to conduct full c
45、ross-comparison tests. Technology evaluation affords the possibility to use the entire testing population as claimants to the identities of all other members (i.e. impostors) and this allows estimates of false match rates to be made to on the order of one in N2, rather than one in N. Ability to cond
46、uct exploratory testing. Technology evaluation can be run with no real-time output demands, and is thus well-suited to research and development. For example, the effects of algorithmic improvements, changes in run time parameters such as effort levels and configurations, or different image databases
47、, can be measured in, essentially, a closed-loop improvement cycle. Ability to conduct multi-instance and multi-algorithmic testing. By using common test procedures, interfaces, and metrics, technology evaluation affords the possibility to conduct repeatable evaluations of multi-instance systems (e.
48、g. three views of a face) and multi-algorithmic (e.g. supplier A and supplier B) performance, or any combination thereof. Provided the corpus contains appropriate sample data, technology testing is potentially capable of testing all modules subsequent to the human-sensor interface, including: a qual
49、ity control and feedback module(s), signal processing module(s), image fusion module(s) (for multi-modal or multi-instance biometrics), feature extraction and normalization module(s), feature-level fusion module(s), comparison score computation and fusion module(s), and score normalization module(s). The nondeterministic aspects of the human-sensor interaction preclude true repeatability and this complicates com
