1、 INCITS/ISO/IEC 19795-6:2012 2013 ISO/IEC 19795-6:2012 Information technology Biometric performance testing and reporting Part 6: Testing methodologies for operational evaluation INCITS/ISO/IEC 19795-6:2012 2013 PDF disclaimer This PDF file may contain embedded typefaces. In accordance with Adobes l
2、icensing policy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobes licensing policy. The
3、 ISO Central Secretariat accepts no liability in this area. Adobe is a trademark of Adobe Systems Incorporated. Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Every care has
4、 been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below. Adopted by INCITS (InterNational Committee for Information Technology Standards) as an America
5、n National Standard. Date of ANSI Approval: 2/25/2013 Published by American National Standards Institute, 25 West 43rd Street, New York, New York 10036 Copyright 2013 by Information Technology Industry Council (ITI). All rights reserved. These materials are subject to copyright claims of Internation
6、al Standardization Organization (ISO), International Electrotechnical Commission (IEC), American National Standards Institute (ANSI), and Information Technology Industry Council (ITI). Not for resale. No part of this publication may be reproduced in any form, including an electronic retrieval system
7、, without the prior written permission of ITI. All requests pertaining to this standard should be submitted to ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of America ii ITIC 2013 All rights reserved ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved iiiContents
8、Page Foreword . v Introduction vi 1 Scope 1 2 Conformance . 1 3 Normative references 1 4 Terms and definitions . 2 5 Operational evaluation overview . 3 5.1 Operational evaluation goals . 3 5.2 Operational performance metrics 4 5.3 Operational evaluation methods 4 5.4 Determining operational perform
9、ance 4 5.5 Use of technology and scenario evaluation methodologies in evaluating operational systems 5 6 Operational evaluation 5 6.1 Purpose and scope . 5 6.1.1 General . 5 6.1.2 Criteria for system inclusion 5 6.1.3 System specification . 5 6.1.4 Biometric functionality 6 6.1.5 Performance measure
10、s . 6 6.2 Application characteristics 6 6.2.1 General . 6 6.2.2 Concept of operations 7 6.2.3 Guidance and instruction . 7 6.2.4 Levels of effort and decision policies . 8 6.2.5 Multiple-instance systems 8 6.2.6 Environment . 9 6.2.7 Deployment factors . 9 6.2.8 Acclimatization 10 6.2.9 Habituation
11、. 10 6.3 Test Plan . 10 6.3.1 General . 10 6.3.2 System implementation and configuration. 11 6.3.3 Test population 11 6.3.4 Test transactions . 12 6.4 Performance measurement 14 6.4.1 Throughput 14 6.4.2 Enrolment analysis 15 6.4.3 Recognition analysis . 15 6.5 Reporting 16 6.5.1 Reporting planned t
12、est results 16 6.5.2 Reporting additional analyses . 16 6.5.3 Reporting observations 17 6.5.4 Report structure 17 6.6 Record keeping 17 Annex A (informative) Non-mandatory performance metrics and reporting 18 Annex B (informative) Sub-transaction events in operational testing 20 ISO/IEC 19795-6:2012
13、(E) iv ISO/IEC 2012 All rights reservedAnnex C (informative) Sample operational test specification 21 Annex D (informative) Methods to determine test size .23 Annex E (informative) Operational system monitoring .25 Annex F (informative) Operational habituation testing .27 Annex G (informative) Sampl
14、e operational test report outline .28 Bibliography 30 ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved vForeword ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. Nationa
15、l bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. O
16、ther international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1. International Standards are drafted in accordance with the ru
17、les given in the ISO/IEC Directives, Part 2. The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires ap
18、proval by at least 75 % of the national bodies casting a vote. Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights. ISO/IEC 19795-6 was prepared b
19、y Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics. ISO/IEC 19795 consists of the following parts, under the general title Information technology Biometric performance testing and reporting: Part 1: Principles and framework Part 2: Testing methodologies
20、 for technology and scenario evaluation Part 3: Modality-specific testing Technical Report Part 4: Interoperability performance testing Part 5: Access control scenario and grading scheme Part 6: Testing methodologies for operational evaluation Part 7: Testing of on-card biometric comparison algorith
21、ms ISO/IEC 19795-6:2012(E) vi ISO/IEC 2012 All rights reservedIntroduction Operational tests evaluate complete biometric systems in the targeted operational environment with the target population. Tests may encompass performance monitoring of operational systems or assessment of performance in opera
22、tional trials. Operational performance assessment may be based on: data collected by the operational system in the course of normal operation; additional data collected during norma l system use, but with the system running in an “evaluation mode” allowing extra data to be collected; data collected
23、with a set of test subjects considered separately from the subject base of the operational system. Operational evaluation differs from technology or scenario evaluation in that the subject base, environment, and system design are no longer controlled for the purpose of repeatable testing, but vary i
24、n accordance with operational use. Examples of uncontrolled variables include the legitimacy of the subjects identity claim, environmental effects from weather or lighting, or the variability of system use by different individuals. The overarching goals of operational testing are to measure or monit
25、or operational biometric system performance over a period of time. Subgoals of operational testing may include: to determine if performance meets the requirements specified for a particular application or the claims asserted by the supplier; to determine the need to adjust or conf igure the system t
26、o improve performance; to predict performance as the numbers of subjects, locations, or devices increase; to obtain information on the target population and environmental parameters found to affect system performance; to obtain performance data from a pilot implementation; to obtain performance data
27、 to benchmark future systems. This part of ISO/IEC 19795 provides the test planning, test conduct, performance measurement, test reporting, and record keeping requirements to be followed during a biometric systems operational evaluation. INTERNATIONAL STANDARD ISO/IEC 19795-6:2012(E) ISO/IEC 2012 Al
28、l rights reserved 1Information technology Biometric performance testing and reporting Part 6: Testing methodologies for operational evaluation 1 Scope This part of ISO/IEC 19795: provides guidance on the operational testing of biometric systems; specifies performance metrics for operational systems;
29、 details data that may be retained by operational systems to enable performance monitoring; and specifies requirements on test methods, recording of data, and reporting of results of operational evaluations. NOTE Some operational biometric systems perform a single biometric function. For example, in
30、 the initial stages of rollout of biometric passports, the operational system might be performing biometric enrolment only. Operational evaluation of such systems is within the scope of this part of ISO/IEC 19795. This part of ISO/IEC 19795 does not: cover testing of operational sy stems in the labo
31、ratory or address vulnerability testing. 2 Conformance An operational evaluation is in conformance with this part of ISO/IEC 19795 if it is planned, executed and reported in accordance with the requirements of Clause 6. 3 Normative references The following referenced documents are indispensable for
32、the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO/IEC 19795-1, Information technology Biometric performance testing and reporting Part 1: Principles and
33、 framework ISO/IEC 19795-6:2012(E) 2 ISO/IEC 2012 All rights reserved4 Terms and definitions For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following apply. 4.1 acclimatization change, over the course of one or more transactions, of a biometric characte
34、ristic that might impact the ability of a system to process a sample NOTE Acclimatization is primarily associated with a subjects temporal adjustment to environmental effects, such as skin temperature. 4.2 attendant agent of the biometric system operator who directly interacts with the biometric cap
35、ture subject 4.3 biometric capture subject individual who is the subject of a biometric capture process 4.4 biometric data subject individual whose individualized biometric data is within the biometric system 4.5 biometric probe biometric data input to an algorithm for comparison to a biometric refe
36、rence(s) 4.6 biometric operational personnel individuals, other than the biometric capture subjects, who take an active role in the operation of the biometric system NOTE Biometric operational personnel includes biometric system administrators, attendants, and examiners. 4.7 biometric system adminis
37、trator person who executes policies and procedures in the administration of a biometric system 4.8 biometric system operator organization responsible for defining policies and procedures in the operation of a biometric system 4.9 biometric reference one or more stored biometric samples, biometric te
38、mplates or biometric models attributed to a biometric data subject and used for comparison 4.10 comparison attempt limit maximum allowed number or duration of attempts in a comparison transaction 4.11 enrolment attempt limit maximum allowed number or duration of attempts in an enrolment transaction
39、4.12 habituation familiarity a subject has with the biometric device, system and application NOTE The level of habituation can affect biometric sample presentation and acquisition device. ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved 34.13 subject base set of individuals whose biometric d
40、ata is intended to be enrolled or compared in operational use of a biometric system 4.14 system acceptance rate proportion of recognition transactions in an operational system in which the subject is recognized NOTE 1 Though the acceptance of an impostor is an incorrect recognition, it can still cou
41、nt as a system acceptance. NOTE 2 System acceptance rate = 1 system rejection rate. 4.15 system identification rate proportion of identification transactions in an operational system in which one or more subjects are identified 4.16 system rejection rate proportion of recognition transactions in an
42、operational system in which the subject is not recognized NOTE The system rejection rate differs from the false reject rate in that, in addition to false rejections, it also includes any rejected impostor transaction and any improper genuine transaction. 4.17 test crew member selected biometric data
43、 subject whose use of the operational system is controlled or monitored as part of the evaluation NOTE In an operational evaluation, test subjects can be subjects of the operational system or they can be members of a test crew using the system specifically for evaluation purposes 5 Operational evalu
44、ation overview 5.1 Operational evaluation goals The overarching goals of operational testing are to measure or monitor operational biometric system performance. Subgoals of operational testing may include: to determine if performance meets the requirements specified for a particular application or t
45、he claims asserted by the supplier; to determine the need to adjust or conf igure the system to improve performance; to predict performance as the numbers of subjects, locations, or devices increase; to obtain information on the target population and environmental parameters found to affect system p
46、erformance; to obtain performance data from a pilot implementation; to obtain performance data to benchmark future systems. Operational evaluation considers the performance of people as well as the equipment, algorithms, and environment. Consequently, operational testing includes aspects of social s
47、cience in addition to physical science, whereas technology testing does not. In general, operational performance will vary over time due to uncontrolled conditions in people, equipment and environment. For example, if the majority of subjects are enrolled at the start of operations, with few new enr
48、olments, performance of the system might improve as ISO/IEC 19795-6:2012(E) 4 ISO/IEC 2012 All rights reservedsubjects habituate or degrade as subjects biometric characteristics age over time away from their enrolled references. The performance observed in testing can depend on the operational perso
49、nnel, such as attendants or biometric examiners, as well as the biometric subjects. Operational personnel based factors should be taken into consideration in all aspects of the test from scope definition to reporting (see e.g., references 1 and 2). 5.2 Operational performance metrics Recognition metrics for operational testing differ from those used in technology and scenari
