1、BSI Standards Publication PD ISO/IEC TR 29189:2015 Information technology Biometrics Evaluation of examiner assisted biometric applicationsPD ISO/IEC TR 29189:2015 PUBLISHED DOCUMENT National foreword This Published Document is the UK implementation of ISO/IEC TR 29189:2015. The UK participation in
2、its preparation was entrusted to Technical Committee IST/44, Biometrics. A list of organizations represented on this committee can be obtained on request to its secretary. This publication does not purport to include all the necessary provisions of a contract. Users are responsible for its correct a
3、pplication. The British Standards Institution 2015. Published by BSI Standards Limited 2015 ISBN 978 0 580 75671 9 ICS 35.040 Compliance with a British Standard cannot confer immunity from legal obligations. This Published Document was published under the authority of the Standards Policy and Strate
4、gy Committee on 31 July 2015. Amendments issued since publication Date Text affectedPD ISO/IEC TR 29189:2015 Information technology Biometrics Evaluation of examiner assisted biometric applications Technologies de linformation Biomtrie valuation des applications biomtriques assistes par un examinate
5、ur TECHNICAL REPORT ISO/IEC TR 29189 First edition 2015-06-15 Reference number ISO/IEC TR 29189:2015(E) ISO/IEC 2015 PD ISO/IEC TR 29189:2015ii ISO/IEC 2015 All rights reserved COPYRIGHT PROTECTED DOCUMENT ISO/IEC 2015, Published in Switzerland All rights reserved. Unless otherwise specified, no par
6、t of this publication may be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below or ISOs member bo
7、dy in the country of the requester. ISO copyright office Ch. de Blandonnet 8 CP 401 CH-1214 Vernier, Geneva, Switzerland Tel. +41 22 749 01 11 Fax +41 22 749 09 47 copyrightiso.org www.iso.org ISO/IEC TR 29189:2015(E)PD ISO/IEC TR 29189:2015ISO/IEC TR 29189:2015(E)Foreword iv Introduction v 1 Scope
8、. 1 2 T erms and definitions . 1 3 Symbols and abbreviated terms . 2 4 Example of an examiner assisted search process 2 5 Factors to consider when evaluating examiner assisted biometric applications .4 5.1 General . 4 5.2 System-related factors to consider when evaluating examiner assisted biometric
9、 applications 5 5.2.1 Dependencies in the flow process Where does the examiner interact with the system? 5 5.2.2 System and stage-level performance measurement . 5 5.2.3 Measuring true operational performance . 7 5.2.4 The impact of prior probabilities on human performance . 8 5.2.5 Confidence Level
10、s . 8 5.2.6 The impact of automated systems on human performance . 9 5.3 Examiner-related factors to consider when evaluating examiner assisted biometric applications 9 5.3.1 An Examiners perception of the systems accuracy . 9 5.3.2 Usability and examiner acceptance 10 5.3.3 Training and expertise .
11、10 5.3.4 Workload .11 5.3.5 Bias in decision making 11 5.3.6 Individual differences between examiners 12 6 Performance evaluation of examiner assisted systems 12 6.1 Types of Evaluation 12 6.2 Performance measures for examiner assisted biometric systems 13 6.2.1 Introduction 13 6.2.2 Measures of acc
12、uracy 13 6.2.3 Examiner-assisted performance considerations in watch list scenarios .14 6.2.4 Discrimination and bias 14 6.2.5 Examiner Decision Confidence .14 6.2.6 Processing speed 14 6.3 Usability assessment .15 6.3.1 Introduction 15 6.3.2 Qualitative observations 15 6.3.3 Questionnaires 15 6.3.4
13、 Interviews and focus groups 15 6.4 Reporting results 15 6.5 Applying controls in evaluations 16 6.5.1 Introduction 16 6.5.2 Controls for examiner expertise 16 6.5.3 Controls for examiner decision bias .16 6.5.4 Controls for the test environment .17 6.5.5 Controls for variations in examiner input .1
14、7 6.6 Evaluation challenges .18 6.6.1 Introduction 18 6.6.2 Challenges with testing on a live operational system .18 6.6.3 Challenges in repeatable operational test .18 Bibliography .19 ISO/IEC 2015 All rights reserved iii Contents PagePD ISO/IEC TR 29189:2015ISO/IEC TR 29189:2015(E) Foreword ISO (t
15、he International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees
16、established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. I
17、n the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1. The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular the different approval criteria n
18、eeded for the different types of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives). Attention is drawn to the possibility that some of the elements of this document may be the subject of patent ri
19、ghts. ISO and IEC shall not be held responsible for identifying any or all such patent rights. Details of any patent rights identified during the development of the document will be in the Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents). Any trade name u
20、sed in this document is information given for the convenience of users and does not constitute an endorsement. For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment, as well as information about ISOs adherence to the WTO principles in the Technical
21、Barriers to Trade (TBT), see the following URL: Foreword Supplementary information. The committee responsible for this document is ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics.iv ISO/IEC 2015 All rights reservedPD ISO/IEC TR 29189:2015ISO/IEC TR 29189:2015(E) Introduction Bi
22、ometric identification systems such as those used in forensic applications are typically examiner assisted and not automated to the extent that most biometric systems are. This is particularly the case for applications such as latent fingerprint searching where sample quality can be so poor that the
23、 system requires human input. Key processes such as sample capture and preparation, enrolment, template generation, matching result adjudication, and final decision that would otherwise require minimal manual intervention are instead heavily reliant on input from experts (fingerprint examiners in th
24、e case of AFIS). These experts can interact with the system at each of these stages to prepare, launch, and/or review the results of biometric searches. The execution and performance of the “end-to-end” search process is thus, a combination of the examiners role (and capability) and the functionalit
25、y of the automated biometric system. This partially automated approach to biometrics using “examiner assisted” biometric systems provides value both in assisting the human examiner to perform their role more effectively, and in allowing the expertise of the human examiner to be exploited to assist t
26、he automated matching process. Therefore, such systems are most likely to be beneficial in non-real time scenarios where the search response is not necessarily required immediately but the throughput of the system is still high. Understanding the role of the examiner is crucial, as it impacts on the
27、 design of the system, the manner in which it is used, how it is tested, and how the system performance and its individual subcomponents are defined and measured. The main objectives of this Technical Report are to describe the characteristics of examiner assisted biometric applications and, where a
28、ppropriate, to contrast such applications with mainstream biometric applications. This Technical Report addresses the issues with assessing the system as a whole, or by testing the examiner assisted and automated elements separately. ISO/IEC 2015 All rights reserved vPD ISO/IEC TR 29189:2015PD ISO/I
29、EC TR 29189:2015Information technology Biometrics Evaluation of examiner assisted biometric applications 1 Scope The purpose of this Technical Report is to identify and characterize those aspects of performance testing that are unique to examiner assisted biometric applications. An examiner assisted
30、 biometric system has the following characteristics: reliant on the interaction and skill of a human examiner for one or more stages of the complete biometric process, be it data capture, enrolment, template generation, or final decision; can incorporate identification functionality, verification fu
31、nctionality, or both; will use a combination of the examiners input and the functionality of the biometric algorithm to execute the complete biometric process; will likely have inbuilt examination toolsets to assist the human examiner when enrolling biometric samples or when comparing the match resu
32、lts provided by the biometric algorithm. Although there is a wide variation in the use of the term “examiner” in the context of an “examiner assisted biometric system”, as defined in this Technical Report, an “examiner” typically has the following characteristics: field expert in the biometric modal
33、ity being exploited; trained to use the system to an advanced degree of proficiency; authorized to override the biometric systems decisions in particular when accepting or rejecting a match decision based on their own examination of the biometric samples and the results returned. Assessing an examin
34、ers level of expertise is excluded from the scope of this Technical Report. However, the skill of the examiner does have a major bearing on system performance and vice versa. Measuring or assessing the ability of an examiner to employ their skills might be necessary to properly evaluate the performa
35、nce of an examiner-assisted system. Other individuals, such as administrative users, or subjects whose biometrics are used within the system are not considered in this Technical Report. It is outside the scope of this Technical Report to consider non-expert examiners. 2 T erms a nd definiti ons For
36、the purposes of this document, the following terms and definitions apply. 2.1 examiner person responsible for examining biometric data and biometric system outputs for the purpose of either preparing data suitable for a system or confirming, overriding, or modifying a decision output from the biomet
37、ric system Note 1 to entry: This decision output could be a match decision or simply the location of a biometric feature point (e.g. a fingerprint core and delta points, or the location of eye co-ordinates on a facial image). TECHNICAL REPORT ISO/IEC TR 29189:2015(E) ISO/IEC 2015 All rights reserved
38、 1PD ISO/IEC TR 29189:2015ISO/IEC TR 29189:2015(E) 2.2 examiner assisted feature or quality of a process, application, system, or any other element that refers to the fact that an examiner takes part by contributing his/her knowledge and expertise 2.3 suspected match decision state indicating qualif
39、ied support on the part of an examiner that a match exists, based on the outcome of the examination process and on the limitations of the relevant comparable data 2.4 suspected non match decision state indicating qualified support on the part of an examiner that no match exists, based on the outcome
40、 of the examination process and on the limitations of the relevant comparable data 3 Symbols and abbreviated terms AFIS Automated Fingerprint Identification System 4 Example of an examiner assisted search process Consider the diagram below in Figure 1 which illustrates at a very high level, some of
41、the basic stages of a biometric search process. With the exception of the “search” which is fully automated, all other processes are potentially assisted by the interaction of a human examiner. Figure 2 shows each of the examiner assisted points in a diagram representing a generic biometric applicat
42、ion. Capture Edit Response(s) Encode Search Parameters Search Examiners Decision 1 3 4 5 2 Figure 1 Basic stages of a biometric search process2 ISO/IEC 2015 All rights reservedPD ISO/IEC TR 29189:2015ISO/IEC TR 29189:2015(E) 1 2 3 4 5 Figure 2 A generic biometric application highlighted to indicate
43、the examiner assisted points T o illustrate the importance of the examiner assisted stages, consider the role of forensic AFIS examiners. These are fingerprint experts, trained specifically to interact with the system, to fully exploit the functionality of the AFIS in order to prepare, launch, and r
44、eview the results of the biometric searches. Their interaction at each of these examiner assisted stages shown in Figure 1 can be described as follows. a) Capture: An image is scanned by an examiner or imported directly into the system. If multiple images are available the examiner may select the im
45、age(s) that they consider as (the most) suitable quality for searching. b) Edit and Encode: The image is displayed for viewing on a monitor and may be enhanced or edited by the examiner to improve the visibility, and subsequent placement, of features by an examiner. User interface tools are provided
46、 to enable the examiner to manually encode features such as fingerprint minutiae, cores, deltas, etc. The examiner may also override system decisions about the placement of features such as minutiae, based on their skill and expertise. Some systems may iterate this process to gradually improve the q
47、uality of the data with each cycle of manual and automated processing. c) Search Parameters: The examiner may specify search parameters to provide additional data to the matcher in order to maximise likelihood of the search resulting in a match if one exists in the database. Finger position, palm re
48、gion, orientation, or pattern type may typically be input by the examiner following careful study of the biometric data being searched, based on their domain specific knowledge. d) Responses: The matcher threshold may be manually configurable in order to adjust the number of responses returned. Alte
49、rnatively, the desired number of responses may be configurable directly, within some system-defined bounds. Some searches may be assigned a certain level of priority (over other searches) depending on the importance of the search outcome. An example of this may be a search conducted on a police system relating to a serious crime. e) Decision: When the output of the search is returned (typically as a ranked list of potential matches when used in a forensic context) the ex
