1、 INCITS/ISO/IEC 19795-4:2008 2009 ISO/IEC 19795-4:2008 Information technology Biometric performance testing and reporting Part 4: Interoperability performance testing INCITS/ISO/IEC 19795-4:2008 2009 PDF disclaimer This PDF file may contain embedded typefaces. In accordance with Adobes licensing pol
2、icy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobes licensing policy. The ISO Central
3、 Secretariat accepts no liability in this area. Adobe is a trademark of Adobe Systems Incorporated. Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Every care has been taken
4、to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below. Adopted by INCITS (InterNational Committee for Information Technology Standards) as an American National S
5、tandard. Date of ANSI Approval: 8/27/2009 Published by American National Standards Institute, 25 West 43rd Street, New York, New York 10036 Copyright 2009 by Information Technology Industry Council (ITI). All rights reserved. These materials are subject to copyright claims of International Standardi
6、zation Organization (ISO), International Electrotechnical Commission (IEC), American National Standards Institute (ANSI), and Information Technology Industry Council (ITI). Not for resale. No part of this publication may be reproduced in any form, including an electronic retrieval system, without th
7、e prior written permission of ITI. All requests pertaining to this standard should be submitted to ITI, 1250 Eye Street NW, Washington, DC 20005. Printed in the United States of America ii ITIC 2009 All rights reserved ISO/IEC 19795-4:2008(E) ISO/IEC 2008 All rights reserved iiiContents Page Forewor
8、d vi Introduction.vii 1 Scope1 2 Conformance .1 3 Normative references2 4 Terms and definitions .2 5 Abbreviated terms .4 6 Goals.5 6.1 Coverage 5 6.2 Target application8 6.2.1 Biometric application8 6.2.2 Interoperable application9 6.3 Purpose 10 6.3.1 Interoperability testing10 6.3.2 Sufficiency t
9、esting.11 7 Metrics 12 7.1 General .12 7.2 Figures of merit .12 7.2.1 Recognition performance figure of merit12 7.2.2 Measuring component failure 13 7.3 Interoperability matrices.14 7.3.1 General .14 7.3.2 Interoperability with sBDB generators14 7.3.3 Interoperability with sBDB generators15 7.3.4 Fi
10、xed operating point interoperability.16 7.3.5 Reporting failure of sBDB generators.16 7.4 Proprietary performance.16 8 Conducting a test 17 8.1 Structure of test.17 8.2 Sample data .17 8.2.1 Acquisition .17 8.2.2 Representative data 18 8.2.3 Collection of ancillary data.18 8.2.4 Corpus size 18 8.2.5
11、 Removal of subject-specific metadata 18 8.2.6 Removal of unrepresentative metadata 18 8.2.7 Origin of samples 19 8.2.8 Untainted samples.19 8.2.9 Sequestered data.19 8.3 Conformance testing.19 8.3.1 Conformance .19 8.3.2 Executing conformance tests 19 8.3.3 Reporting20 8.4 Constraints on the sBDBs2
12、0 8.4.1 Optional encodings .20 8.4.2 Optional encodings from profile standards20 8.4.3 Deviation from the base standard .20 8.4.4 Data encapsulation20 ISO/IEC 19795-4:2008(E) iv ISO/IEC 2008 All rights reserved8.5 Components.21 8.5.1 Components for sufficiency testing 21 8.5.2 Establishing modularit
13、y requirements 21 8.5.3 Components for interoperability testing .21 8.5.4 Underlying algorithms.21 8.5.5 Capture device user interfaces 21 8.5.6 Multimodal components .22 8.5.7 Component variability .22 8.5.8 Component reporting requirements 22 8.6 Planning decisions 22 8.6.1 Computational intensity
14、22 8.6.2 Supplier recruitment23 8.6.3 Provision of samples to suppliers .23 8.6.4 Equivalency of generator resources23 8.6.5 Handling violations of test requirements24 8.6.6 Comparison subsystem output data encapsulation24 8.6.7 Fundamental generator requirement.24 8.6.8 Fundamental comparison subsy
15、stem requirement .25 8.6.9 General requirements on software implementations.25 8.7 Prevention and detection of gaming26 8.7.1 General aspects .26 8.7.2 Modes of gaming .26 8.7.3 Prevention and detection of gaming28 8.8 Test procedure.29 8.8.1 Primary test 29 8.8.2 Uncertainty measurement.30 8.8.3 Va
16、riance estimation.30 8.8.4 Remedial testing 30 8.8.5 Survey of configurable parameters .30 9 Interpretation of the interoperability matrix30 9.1 Determination of interoperable subsystems 30 9.1.1 General30 9.1.2 Identifying interoperable combinations of subsystems31 9.1.3 Acceptable numbers of inter
17、operable subsystems .33 9.1.4 Combinatorial search for maximum interoperability-classes.33 9.1.5 Multiple interoperable subgroups34 9.1.6 Statistical stability of the test result 34 9.2 Interoperability with previously certified products35 9.2.1 Decertification considerations .35 9.2.2 Continuity of
18、 testing35 9.2.3 Interoperability with previously certified generators.35 9.2.4 Interoperability with previously certified comparison subsystems.36 9.2.5 Treatment of systematic effects.36 9.2.6 Retroactive exclusion from analysis .37 9.3 Overall sufficiency.37 Annex A (informative) Procedures for c
19、onducting a test of sufficiency and/or interoperability38 Annex B (informative) Example Interoperability Test42 Bibliography 45 Figure 1 General biometric interoperability 6 Figure 2 Specific interoperability: enrolment BDB is standardized .6 Figure 3 Specific interoperability: enrolment BDB is prop
20、rietary.7 Figure 4 Offline interoperability testing.7 Figure 5 Biometric capture device interoperability 8 ISO/IEC 19795-4:2008(E) ISO/IEC 2008 All rights reserved vFigure 6 Cells of an example interoperability space10 Figure 7 Sufficiency testing: proprietary vs. standard interchange formats 12 Fig
21、ure 8 Cross-generator performance matrix .15 Figure 9 Example performance matrix 15 Figure 10 Proprietary performance matrix16 Table 1 Conformity with ISO/IEC 19795-2 .1 Table 2 Sample size adjustment of error rate requirement.31 Table 3 Confidence levels of the standard Normal distribution.32 Table
22、 A.1 Interoperability test procedure, phase 1: planning.38 Table A.2 Interoperability test procedure, phase 2: setup39 Table A.3 Interoperability test procedure, phase 3: sBDB and pBDB generation39 Table A.4 Interoperability test procedure, phase 4: verification 40 Table A.5 Interoperability test pr
23、ocedure, phase 5: identification .40 Table A.6 Interoperability test procedure, phase 6: reporting41 Table A.7 Interoperability test procedure, phase 7: variance estimation41 ISO/IEC 19795-4:2008(E) vi ISO/IEC 2008 All rights reservedForeword ISO (the International Organization for Standardization)
24、and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal
25、with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC
26、have established a joint technical committee, ISO/IEC JTC 1. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the jo
27、int technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote. Attention is drawn to the possibility that some of the elements of this document may be the subject of patent right
28、s. ISO and IEC shall not be held responsible for identifying any or all such patent rights. ISO/IEC 19795-4 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics. ISO/IEC 19795 consists of the following parts, under the general title Informat
29、ion technology Biometric performance testing and reporting: Part 1: Principles and framework Part 2: Testing methodologies for technology and scenario evaluation Part 3: Modality-specific testing Technical Report Part 4: Interoperability performance testing Part 6: Testing methodologies for operatio
30、nal evaluation is under preparation. ISO/IEC 19795-4:2008(E) ISO/IEC 2008 All rights reserved viiIntroduction The multi-part biometric data interchange format standard, ISO/IEC 19794, has been developed to foster interoperable exchange of biometric data. By defining open containers for image, signal
31、 and feature data, and constraining some of the properties of the samples, the standards enhance interoperability by requiring implementers to be able to handle a restricted set of all possible biometric samples. Examples of this are the template standards of ISO/IEC 19794-2 and ISO/IEC 19794-8 whic
32、h embed compact processed data from fingerprint images. Only samples of the same format type (several of which can be defined in the same part of ISO/IEC 19794) are intended to be interchangeable. One common assertion prior to SC 37s formulation of data interchange standards was that proprietary tem
33、plates offer greater recognition performance than any likely standard on the grounds that the proprietary instances are the product of processes that embed considerable, private, intellectual property. The question of whether the emerging standards are sufficient then arises: that is, do they code d
34、ata (feature, image, etc.) representations that allow matching with accuracy comparable to that available from the proprietary solutions? A second issue, interoperability, arises in those applications where standardized data are generated and matched by different institutions and systems. If a compa
35、nys feature extraction subsystem processes acquired samples to produce ISO/IEC 19794-x compliant instances, then can other companies comparison subsystems attain performance comparable with that obtained from the originators own comparison subsystem? A further question is then whether a third compan
36、y can successfully recognize enrolment and user samples from two different sources. This part of ISO/IEC 19795 defines tests to specifically address absolute performance, sufficiency, and interoperability available from biometric data formatted to comply with established standards, particularly thos
37、e developed in the various parts of ISO/IEC 19794. However, because this part of ISO/IEC 19795 references interchange formats generically, by referencing only their black box generation and use, it also applies to other open standards. One consequence of this approach is that the success of a test i
38、s predicated on the correctness and appropriateness of lower-level data elements and values, i.e. conformance to the respective standards. Therefore, the approach here is to require conformance testing as an integral part of the test. This is achieved by referencing formal published conformance test
39、s or profiles of standards. For instance, an interoperability test of the ISO/IEC 19794-5 face format might reference an application profile of its Token image, which in turn might rely on ISO/IEC 15444-1 (JPEG 2000 core coding system). This part of ISO/IEC 19795 conceives of the following three kin
40、ds of tests: online: a scenario test in which a volunteer population enrols on suppliers products and subsequently uses suppliers verification or identification implementations to make genuine and impostor attempts; offline: a technology test in which an archived corpus of captured samples, not nece
41、ssarily collected with any intent to simulate the operational conditions of a particular application, is used as input to suppliers enrolment, verification or identification products to make genuine and impostor attempts; hybrid: a test in which the sample corpus is collected online under conditions
42、 which attempt to simulate the operational conditions of a particular application, and is then processed offline. In each case, an interoperability test needs to embed multi-supplier generation, exchange, and comparison of samples of the standard interchange format. Online collection from a live pop
43、ulation is appropriate when the biometric capture device, and/or the subject interaction with the biometric capture device, is considered to have a material effect on the interoperable performance of the intended application. An offline test is appropriate when a representative corpus of samples is
44、already available (for example passport photographs to be converted into Token instances of ISO/IEC 19794-5). An offline test may be appropriate when the collection of representative data is neither practical nor necessary to determine the interoperable performance of specific subsystems, such as fe
45、ature extraction and/or comparison. ISO/IEC 19795-4:2008(E) viii ISO/IEC 2008 All rights reservedIn all cases, an interoperability test must enrol subjects on one or more products and verify or identify on one or more others. This should involve subjects making transactions as themselves (genuine tr
46、ials) and as one or more other people (impostor trials). If a large enough population is available, a disjoint impostor population can be used. Since online tests can become onerous on the test population when many products and impostor attempts are needed, hybrid and offline testing allow execution
47、 of many zero-effort impostor attempts. In an interoperability performance test, J generators of standardized biometric data blocks (BDBs) are applied to the samples assembled as part of a hybrid or offline test. By applying K comparison subsystems to the standard BDBs, up to KJ2verification or iden
48、tification trials are conducted, each following ISO/IEC 19795-2. The BDB may be an image or signal, or a standardized template. Optional encodings allowed by the standard interchange format should be fully specified. This might be achieved by normatively referencing one of the ISO/IEC 24713-x profil
49、es. If the format in question is an image, a subsequent internal (usually proprietary) template would be used, but its existence here is subsumed by the notion of a black-box comparison of two instances of the given format. The test advanced by this part of ISO/IEC 19795 demarcates the generic aspects of interoperability from the meaning associated with each particular biometric format of I
