1、Photography Guidelines for reliable testing of digital still cameras PD ISO/TR 19247:2016 BSI Standards Publication WB11885_BSI_StandardCovs_2013_AW.indd 1 15/05/2013 15:06National foreword This Published Document is the UK implementation of ISO/TR 19247:2016. The UK participation in its preparation
2、 was entrusted to Technical Committee CPW/42, Photography. A list of organizations represented on this committee can be obtained on request to its secretary. This publication does not purport to include all the necessary provisions of a contract. Users are responsible for its correct application. Th
3、e British Standards Institution 2016. Published by BSI Standards Limited 2016 ISBN 978 0 580 85104 9 ICS 37.040.10 Compliance with a British Standard cannot confer immunity from legal obligations. This Published Document was published under the authority of the Standards Policy and Strategy Committe
4、e on 30 June 2016. Amendments/corrigenda issued since publication Date Text affected PUBLISHED DOCUMENT PD ISO/TR 19247:2016 ISO 2016 Photography Guidelines for reliable testing of digital still cameras Photographie Lignes directrices pour des essais fiables des camras numriques TECHNICAL REPORT ISO
5、/TR 19247 Reference number ISO/TR 19247:2016(E) First edition 2016-07-01 PD ISO/TR 19247:2016 ISO/TR 19247:2016(E)ii ISO 2016 All rights reserved COPYRIGHT PROTECTED DOCUMENT ISO 2016, Published in Switzerland All rights reserved. Unless otherwise specified, no part of this publication may be reprod
6、uced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below or ISOs member body in the country of the requester.
7、 ISO copyright office Ch. de Blandonnet 8 CP 401 CH-1214 Vernier, Geneva, Switzerland Tel. +41 22 749 01 11 Fax +41 22 749 09 47 copyrightiso.org www.iso.org PD ISO/TR 19247:2016 ISO/TR 19247:2016(E)Foreword iv Introduction v 1 Scope . 1 2 Tests based on visual analysis 1 2.1 General . 1 2.2 Tests u
8、nder conditions which are not replicable 2 2.3 Simple tests under conditions which are replicable (with few test persons) 2 3 Tests based on technical measurements . 3 3.1 Catalogue of individual technical aspects and suggestions 3 3.1.1 General 3 3.2 Test criteria 4 3.2.1 General 4 3.3 Test conditi
9、ons 4 3.4 Software and samples . 5 3.4.1 Test charts . 5 3.4.2 Evaluation of images . 5 3.5 Archiving and documentation . 6 3.6 Minimization of deviations . 6 3.7 Preparation of personnel for reliable testing . 6 Bibliography 8 ISO 2016 All rights reserved iii Contents Page PD ISO/TR 19247:2016 ISO/
10、TR 19247:2016(E) Foreword ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Standards is normally carried out through ISO technical committees. Each member body interested in a subj
11、ect for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IE
12、C) on all matters of electrotechnical standardization. The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the different types of ISO documents should be
13、noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives). Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO shall not be held responsible for identify
14、ing any or all such patent rights. Details of any patent rights identified during the development of the document will be in the Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents). Any trade name used in this document is information given for the convenienc
15、e of users and does not constitute an endorsement. For an explanation on the meaning of ISO specific terms and expressions related to conformit y assessment, as well as information about ISOs adherence to the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the
16、following URL: www.iso.org/iso/foreword.html. The committee responsible for this document is ISO/TC 42, Photography.iv ISO 2016 All rights reserved PD ISO/TR 19247:2016 ISO/TR 19247:2016(E) Introduction The motivation for producing these guidelines for testing digital still cameras (DSCs) is the inc
17、reasing use of tests by industries and the press. On one hand, the booming DSC market attracts published evaluations and comparisons of its products. On the other hand, complex and, more importantly, reliable testing needs expensive test equipment and experienced and knowledgeable testing technician
18、s. This guide provides direction and procedures for reliable testing of DSCs for those people with neither a photographic nor a scientific background. The growing commercial development of DSC test methods and equipment is additional motivation for producing these guidelines as it is often difficult
19、 to make specific assessments about the usability of new test equipment and methods and about the reliability of their results. For the credibility of test methods and equipment the disclosure of the measurement methods and the mathematical analysis used by the manufacturer of the test equipment is
20、important. These suggestions follow two goals in order to improve the current situation for the parties involved: a) Technical guidelines for the testing process This guide gives an overview of the important, useful and useable test criteria, procedures and basic evaluation for the users and produce
21、rs of test methods. It will help to avoid typical mistakes and will improve and secure the validity and reliability of the used methods. b) Guidelines for the interpretation of test results At the moment, there are no guidelines by which the receiver and user of test results, especially magazine rea
22、ders and digital camera producers, can estimate the reliability and practicability, or in extreme cases the credibility, of test results. This guide will provide support in securing a basis for the assessment, analysis and even discussion with the testers. Therefore, this guide will provide more tra
23、nsparency in this complex area. ISO 2016 All rights reserved v PD ISO/TR 19247:2016 Photography Guidelines for reliable testing of digital still cameras 1 Scope This Technical Report is applied to the reliable testing of digital cameras (DSCs). 2 Tests based on visual analysis 2.1 General The visual
24、 analysis of digital image data using test persons is not easy as it seems. This becomes more evident when one considers that digital images are not visible. In order to visualize an image, a device needs to be used, i.e. a monitor or a printer. In order for the correct interpretation of the image d
25、ata to be possible, it is important to assure that the visible representation created by the device doesnt have any imponderable or even unknown limitations. This means that the devices needs to be calibrated and profiled, and with the help of a suitable colour management workflow allow the correct
26、reproduction of the image. The tester needs to know about the limitations that can be introduced by the device especially in the areas of colour and resolution. Use DIN 15708-1 as a guideline to find out if an acceptably good colour reproduction is available. It is important to consider that in some
27、 cases not all colours from the test specimen can be correctly captured and that colours, which are accurately captured, are not always correctly reproduced on the display or paper. As a prerequisite for the test person, their ability to see especially with reference to dyschromatopsia (see Ishihara
28、 Test) and lack of 20/20 vision shall be given and tested. It is preferable that the test persons be composed of people of different sex and age, so that a representative selection of basic population can be achieved. When the testing starts, all test persons need to be relaxed and healthy. ISO 2046
29、2-3 describes the recommended procedure for subjective evaluation in the “quality ruler method”. Table 1 shows the magnitude of the influence of a single person in a group of persons on the median of the subjectively acquired evaluation results. This is derived from the standard deviation: (1) The f
30、actor 1 1in Table 1 represents the impact of a single person on the result based on the total number of observers. It is the standard deviation normalized to the square root of the sum of all squared deviations. (2) TECHNICAL REPORT ISO/TR 19247:2016(E) ISO 2016 All rights reserved 1 PD ISO/TR 19247
31、:2016 ISO/TR 19247:2016(E) Table 1 Influence of a single measurement on median of subjective results Number of test persons e 1 2 1,00 3 0,71 4 0,58 5 0,50 10 0,33 20 0,23 30 0,19 50 0,14 100 0,10 The number of test persons needs to be large enough. Using less than 5 subjects is generally insufficie
32、nt for a qualified evaluation. 2.2 Tests under conditions which are not replicable It is preferable that tests fulfil as many of the criteria outlined in 3.2 as possible so that the results are as meaningful and useful as possible. When using a comparative product test, those tests made under condit
33、ions that are not replicable are not acceptable. Tests run under conditions that are not replicable are usable only be used as a rough indication of the image quality and are not suitable for reliable conclusions. 2.3 Simple tests under conditions which are replicable (with few test persons) It is p
34、referable that the following conditions are ensured by those responsible for the testing of the evaluation of image quality under conditions that are replicable. a) The image display equipment needs to be calibrated and profiled accordingly. b) Provide complete and accurate documentation for the col
35、our space, reproduction devices, and reproduction media used. c) Test persons are at least 5. d) The image quality evaluation scale needs to be clearly defined before beginning the tests. e) All test criteria are to be collected on the same evaluation scale in order to avoid mistakes and confusing t
36、he test persons. f) As part of the test and before evaluators are asked to evaluate image samples, it is suggested that the evaluators are shown reference samples of nominally excellent quality and nominally poor quality on the quality evaluation scale with an explanation of images captured by the D
37、SC are ranked with respect to these reference images. g) The image quality aspect that is to be evaluated, needs to be represented in the motif captured by the device under test. Scenes that illustrate one or more of the following image quality aspects should be identified or created. Sample images
38、for reference and evaluation should be made from these scenes. 1) For the subjective evaluation of colour, an adequate number of relevant colours needs to be present in the photographed objects. This especially includes the green tones of different plants, different light and darker skin tones and b
39、lue tones, which occur in pictures of the sky when the sun is shining; as well as grey tones and highly saturated colours, for example in flowers. In addition, colour, which are mixed from the primary colours of cyan, magenta or 2 ISO 2016 All rights reserved PD ISO/TR 19247:2016 ISO/TR 19247:2016(E
40、) yellow represented in different brightness and saturation levels are useful because these often show significant deviations between the original and the captured image. 2) For the subjective evaluation of the dynamic range in the images, specular highlights reflecting off metallic surfaces and dar
41、k structures in shaded image areas can be useful. Sufficient detail is also important in the mid tones of the image. 3) For the subjective evaluation of the image sharpness, fine structures distributed throughout the image, including the corners of the image are useful, especially structures with va
42、rying orientation (for example Siemens stars or other contrast rich patterns like, for example, the bristles on a brush). 4) If the corner shading is to be subjectively evaluated, an evenly lit partial area in the image that stretches from the centre to the image corners is important. 5) The evaluat
43、ion of distortion in the image needs a regular grid of structures such as straight lines, which are suggested to be horizontally and vertically oriented (for example, in quadrangle test charts or edges of buildings). 6) If the chromatic aberration in the images is to be subjectively evaluated, then
44、it is useful to have contrast-rich and colour-neutral light to dark transitions in which, especially in the image corners, colour fringes can be identified. 7) For the subjective evaluation of flare, images can be taken in which light sources outside of the image field can be directed at the camera
45、lens. 8) For the subjective evaluation of face recognition functions, it can be useful to capture several photographed faces at a time using different object distances. The situation is made more difficult when a light and contrast-rich scene with an obviously different distance to that of the faces
46、 is placed in the centre of the image. h) If using fewer than 20 test persons, it is best that each person evaluates each image twice. i) The sequence of the images to be examined is suggested to be randomly altered before each cycle in order to prevent a systematic bias. j) In each cycle, it is pre
47、ferable that not more than 15 images to 25 images are examined depending on the number of criteria to be evaluated. After this, it is preferable that test persons are given a suitable break. k) In each cycle, it is preferable that there are at least one good and one bad reference image of a previous
48、ly determined quality included. l) Drastic deviations by individual test persons need to be reported to those responsible for the testing and discussed with those persons. If this deviation remains, then it is preferable that this should be commented and documented. NOTE The literature sources on th
49、e subject of subjective evaluation of the image quality are Reference 19 and the ISO 20462 series. 3 Tests based on technical measurements 3.1 Catalogue of individual technical aspects and suggestions 3.1.1 General The following technical aspects are relevant for the characterization of cameras. Depending on the use case of the camera, the individual aspects are more or less important. Based on the importance, a subset of the aspects
