1、Designation: E3016 15Standard Guide forEstablishing Confidence in Digital Forensic Results by ErrorMitigation Analysis1This standard is issued under the fixed designation E3016; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the y
2、ear of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This guide provides a process for recognizing anddescribing both errors and limitations associated with toolsus
3、ed to support digital forensics. This is accomplished byexplaining how the concepts of errors and error rates should beaddressed in digital forensics. It is important for practitionersand stakeholders to understand that digital forensic techniquesand tools have known limitations, but those limitatio
4、ns havedifferences from errors and error rates in other forensicdisciplines. This guide proposes that confidence in digitalforensic results is best achieved by using an error mitigationanalysis approach that focuses on recognizing potential sourcesof error and then applying techniques used to mitiga
5、ting them,including trained and competent personnel using tested andvalidated methods and practices.2. Referenced Documents2.1 ISO Standard:2ISO/IEC 17025 General Requirements for the Competenceof Testing and Measurement Laboratories2.2 SWGDE Standards:3SWGDE Model Quality Assurance Manual for Digit
6、al Evi-denceSWGDE Standards and Controls Position PaperSWGDE/SWGIT Proficiency Test Program GuidelinesSWGDE/SWGIT Guidelines however, they often struggle to establish their confidence on ascientific basis. Some forensic disciplines use an error rate todescribe the chance of false positives, false ne
7、gatives, orotherwise inaccurate results when determining whether twosamples actually come from the same source. But in digitalforensics, there are fundamental differences in the nature ofmany processes that can make trying to use statistical errorrates inappropriate or misleading.1This guide is unde
8、r the jurisdiction of ASTM Committee E30 on ForensicSciences and is the direct responsibility of Subcommittee E30.12 on Digital andMultimedia Evidence.Current edition approved May 1, 2015. Published June 2015. DOI: 10.1520/E3016-15.2Available from American National Standards Institute (ANSI), 25 W.
9、43rd St.,4th Floor, New York, NY 10036, http:/www.ansi.org.3Available from the Scientific Working Group on Digital Evidence (SWDGE),https:/www.swgde.org.Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States14.2 The key point to keep in mind
10、 is the difference betweenrandom errors and systematic errors. Random errors are basedin natural processes and the inability to perfectly measurethem. Systematic errors, in contrast, are caused by imperfectimplementations. Digital forensics being based on computerscience is far more prone to systema
11、tic than random errors.Additionally, the rapid change in technology including theinnumerable permutations of hardware, software and firmwaremakes it close to impossible to address all situations.4.3 One fundamental difference between digital forensicsand other forensic disciplines is that many foren
12、sic disciplinestry to determine whether or not two artifacts are a match (forexample, from the same source), whereas digital forensicspredominantly endeavors to find multiple artifacts that mayshow or imply actions by an individual. An error rate for amatching task focuses on establishing how often
13、a falsepositive or a false negative occurs. Error rates for matchingtasks are often statistical in nature and may derive from takinga measurement or sample from a population. Conversely, indigital forensics, there is often a series of tasks, any one ofwhich could introduce error of a systematic rath
14、er thanstatistical nature. Even though there are errors, the errors indigital forensic tasks/processes are not always characterized ina useful or meaningful way by an error rate.4.4 For each digital forensic task, there is an underlyingalgorithm (how the task should be done) and an implementa-tion o
15、f the algorithm (how the task is done in software by atool). There can be different errors and error rates with both thealgorithm and the implementation. For example, hash algo-rithms used to determine if two files are identical have aninherent false positive rate, but the rate is so small as to bee
16、ssentially zero. Characterizing hashing algorithms with anerror rate is appropriate because the algorithms assume a fileselected at random for the population of all possible files.4.5 Once an algorithm is implemented in software, inaddition to the inherent error rate of the algorithm, theimplementat
17、ion may introduce systematic errors that are notstatistical in nature. Software errors manifest when somecondition is present either in the data or in the executionenvironment. It is often misleading to try to characterizesoftware errors in a statistical manner since such errors are notthe result of
18、 variations in measurement or sampling. Forexample, the software containing the hash algorithm may bebadly written and may produce the same hash every time aninput file starts with the symbol “$.”4.6 The primary types of errors found in digital forensic toolimplementations are:4.6.1 IncompletenessAl
19、l the relevant information has notbeen acquired or found by the tool. For example, an acquisitionmight be incomplete or not all relevant artifacts identified froma search.4.6.2 InaccuracyThe tool does not report accurate infor-mation. Specifically, the tool should not report things that arenot there
20、, should not group together unrelated items, andshould not alter data in a way that changes the meaning.Assessment of accuracy in digital forensic tool implementa-tions can be categorized as follows:4.6.2.1 ExistenceAre all reported artifacts reported aspresent actually present? For example, a fault
21、y tool might adddata that was not present in the original.4.6.2.2 AlterationDoes a forensic tool alter data in a waythat changes its meaning, such as updating an existing date-time stamp (for example, associated with a file or e-mailmessage) to the current date.4.6.2.3 AssociationDo all items associ
22、ated together actu-ally belong together? A faulty tool might incorrectly associateinformation pertaining to one item with a different, unrelateditem. For instance, a tool might parse a web browser history fileand incorrectly report that a web search on “how to murderyour wife” was executed 75 times
23、when in fact it was onlyexecuted once while “history of Rome” (the next item in thehistory file) was executed 75 times, erroneously associating thecount for the second search with the first search.4.6.2.4 CorruptionDoes the forensic tool detect and com-pensate for missing and corrupted data? Missing
24、 or corruptdata can arise from many sources, such as bad sectorsencountered during acquisition or incomplete deleted filerecovery or file carving. For example, a missing piece of datafrom an incomplete carving of the above web history file couldalso produce the same incorrect association.4.6.3 Misin
25、terpretationThe results have been incorrectlyunderstood. Misunderstandings of what certain informationmeans can result from a lack of understanding of the underly-ing data or from ambiguities in the way digital forensic toolspresent information.4.7 The basic strategy to develop confidence in the dig
26、italforensic results is to mitigate errors, including known errorrates, by applying tool testing and sound quality controlmeasures as described in this document including:4.7.1 Tool Testing:4.7.1.1 Determine applicable scenarios that have been con-sidered in tool testing.4.7.1.2 Assess known tool an
27、omalies and how they apply tothe current case.4.7.1.3 Find untested scenarios that introduce uncertainty intool results.4.7.2 Sound Quality Control Procedures:4.7.2.1 Tool performance verification.4.7.2.2 Personnel training, certification and regular profi-ciency testing.4.7.2.3 Follow written proce
28、dures and document any nec-essary deviations/exceptions.4.7.2.4 Laboratory accreditation.4.7.2.5 Technical/peer review.4.7.2.6 Technical and management oversight.4.7.2.7 Use multiple tools and methods.4.7.2.8 Maintain awareness of past and current problems.4.7.2.9 Reasonableness and consistency of r
29、esults for thecase context.4.8 A more formalized approach to handling potentialsources of error in digital forensic processes is needed in orderto address considerations such as those in Daubert.4.9 The error mitigation analysis process involves recogniz-ing sources of potential error, taking steps
30、to mitigate anyE3016 152errors, and employing a quality assurance approach of continu-ous human oversight and improvement. Rather than focusingonly on error rates, this more comprehensive approach takesinto account all of the careful measures that can be taken toensure that digital forensics process
31、es produce reliable results.When error rates can be calculated, they can and should beincluded in the overall error mitigation analysis.5. Procedures5.1 Mitigating errors in a digital forensics process begins byanswering the following questions:5.1.1 Are the techniques (for example, hashing algorith
32、msor string searching) used to process the evidence valid science?5.1.2 Are the implementations of the techniques (forexample, software or hardware tools) correct and appropriatefor the environment where they are used?5.1.3 Are the results of the tools interpreted correctly?5.2 Considering each of t
33、hese questions is critical to under-standing errors in digital forensics. The next three sectionsexplain the types of error associated with each question. In thefirst section, Techniques (5.3), the basic concept of error ratesis addressed along with a discussion of how error rates dependon a stable
34、population. The second section, Implementation ofTechniques in Tools (5.4), addresses systematic errors and howtool testing is used to find these errors. The third section, ToolUsage and Interpreting Results (5.5), summarizes how practi-tioners use the results of digital forensic tools. This overall
35、approach to handling errors in digital forensics helps addressDaubert considerations.5.3 TechniquesIn computer science, the techniques thatare the basis for digital processing includes copying bits andthe use of algorithms to search and manipulate data (forexample, recover files). These techniques c
36、an sometimes becharacterized with an error rate.5.3.1 Error RatesAn error rate has an explicit purpose toshow how strong the technique is and what its limitations are.There are many factors that can influence an error rateincluding uncertainties associated with physical measurements,algorithm weakne
37、sses, statistical probabilities, and humanerror.NOTE 1Systematic and Random Errors: Error rates for many proce-dures can be treated statistically, however not all types of experimentaluncertainty can be assessed by statistical analysis based on repeatedmeasurements. For this reason, uncertainties ar
38、e classified into twogroups: the random uncertainties, which can be treated statistically, andthe systematic uncertainties, which cannot.4The uncertainty of the resultsfrom software tools used in digital forensics is similar to the problems ofmeasurement in that there may be both a random component
39、(often fromthe underlying algorithm) and a systematic component (usually comingfrom the implementation).5.3.1.1 Error rates are one of the factors described inDaubert to ascertain the quality of the science in experttestimony.5The underlying computer techniques are compa-rable to the type of science
40、 that is described in Daubert.Are theunderlying techniques sound science or junk science?Are theyused appropriately? In computer science, the types of tech-niques used are different from DNA analysis or trace chemicalanalysis. In those sciences, the technique or method is oftenused to establish an a
41、ssociation between samples. Thesetechniques require a measurement of the properties of thesamples. Both the measurements of the samples and theassociations have random errors and are well described byerror rates.5.3.1.2 Differences between digital and other forensic dis-ciplines change how digital f
42、orensics uses error rates.There areerror rates associated with some digital forensic techniques.For example, there are false positive rates for cryptographichashing; however, the rate is so small as to be essentially zero.Similarly, many algorithms such as copying bits also have anerror rate that is
43、 essentially zero. See Appendix X1, X1.2 andX1.3, for a discussion of error rates associated with hashingand copying.5.3.2 Error Rates and PopulationsThere are other majordifferences between digital forensics and natural sciences-based forensic disciplines. In biology and chemistry-baseddisciplines,
44、 the natural components of a sample remain fairlystatic (for example, blood, hair, cocaine). Basic biology andchemistry do not change (although new drugs are developedand new means of processing are created). In contrast, infor-mation technology changes constantly. New types of drives(for example, s
45、olid-state drives) and applications (for example,Facebook) may radically differ from previous ones. There area virtually unlimited number of combinations of hardware,firmware, and software.5.3.2.1 The rapid and significant changes in informationtechnology lead to another significant difference. Erro
46、r rates, aswith other areas of statistics, require a “population.” One of thekey features of a statistical population is that it is stable, that is,the composition remains constant. This allows predictions tobe made. Since IT changes quickly and unpredictably, it isoften infeasible to statistically
47、describe a population in a usableway because, while the description may reflect an average overthe entire population, it may not be useful for individualsituations. See Note 2 for an example of this.NOTE 2Deleted File Recovery Example: File fragmentation is signifi-cant to the performance of the del
48、eted file recovery algorithm. If some filesystems have low fragmentation, many deleted files will be recoverable.However, if there is a large amount of fragmentation, the recovered fileswill tend to be mixtures of multiples files and therefore harder to recover.So the error rate will be low for the
49、algorithm applied to a drive with lowfragmentation and high for a drive with high fragmentation. If one tries tolook at a large number of drives to derive a single error rate, it would notbe applicable for a particular drive because each drive is very likely to bedifferent from the average. (The average will not address drives with eitherhigh or low fragmentation.) Furthermore, the error rate would not apply tosolid-state drives or other file systems.5.3.2.2 In examining these two differences (1) the virtu-ally infinite number of combinations, and (2) the rapid pac
