1、Designation: E3016 151Standard Guide forEstablishing Confidence in Digital Forensic Results by ErrorMitigation Analysis1This standard is issued under the fixed designation E3016; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the
2、year of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1NOTEEditorial changes were made throughout in September 2016.1. Scope1.1 This guide provides a process for recognizing an
3、ddescribing both errors and limitations associated with toolsused to support digital forensics. This is accomplished byexplaining how the concepts of errors and error rates should beaddressed in digital forensics. It is important for practitionersand stakeholders to understand that digital forensic
4、techniquesand tools have known limitations, but those limitations havedifferences from errors and error rates in other forensicdisciplines. This guide proposes that confidence in digitalforensic results is best achieved by using an error mitigationanalysis approach that focuses on recognizing potent
5、ial sourcesof error and then applying techniques used to mitigating them,including trained and competent personnel using tested andvalidated methods and practices.2. Referenced Documents2.1 ISO Standard:2ISO/IEC 17025 General Requirements for the Competenceof Testing and Measurement Laboratories2.2
6、SWGDE Standards:3SWGDE Model Quality Assurance Manual for Digital Evi-denceSWGDE Standards and Controls Position PaperSWGDE/SWGIT Proficiency Test Program GuidelinesSWGDE/SWGIT Guidelines however, they often struggle to establish their confidence on ascientific basis. Some forensic disciplines use a
7、n error rate todescribe the chance of false positives, false negatives, orotherwise inaccurate results when determining whether twosamples actually come from the same source. But in digitalforensics, there are fundamental differences in the nature of1This guide is under the jurisdiction of ASTM Comm
8、ittee E30 on ForensicSciences and is the direct responsibility of Subcommittee E30.12 on Digital andMultimedia Evidence.Current edition approved May 1, 2015. Published June 2015. DOI: 10.1520/E3016-15E01.2Available from American National Standards Institute (ANSI), 25 W. 43rd St.,4th Floor, New York
9、, NY 10036, http:/www.ansi.org.3Available from the Scientific Working Group on Digital Evidence (SWDGE),https:/www.swgde.org.Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States1many processes that can make trying to use statistical errorr
10、ates inappropriate or misleading.4.2 The key point to keep in mind is the difference betweenrandom errors and systematic errors. Random errors are basedin natural processes and the inability to perfectly measurethem. Systematic errors, in contrast, are caused by imperfectimplementations. Digital for
11、ensics being based on computerscience is far more prone to systematic than random errors.Additionally, the rapid change in technology including theinnumerable permutations of hardware, software and firmwaremakes it close to impossible to address all situations.4.3 One fundamental difference between
12、digital forensicsand other forensic disciplines is that many forensic disciplinestry to determine whether or not two artifacts are a match (forexample, from the same source), whereas digital forensicspredominantly endeavors to find multiple artifacts that mayshow or imply actions by an individual. A
13、n error rate for amatching task focuses on establishing how often a falsepositive or a false negative occurs. Error rates for matchingtasks are often statistical in nature and may derive from takinga measurement or sample from a population. Conversely, indigital forensics, there is often a series of
14、 tasks, any one ofwhich could introduce error of a systematic rather thanstatistical nature. Even though there are errors, the errors indigital forensic tasks/processes are not always characterized ina useful or meaningful way by an error rate.4.4 For each digital forensic task, there is an underlyi
15、ngalgorithm (how the task should be done) and an implementa-tion of the algorithm (how the task is done in software by atool). There can be different errors and error rates with both thealgorithm and the implementation. For example, hash algo-rithms used to determine if two files are identical have
16、aninherent false positive rate, but the rate is so small as to beessentially zero. Characterizing hashing algorithms with anerror rate is appropriate because the algorithms assume a fileselected at random for the population of all possible files.4.5 Once an algorithm is implemented in software, inad
17、dition to the inherent error rate of the algorithm, theimplementation may introduce systematic errors that are notstatistical in nature. Software errors manifest when somecondition is present either in the data or in the executionenvironment. It is often misleading to try to characterizesoftware err
18、ors in a statistical manner since such errors are notthe result of variations in measurement or sampling. Forexample, the software containing the hash algorithm may bebadly written and may produce the same hash every time aninput file starts with the symbol “$”.4.6 The primary types of errors found
19、in digital forensic toolimplementations are:4.6.1 IncompletenessAll the relevant information has notbeen acquired or found by the tool. For example, an acquisitionmight be incomplete or not all relevant artifacts identified froma search.4.6.2 InaccuracyThe tool does not report accurate infor-mation.
20、 Specifically, the tool should not report things that arenot there, should not group together unrelated items, andshould not alter data in a way that changes the meaning.Assessment of accuracy in digital forensic tool implementa-tions can be categorized as follows:4.6.2.1 ExistenceAre all reported a
21、rtifacts reported aspresent actually present? For example, a faulty tool might adddata that was not present in the original.4.6.2.2 AlterationDoes a forensic tool alter data in a waythat changes its meaning, such as updating an existing date-time stamp (for example, associated with a file or e-mailm
22、essage) to the current date.4.6.2.3 AssociationDo all items associated together actu-ally belong together? A faulty tool might incorrectly associateinformation pertaining to one item with a different, unrelateditem. For instance, a tool might parse a web browser history fileand incorrectly report th
23、at a web search on “how to murderyour wife” was executed 75 times when in fact it was onlyexecuted once while “history of Rome” (the next item in thehistory file) was executed 75 times, erroneously associating thecount for the second search with the first search.4.6.2.4 CorruptionDoes the forensic t
24、ool detect and com-pensate for missing and corrupted data? Missing or corruptdata can arise from many sources, such as bad sectorsencountered during acquisition or incomplete deleted filerecovery or file carving. For example, a missing piece of datafrom an incomplete carving of the above web history
25、 file couldalso produce the same incorrect association.4.6.3 MisinterpretationThe results have been incorrectlyunderstood. Misunderstandings of what certain informationmeans can result from a lack of understanding of the underly-ing data or from ambiguities in the way digital forensic toolspresent i
26、nformation.4.7 The basic strategy to develop confidence in the digitalforensic results is to mitigate errors, including known errorrates, by applying tool testing and sound quality controlmeasures as described in this document including:4.7.1 Tool Testing:4.7.1.1 Determine applicable scenarios that
27、have been con-sidered in tool testing.4.7.1.2 Assess known tool anomalies and how they apply tothe current case.4.7.1.3 Find untested scenarios that introduce uncertainty intool results.4.7.2 Sound Quality Control Procedures:4.7.2.1 Tool performance verification.4.7.2.2 Personnel training, certifica
28、tion and regular profi-ciency testing.4.7.2.3 Follow written procedures and document any nec-essary deviations/exceptions.4.7.2.4 Laboratory accreditation.4.7.2.5 Technical/peer review.4.7.2.6 Technical and management oversight.4.7.2.7 Use multiple tools and methods.4.7.2.8 Maintain awareness of pas
29、t and current problems.4.7.2.9 Reasonableness and consistency of results for thecase context.4.8 A more formalized approach to handling potentialsources of error in digital forensic processes is needed in orderto address considerations such as those in Daubert.E3016 15124.9 The error mitigation anal
30、ysis process involves recogniz-ing sources of potential error, taking steps to mitigate anyerrors, and employing a quality assurance approach of continu-ous human oversight and improvement. Rather than focusingonly on error rates, this more comprehensive approach takesinto account all of the careful
31、 measures that can be taken toensure that digital forensics processes produce reliable results.When error rates can be calculated, they can and should beincluded in the overall error mitigation analysis.5. Procedures5.1 Mitigating errors in a digital forensics process begins byanswering the followin
32、g questions:5.1.1 Are the techniques (for example, hashing algorithmsor string searching) used to process the evidence valid science?5.1.2 Are the implementations of the techniques (forexample, software or hardware tools) correct and appropriatefor the environment where they are used?5.1.3 Are the r
33、esults of the tools interpreted correctly?5.2 Considering each of these questions is critical to under-standing errors in digital forensics. The next three sectionsexplain the types of error associated with each question. In thefirst section, Techniques (5.3), the basic concept of error ratesis addr
34、essed along with a discussion of how error rates dependon a stable population. The second section, Implementation ofTechniques in Tools (5.4), addresses systematic errors and howtool testing is used to find these errors. The third section, ToolUsage and Interpreting Results (5.5), summarizes how pra
35、cti-tioners use the results of digital forensic tools. This overallapproach to handling errors in digital forensics helps addressDaubert considerations.5.3 TechniquesIn computer science, the techniques thatare the basis for digital processing includes copying bits andthe use of algorithms to search
36、and manipulate data (forexample, recover files). These techniques can sometimes becharacterized with an error rate.5.3.1 Error RatesAn error rate has an explicit purpose toshow how strong the technique is and what its limitations are.There are many factors that can influence an error rateincluding u
37、ncertainties associated with physical measurements,algorithm weaknesses, statistical probabilities, and humanerror.NOTE 1Systematic and Random Errors: Error rates for many proce-dures can be treated statistically, however not all types of experimentaluncertainty can be assessed by statistical analys
38、is based on repeatedmeasurements. For this reason, uncertainties are classified into twogroups: the random uncertainties, which can be treated statistically, andthe systematic uncertainties, which cannot.4The uncertainty of the resultsfrom software tools used in digital forensics is similar to the p
39、roblems ofmeasurement in that there may be both a random component (often fromthe underlying algorithm) and a systematic component (usually comingfrom the implementation).5.3.1.1 Error rates are one of the factors described inDaubert to ascertain the quality of the science in experttestimony.5The un
40、derlying computer techniques are compa-rable to the type of science that is described in Daubert.Are theunderlying techniques sound science or junk science?Are theyused appropriately? In computer science, the types of tech-niques used are different from DNA analysis or trace chemicalanalysis. In tho
41、se sciences, the technique or method is oftenused to establish an association between samples. Thesetechniques require a measurement of the properties of thesamples. Both the measurements of the samples and theassociations have random errors and are well described byerror rates.5.3.1.2 Differences b
42、etween digital and other forensic dis-ciplines change how digital forensics uses error rates.There areerror rates associated with some digital forensic techniques.For example, there are false positive rates for cryptographichashing; however, the rate is so small as to be essentially zero.Similarly,
43、many algorithms such as copying bits also have anerror rate that is essentially zero. See Appendix X1, X1.2 andX1.3, for a discussion of error rates associated with hashingand copying.5.3.2 Error Rates and PopulationsThere are other majordifferences between digital forensics and natural sciences-bas
44、ed forensic disciplines. In biology and chemistry-baseddisciplines, the natural components of a sample remain fairlystatic (for example, blood, hair, cocaine). Basic biology andchemistry do not change (although new drugs are developedand new means of processing are created). In contrast, infor-matio
45、n technology changes constantly. New types of drives(for example, solid-state drives) and applications (for example,Facebook) may radically differ from previous ones. There area virtually unlimited number of combinations of hardware,firmware, and software.5.3.2.1 The rapid and significant changes in
46、 informationtechnology lead to another significant difference. Error rates, aswith other areas of statistics, require a “population.” One of thekey features of a statistical population is that it is stable, that is,the composition remains constant. This allows predictions tobe made. Since IT changes
47、 quickly and unpredictably, it isoften infeasible to statistically describe a population in a usableway because, while the description may reflect an average overthe entire population, it may not be useful for individualsituations. See Note 2 for an example of this.NOTE 2Deleted File Recovery Exampl
48、e: File fragmentation is signifi-cant to the performance of the deleted file recovery algorithm. If some filesystems have low fragmentation, many deleted files will be recoverable.However, if there is a large amount of fragmentation, the recovered fileswill tend to be mixtures of multiples files and
49、 therefore harder to recover.So the error rate will be low for the algorithm applied to a drive with lowfragmentation and high for a drive with high fragmentation. If one tries tolook at a large number of drives to derive a single error rate, it would notbe applicable for a particular drive because each drive is very likely to bedifferent from the average. (The average will not address drives with eitherhigh or low fragmentation.) Furthermore, the error rate would not apply tosolid-state drives or other file systems.5.3.2.2 In examining these two differences (1
