1、Designation: E178 16a An American National StandardStandard Practice forDealing With Outlying Observations1This standard is issued under the fixed designation E178; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last r
2、evision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.NoteCorrections were made to Table 2 and the year date was changed on Sept. 7, 2016.1. Scope1.1 This practice covers outlying observatio
3、ns in samplesand how to test the statistical significance of outliers.1.2 The system of units for this standard is not specified.Dimensional quantities in the standard are presented only asillustrations of calculation methods. The examples are notbinding on products or test methods treated.1.3 This
4、standard does not purport to address all of thesafety concerns, if any, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and determine the applica-bility of regulatory requirements prior to use.2. Referenced Documents
5、2.1 ASTM Standards:2E456 Terminology Relating to Quality and StatisticsE2586 Practice for Calculating and Using Basic Statistics3. Terminology3.1 DefinitionsThe terminology defined in TerminologyE456 applies to this standard unless modified herein.3.1.1 order statistic x(k),nvalue of the kth observe
6、d valuein a sample after sorting by order of magnitude. E25863.1.1.1 DiscussionIn this practice, xkis used to denoteorder statistics in place of x(k), to simplify the notation.3.1.2 outliersee outlying observation.3.1.3 outlying observation, nan extreme observation ineither direction that appears to
7、 deviate markedly in value fromother members of the sample in which it appears.4. Significance and Use4.1 An outlying observation, or “outlier,” is an extreme onein either direction that appears to deviate markedly from othermembers of the sample in which it occurs.4.2 Statistical rules test the nul
8、l hypothesis of no outliersagainst the alternative of one or more actual outliers. Theprocedures covered were developed primarily to apply to thesimplest kind of experimental data, that is, replicate measure-ments of some property of a given material or observations ina supposedly random sample.4.3
9、A statistical test may be used to support a judgment thata physical reason does actually exist for an outlier, or thestatistical criterion may be used routinely as a basis to initiateaction to find a physical cause.5. Procedure5.1 In dealing with an outlier, the following alternativesshould be consi
10、dered:5.1.1 An outlying observation might be the result of grossdeviation from prescribed experimental procedure or an errorin calculating or recording the numerical value. When theexperimenter is clearly aware that a deviation from prescribedexperimental procedure has taken place, the resultant obs
11、erva-tion should be discarded, whether or not it agrees with the restof the data and without recourse to statistical tests for outliers.If a reliable correction procedure is available, the observationmay sometimes be corrected and retained.5.1.2 An outlying observation might be merely an extrememani
12、festation of the random variability inherent in the data. Ifthis is true, the value should be retained and processed in thesame manner as the other observations in the sample. Trans-formation of data or using methods of data analysis designedfor a non-normal distribution might be appropriate.5.1.3 T
13、est units that give outlying observations might be ofspecial interest. If this is true, once identified they should besegregated for more detailed study.5.2 In many cases, evidence for deviation from prescribedprocedure will consist primarily of the discordant value itself.In such cases it is advisa
14、ble to adopt a cautious attitude. Use ofone of the criteria discussed below will sometimes permit aclearcut decision to be made.1This practice is under the jurisdiction ofASTM Committee E11 on Quality andStatistics and is the direct responsibility of Subcommittee E11.10 on Sampling /Statistics.Curre
15、nt edition approved Sept. 7, 2016. Published September 2016. Originallyapproved in 1961. Last previous edition approved in 2016 as E178 16. DOI:10.1520/E0178-16A.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book
16、of ASTMStandards volume information, refer to the standards Document Summary page onthe ASTM website.Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States15.2.1 When the experimenter cannot identify abnormalconditions, he should report the
17、discordant values and indicateto what extent they have been used in the analysis of the data.5.3 Thus, as part of the over-all process of experimentation,the process of screening samples for outlying observations andacting on them is the following:5.3.1 Physical Reason Known or Discovered for Outlie
18、r(s):5.3.1.1 Reject observation(s) and possibly take additionalobservation(s).5.3.1.2 Correct observation(s) on physical grounds.5.3.2 Physical Reason UnknownUse Statistical Test:5.3.2.1 Reject observation(s) and possibly take additionalobservation(s).5.3.2.2 Transform observation(s) to improve fit
19、to a normaldistribution.5.3.2.3 Use estimation appropriate for non-normal distribu-tions.5.3.2.4 Segregate samples for further study.6. Basis of Statistical Criteria for Outliers6.1 In testing outliers, the doubtful observation is includedin the calculation of the numerical value of a sample criteri
20、on(or statistic), which is then compared with a critical valuebased on the theory of random sampling to determine whetherthe doubtful observation is to be retained or rejected. Thecritical value is that value of the sample criterion which wouldbe exceeded by chance with some specified (small) probab
21、ilityon the assumption that all the observations did indeed consti-tute a random sample from a common system of causes, asingle parent population, distribution or universe. The specifiedsmall probability is called the “significance level” or “percent-age point” and can be thought of as the risk of e
22、rroneouslyrejecting a good observation. If a real shift or change in thevalue of an observation arises from nonrandom causes (humanerror, loss of calibration of instrument, change of measuringinstrument, or even change of time of measurements, and soforth), then the observed value of the sample crit
23、erion used willexceed the “critical value” based on random-sampling theory.Tables of critical values are usually given for several differentsignificance levels. In particular for this practice, significancelevels 10, 5, and 1 % are used.NOTE 1In this practice, we will usually illustrate the use of t
24、he 5 %significance level. Proper choice of level in probability depends on theparticular problem and just what may be involved, along with the risk thatone is willing to take in rejecting a good observation, that is, if thenull-hypothesis stating “all observations in the sample come from thesame nor
25、mal population” may be assumed correct.6.2 Almost all criteria for outliers are based on an assumedunderlying normal (Gaussian) population or distribution. Thenull hypothesis that we are testing in every case is that allobservations in the sample come from the same normalpopulation. In choosing an a
26、ppropriate alternative hypothesis(one or more outliers, separated or bunched, on same side ordifferent sides, and so forth) it is useful to plot the data asshown in the dot diagrams of the figures. When the data are notnormally or approximately normally distributed, the probabili-ties associated wit
27、h these tests will be different. The experi-menter is cautioned against interpreting the probabilities tooliterally.6.3 Although our primary interest here is that of detectingoutlying observations, some of the statistical criteria presentedmay also be used to test the hypothesis of normality or that
28、 therandom sample taken come from a normal or Gaussianpopulation. The end result is for all practical purposes thesame, that is, we really wish to know whether we ought toproceed as if we have in hand a sample of homogeneousnormal observations.6.4 One should distinguish between data to be used toest
29、imate a central value from data to be used to assessvariability. When the purpose is to estimate a standarddeviation, it might be seriously underestimated by dropping toomany “outlying” observations.7. Recommended Criteria for Single Samples7.1 Criterion for a Single OutlierLet the sample of nobserv
30、ations be denoted in order of increasing magnitude by x1 x2 x3 . xn. Let the largest value, xn, be the doubtfulvalue, that is the largest value. The test criterion, Tn, for asingle outlier is as follows:Tn5 xn2 x!/s (1)where:x = arithmetic average of all n values, ands = estimate of the population s
31、tandard deviation based onthe sample data, calculated as follows:s =!(i51nxi2x!2n215!(i51nxi22nx2n215!(i51nxi22S(i51nxiD2/nn21If x1rather than xnis the doubtful value, the criterion is asfollows:T15 x 2 x1!/s (2)The critical values for either case, for the 1, 5, and 10 %levels of significance, are g
32、iven in Table 1.7.1.1 The test criterion Tncan be equated to the Students ttest statistic for equality of means between a population withone observation xnand another with the remaining observa-tions x1, . , xn 1, and the critical value of Tnfor significancelevel can be approximated using the /n per
33、centage point ofStudents t withn2degrees of freedom. The approximationis exact for small enough values of , depending on n, andotherwise a slight overestimate unless both and n are large:Tn! #tn,n2211ntn,n2222 1n 2 1!27.1.2 To test outliers on the high side, use the statistic Tn=(xnx )/s and take as
34、 critical value the 0.05 point of Table 1.To test outliers on the low side, use the statistic T1=(xx1)/sand again take as a critical value the 0.05 point of Table 1.Ifwe are interested in outliers occurring on either side, use thestatistic Tn=(xnx )/s or the statistic T1=(xx1)/s whicheveris larger.
35、If in this instance we use the 0.05 point of Table 1 asE178 16a2our critical value, the true significance level would be twice0.05 or 0.10. Similar considerations apply to the other testsgiven below.7.1.3 Example 1As an illustration of the use of TnandTable 1, consider the following ten observations
36、 on breakingstrength (in pounds) of 0.104-in. hard-drawn copper wire: 568,570, 570, 570, 572, 572, 572, 578, 584, 596. See Fig. 1. Thedoubtful observation is the high value, x10= 596. Is the valueof 596 significantly high? The mean is x = 575.2 and theestimated standard deviation is s = 8.70. We com
37、pute:T105 596 2 575.2!/8.70 5 2.39 (3)From Table 1, for n = 10, note that a T10as large as 2.39would occur by chance with probability less than 0.05. In fact,so large a value would occur by chance not much more oftenthan1%ofthetime. Thus, the weight of the evidence isagainst the doubtful value havin
38、g come from the same popu-lation as the others (assuming the population is normallydistributed). Investigation of the doubtful value is thereforeindicated.7.2 Dixon Criteria for a Single OutlierAn alternativesystem, the Dixon criteria (2),3based entirely on ratios ofdifferences between the observati
39、ons may be used in caseswhere it is desirable to avoid calculation of s or where quickjudgment is called for. For the Dixon test, the sample criterionor statistic changes with sample size. Table 2 gives theappropriate statistic to calculate and also gives the criticalvalues of the statistic for the
40、1, 5, and 10 % levels ofsignificance. In most situations, the Dixon criteria is lesspowerful at detecting an outlier than the criterion given in 7.1.7.2.1 Example 2As an illustration of the use of Dixonstest, consider again the observations on breaking strength givenin Example 1. Table 2 indicates u
41、se of:r115xn2 xn21!/xn2 x2! (4)Thus, for n = 10:r115x102 x9!/x102 x2! (5)For the measurements of breaking strength above:r115 596 2 584!/596 2 570! 5 0.462 (6)Which is a little less than 0.478, the 5 % critical value for n= 10. Under the Dixon criterion, we should therefore notconsider this observat
42、ion as an outlier at the 5 % level ofsignificance. These results illustrate how borderline cases maybe accepted under one test but rejected under another.7.3 Recursive Testing for Multiple Outliers in UnivariateSamplesFor testing multiple outliers in a sample, recursiveapplication of a test for a si
43、ngle outlier may be used. Inrecursive testing, a test for an outlier, x1or xn, is firstconducted. If this is found to be significant, then the test isrepeated, omitting the outlier found, to test the point on theopposite side of the sample, or an additional point on the sameside. The performance of
44、most tests for single outliers isaffected by masking, where the probability of detecting anoutlier using a test for a single outlier is reduced when there aretwo or more outliers. Therefore, the recommended procedure isto use a criterion designed to test for multiple outliers, usingrecursive testing
45、 to investigate after the initial criterion issignificant.7.4 Criterion for Two Outliers on Opposite Sides of aSampleIn testing the least and the greatest observationssimultaneously as probable outliers in a sample, use the ratio ofsample range to sample standard deviation test of David,Hartley, and
46、 Pearson (5):w/s 5xn2 x1!/s (7)The significance levels for this sample criterion are given inTable 3. Alternatively, the largest residuals test of Tietjen andMoore (7.5) could be used.7.4.1 Example 3This classic set consists of a sample of 15observations of the vertical semidiameters of Venus made b
47、yLieutenant Herndon in 1846 (6). In the reduction of theobservations, Prof. Pierce found the following residuals (in3The boldface numbers in parentheses refer to a list of references at the end ofthis standard.TABLE 1 Critical Values for T (One-Sided Test) When StandardDeviation is Calculated from t
48、he Same SampleANumber ofObservations,nUpper 10 %SignificanceLevelUpper 5 %SignificanceLevelUpper 1 %SignificanceLevel3 1.1484 1.1531 1.15464 1.4250 1.4625 1.49255 1.602 1.672 1.7496 1.729 1.822 1.9447 1.828 1.938 2.0978 1.909 2.032 2.2219 1.977 2.110 2.32310 2.036 2.176 2.41011 2.088 2.234 2.48512 2
49、.134 2.285 2.55013 2.175 2.331 2.60714 2.213 2.371 2.65915 2.247 2.409 2.70516 2.279 2.443 2.74717 2.309 2.475 2.78518 2.335 2.504 2.82119 2.361 2.532 2.85420 2.385 2.557 2.88421 2.408 2.580 2.91222 2.429 2.603 2.93923 2.448 2.624 2.96324 2.467 2.644 2.98725 2.486 2.663 3.00926 2.502 2.681 3.02927 2.519 2.698 3.04928 2.534 2.714 3.06829 2.549 2.730 3.08530 2.563 2.745 3.10335 2.628 2.811 3.17840 2.682 2.866 3.24045 2.727 2.914 3.29250 2.768 2.956 3.336AValues of T are taken from Grubbs (1),3Table 1. All values have bee
