1、Designation: E 178 08An American National StandardStandard Practice forDealing With Outlying Observations1This standard is issued under the fixed designation E 178; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last r
2、evision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This practice covers outlying observations in samplesand how to test the statistical significance of them.An outlyingobserva
3、tion, or “outlier,” is one that appears to deviate mark-edly from other members of the sample in which it occurs. Inthis connection, the following two alternatives are of interest:1.1.1 An outlying observation may be merely an extrememanifestation of the random variability inherent in the data. Ifth
4、is is true, the value should be retained and processed in thesame manner as the other observations in the sample.1.1.2 On the other hand, an outlying observation may be theresult of gross deviation from prescribed experimental proce-dure or an error in calculating or recording the numerical value.In
5、 such cases, it may be desirable to institute an investigationto ascertain the reason for the aberrant value. The observationmay even actually be rejected as a result of the investigation,though not necessarily so. At any rate, in subsequent dataanalysis the outlier or outliers will be recognized as
6、 probablybeing from a different population than that of the other samplevalues.1.2 It is our purpose here to provide statistical rules that willlead the experimenter almost unerringly to look for causes ofoutliers when they really exist, and hence to decide whetheralternative 1.1.1 above, is not the
7、 more plausible hypothesis toaccept, as compared to alternative 1.1.2, in order that the mostappropriate action in further data analysis may be taken. Theprocedures covered herein apply primarily to the simplest kindof experimental data, that is, replicate measurements of someproperty of a given mat
8、erial, or observations in a supposedlysingle random sample. Nevertheless, the tests suggested docover a wide enough range of cases in practice to have broadutility.2. Referenced Documents2.1 ASTM Standards:2E 456 Terminology Relating to Quality and Statistics3. Terminology3.1 Definitions: The termin
9、ology defined in TerminologyE 456 applies to this standard unless modified herein.3.1.1 outliersee outlying observation.3.1.2 outlying observation, nan observation that appearsto deviate markedly in value from other members of the samplein which it appears.4. Significance and Use4.1 When the experim
10、enter is clearly aware that a grossdeviation from prescribed experimental procedure has takenplace, the resultant observation should be discarded, whether ornot it agrees with the rest of the data and without recourse tostatistical tests for outliers. If a reliable correction procedure,for example,
11、for temperature, is available, the observation maysometimes be corrected and retained.4.2 In many cases evidence for deviation from prescribedprocedure will consist primarily of the discordant value itself.In such cases it is advisable to adopt a cautious attitude. Use ofone of the criteria discusse
12、d below will sometimes permit aclear-cut decision to be made. In doubtful cases the experi-menters judgment will have considerable influence. When theexperimenter cannot identify abnormal conditions, he should atleast report the discordant values and indicate to what extentthey have been used in the
13、 analysis of the data.4.3 Thus, for purposes of orientation relative to the over-allproblem of experimentation, our position on the matter ofscreening samples for outlying observations is precisely thefollowing:4.3.1 Physical Reason Known or Discovered for Outlier(s):4.3.1.1 Reject observation(s).4.
14、3.1.2 Correct observation(s) on physical grounds.4.3.1.3 Reject it (them) and possibly take additional obser-vation(s).4.3.2 Physical Reason UnknownUse Statistical Test:4.3.2.1 Reject observation(s).4.3.2.2 Correct observation(s) statistically.4.3.2.3 Reject it (them) and possibly take additional ob
15、ser-vation(s).4.3.2.4 Employ truncated-sample theory for censored obser-vations.1This practice is under the jurisdiction ofASTM Committee E11 on Quality andStatistics and is the direct responsibility of Subcommittee E11.10 on Sampling /Statistics.Current edition approved Oct. 1, 2008. Published Nove
16、mber 2008. Originallyapproved in 1961. Last previous edition approved in 2002 as E 178 02.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book of ASTMStandards volume information, refer to the standards Document Sum
17、mary page onthe ASTM website.1Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.4.4 The statistical test may always be used to support ajudgment that a physical reason does actually exist for anoutlier, or the statistical criterion may
18、be used routinely as abasis to initiate action to find a physical cause.5. Basis of Statistical Criteria for Outliers5.1 There are a number of criteria for testing outliers. In allof these, the doubtful observation is included in the calculationof the numerical value of a sample criterion (or statis
19、tic), whichis then compared with a critical value based on the theory ofrandom sampling to determine whether the doubtful observa-tion is to be retained or rejected. The critical value is that valueof the sample criterion which would be exceeded by chancewith some specified (small) probability on th
20、e assumption thatall the observations did indeed constitute a random samplefrom a common system of causes, a single parent population,distribution or universe. The specified small probability iscalled the “significance level” or “percentage point” and can bethought of as the risk of erroneously reje
21、cting a good obser-vation. It becomes clear, therefore, that if there exists a realshift or change in the value of an observation that arises fromnonrandom causes (human error, loss of calibration of instru-ment, change of measuring instrument, or even change of timeof measurements, etc.), then the
22、observed value of the samplecriterion used would exceed the “critical value” based onrandom-sampling theory. Tables of critical values are usuallygiven for several different significance levels, for example,5 %, 1 %. For statistical tests of outlying observations, it isgenerally recommended that a l
23、ow significance level, such as1 %, be used and that significance levels greater than 5 %should not be common practice.NOTE 1In this practice, we will usually illustrate the use of the 5 %significance level. Proper choice of level in probability depends on theparticular problem and just what may be i
24、nvolved, along with the risk thatone is willing to take in rejecting a good observation, that is, if thenull-hypothesis stating “all observations in the sample come from thesame normal population” may be assumed correct.5.2 It should be pointed out that almost all criteria foroutliers are based on a
25、n assumed underlying normal (Gaussian)population or distribution. When the data are not normally orapproximately normally distributed, the probabilities associ-ated with these tests will be different. Until such time as criterianot sensitive to the normality assumption are developed, theexperimenter
26、 is cautioned against interpreting the probabilitiestoo literally.5.3 Although our primary interest here is that of detectingoutlying observations, we remark that some of the statisticalcriteria presented may also be used to test the hypothesis ofnormality or that the random sample taken did come fr
27、om anormal or Gaussian population. The end result is for allpractical purposes the same, that is, we really wish to knowwhether we ought to proceed as if we have in hand a sample ofhomogeneous normal observations.6. Recommended Criteria for Single Samples6.1 Let the sample of n observations be denot
28、ed in order ofincreasing magnitude by x1# x2# x3# . # xn. Let xnbe thedoubtful value, that is the largest value. The test criterion, Tn,recommended here for a single outlier is as follows:Tn5 xn2 x!/s (1)where:x = arithmetic average of all n values, ands = estimate of the population standard deviati
29、on based onthe sample data, calculated as follows:s =(i 5 1nxi2 x!2n 2 15(i 5 1nxi22 n x2n 2 15(i 5 1nxi22 (i 5 1nxi!2/ nn 2 1If x1rather than xnis the doubtful value, the criterion is asfollows:T15 x 2 x1!/s (2)The critical values for either case, for the 1 and 5 % levels ofsignificance, are given
30、in Table 1. Table 1 and the followingtables give the “one-sided” significance levels. In the previoustentative recommended practice (1961), the tables listed valuesof significance levels double those in the present practice, sinceit was considered that the experimenter would test either thelowest or
31、 the highest observation (or both) for statisticalsignificance. However, to be consistent with actual practice andin an attempt to avoid further misunderstanding, single-sidedsignificance levels are tabulated here so that both viewpointscan be represented.6.2 The hypothesis that we are testing in ev
32、ery case is thatall observations in the sample come from the same normalpopulation. Let us adopt, for example, a significance level of0.05. If we are interested only in outliers that occur on the highside, we should always use the statistic Tn= (xn x)/s and takeas critical value the 0.05 point of Ta
33、ble 1. On the other hand,if we are interested only in outliers occurring on the low side,we would always use the statistic T1= (xx1)/s and againtake as a critical value the 0.05 point of Table 1. Suppose,however, that we are interested in outliers occurring on eitherside, but do not believe that out
34、liers can occur on both sidessimultaneously. We might, for example, believe that at sometime during the experiment something possibly happened tocause an extraneous variation on the high side or on the lowside, but that it was very unlikely that two or more such eventscould have occurred, one being
35、an extraneous variation on thehigh side and the other an extraneous variation on the low side.With this point of view we should use the statistic Tn=(xnx)/s or the statistic T1=(x x1)/s whichever is larger. If in thisinstance we use the 0.05 point of Table 1 as our critical value,the true significan
36、ce level would be twice 0.05 or 0.10. If wewish a significance level of 0.05 and not 0.10, we must in thiscase use as a critical value the 0.025 point of Table 1. Similarconsiderations apply to the other tests given below.6.2.1 Example 1As an illustration of the use of TnandTable 1, consider the fol
37、lowing ten observations on breakingstrength (in pounds) of 0.104-in. hard-drawn copper wire: 568,570, 570, 570, 572, 572, 572, 578, 584, 596. See Fig. 1. Thedoubtful observation is the high value, x10= 596. Is the valueE178082of 596 significantly high? The mean is x = 575.2 and theestimated standard
38、 deviation is s = 8.70. We computeT105 596 2 575.2!/8.70 5 2.39 (3)From Table 1, for n = 10, note that a T10as large as 2.39would occur by chance with probability less than 0.05. In fact,so large a value would occur by chance not much more oftenthan 1 % of the time. Thus, the weight of the evidence
39、isagainst the doubtful value having come from the same popu-lation as the others (assuming the population is normallydistributed). Investigation of the doubtful value is thereforeindicated.TABLE 1 Critical Values for T (One-Sided Test) When Standard Deviation is Calculated from the Same SampleANumbe
40、r ofObservations,nUpper 0.1 %SignificanceLevelUpper 0.5 %SignificanceLevelUpper 1 %SignificanceLevelUpper 2.5 %SignificanceLevelUpper 5 %SignificanceLevelUpper 10 %SignificanceLevel3 1.155 1.155 1.155 1.155 1.153 1.1484 1.499 1.496 1.492 1.481 1.463 1.4255 1.780 1.764 1.749 1.715 1.672 1.6026 2.011
41、1.973 1.944 1.887 1.822 1.7297 2.201 2.139 2.097 2.020 1.938 1.8288 2.358 2.274 2.221 2.126 2.032 1.9099 2.492 2.387 2.323 2.215 2.110 1.97710 2.606 2.482 2.410 2.290 2.176 2.03611 2.705 2.564 2.485 2.355 2.234 2.08812 2.791 2.636 2.550 2.412 2.285 2.13413 2.867 2.699 2.607 2.462 2.331 2.17514 2.935
42、 2.755 2.659 2.507 2.371 2.21315 2.997 2.806 2.705 2.549 2.409 2.24716 3.052 2.852 2.747 2.585 2.443 2.27917 3.103 2.894 2.785 2.620 2.475 2.30918 3.149 2.932 2.821 2.651 2.504 2.33519 3.191 2.968 2.854 2.681 2.532 2.36120 3.230 3.001 2.884 2.709 2.557 2.38521 3.266 3.031 2.912 2.733 2.580 2.40822 3
43、.300 3.060 2.939 2.758 2.603 2.42923 3.332 3.087 2.963 2.781 2.624 2.44824 3.362 3.112 2.987 2.802 2.644 2.46725 3.389 3.135 3.009 2.822 2.663 2.48626 3.415 3.157 3.029 2.841 2.681 2.50227 3.440 3.178 3.049 2.859 2.698 2.51928 3.464 3.199 3.068 2.876 2.714 2.53429 3.486 3.218 3.085 2.893 2.730 2.549
44、30 3.507 3.236 3.103 2.908 2.745 2.56331 3.528 3.253 3.119 2.924 2.759 2.57732 3.546 3.270 3.135 2.938 2.773 2.59133 3.565 3.286 3.150 2.952 2.786 2.60434 3.582 3.301 3.164 2.965 2.799 2.61635 3.599 3.316 3.178 2.979 2.811 2.62836 3.616 3.330 3.191 2.991 2.823 2.63937 3.631 3.343 3.204 3.003 2.835 2
45、.65038 3.646 3.356 3.216 3.014 2.846 2.66139 3.660 3.369 3.228 3.025 2.857 2.67140 3.673 3.381 3.240 3.036 2.866 2.68241 3.687 3.393 3.251 3.046 2.877 2.69242 3.700 3.404 3.261 3.057 2.887 2.70043 3.712 3.415 3.271 3.067 2.896 2.71044 3.724 3.425 3.282 3.075 2.905 2.71945 3.736 3.435 3.292 3.085 2.9
46、14 2.72746 3.747 3.445 3.302 3.094 2.923 2.73647 3.757 3.455 3.310 3.103 2.931 2.74448 3.768 3.464 3.319 3.111 2.940 2.75349 3.779 3.474 3.329 3.120 2.948 2.760FIG. 1 Ten Observations of Breaking Strength from Example 1 in6.2.1E178083TABLE 1 ContinuedNumber ofObservations,nUpper 0.1 %SignificanceLev
47、elUpper 0.5 %SignificanceLevelUpper 1 %SignificanceLevelUpper 2.5 %SignificanceLevelUpper 5 %SignificanceLevelUpper 10 %SignificanceLevel50 3.789 3.483 3.336 3.128 2.956 2.76851 3.798 3.491 3.345 3.136 2.964 2.77552 3.808 3.500 3.353 3.143 2.971 2.78353 3.816 3.507 3.361 3.151 2.978 2.79054 3.825 3.
48、516 3.368 3.158 2.986 2.79855 3.834 3.524 3.376 3.166 2.992 2.80456 3.842 3.531 3.383 3.172 3.000 2.81157 3.851 3.539 3.391 3.180 3.006 2.81858 3.858 3.546 3.397 3.186 3.013 2.82459 3.867 3.553 3.405 3.193 3.019 2.83160 3.874 3.560 3.411 3.199 3.025 2.83761 3.882 3.566 3.418 3.205 3.032 2.84262 3.88
49、9 3.573 3.424 3.212 3.037 2.84963 3.896 3.579 3.430 3.218 3.044 2.85464653.9033.9103.5863.5923.4373.4423.2243.2303.0493.0552.8602.86666 3.917 3.598 3.449 3.235 3.061 2.87167 3.923 3.605 3.454 3.241 3.066 2.87768 3.930 3.610 3.460 3.246 3.071 2.88369 3.936 3.617 3.466 3.252 3.076 2.88870 3.942 3.622 3.471 3.257 3.082 2.89371 3.948 3.627 3.476 3.262 3.087 2.89772 3.954 3.633 3.482 3.267 3.092 2.90373 3.960 3.638 3.487 3.272 3.098 2.90874 3.965 3.643 3.492 3.278 3.102 2.91275 3.971 3.648 3.496 3.282 3.107 2.91776 3.977 3.654 3.502 3.287 3.111 2.92277 3.982
