1、Designation: E178 08E178 16 An American National StandardStandard Practice forDealing With Outlying Observations1This standard is issued under the fixed designation E178; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of
2、last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This practice covers outlying observations in samples and how to test the statistical significance of them. An outlyin
3、gobservation, or “outlier,” is one that appears to deviate markedly from other members of the sample in which it occurs. In thisconnection, the following two alternatives are of interest:outliers.1.1.1 An outlying observation may be merely an extreme manifestation of the random variability inherent
4、in the data. If thisis true, the value should be retained and processed in the same manner as the other observations in the sample.1.1.2 On the other hand, an outlying observation may be the result of gross deviation from prescribed experimental procedureor an error in calculating or recording the n
5、umerical value. In such cases, it may be desirable to institute an investigation toascertain the reason for the aberrant value. The observation may even actually be rejected as a result of the investigation, thoughnot necessarily so. At any rate, in subsequent data analysis the outlier or outliers w
6、ill be recognized as probably being from adifferent population than that of the other sample values.1.2 It is our purpose here to provide statistical rules that will lead the experimenter almost unerringly to look for causes ofoutliers when they really exist, and hence to decide whether alternative
7、The system of units for this standard 1.1.1 above, is notthe more plausible hypothesis to accept, as compared to alternative is not specified. Dimensional quantities 1.1.2, in order that themost appropriate action in further data analysis may be taken. The procedures covered herein apply primarily t
8、o the simplest kindof experimental data, that is, replicate measurements of some property of a given material, or observations in a supposedly singlerandom sample. Nevertheless, the tests suggested do cover a wide enough range of cases in practice to have broad utility.thestandard are presented only
9、 as illustrations of calculation methods. The examples are not binding on products or test methodstreated.1.3 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibilityof the user of this standard to establish appropriate safety an
10、d health practices and determine the applicability of regulatoryrequirements prior to use.2. Referenced Documents2.1 ASTM Standards:2E456 Terminology Relating to Quality and StatisticsE2586 Practice for Calculating and Using Basic Statistics3. Terminology3.1 Definitions: The terminology defined in T
11、erminology E456 applies to this standard unless modified herein.3.1.1 order statistic x(k), nvalue of the kth observed value in a sample after sorting by order of magnitude. (Practice E2586.)3.1.1.1 DiscussionIn this Practice, xk is used to denote order statistics in place of x(k), to simplify the n
12、otation.3.1.2 outliersee outlying observation.1 This practice is under the jurisdiction of ASTM Committee D19E11 on WaterQuality and Statistics and is the direct responsibility of Subcommittee D19.05E11.10 onInorganic Constituents in WaterSampling / Statistics.Current edition approved June 1, 2016.
13、Published November 2008June 2016. Originally approved in 1961. Last previous edition approved in 20022008 asE178 02.E178 08. DOI: 10.1520/E0178-08.10.1520/E0178-16.2 For referencedASTM standards, visit theASTM website, www.astm.org, or contactASTM Customer Service at serviceastm.org. For Annual Book
14、 of ASTM Standardsvolume information, refer to the standards Document Summary page on the ASTM website.This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Becauseit may not be technic
15、ally possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current versionof the standard as published by ASTM is to be considered the official document.Copyright ASTM International, 100 Barr Harbor Drive, PO Box
16、 C700, West Conshohocken, PA 19428-2959. United States13.1.3 outlying observation, nan observation extreme observation in either direction that appears to deviate markedly in valuefrom other members of the sample in which it appears.4. Significance and Use4.1 When the experimenter is clearly aware t
17、hat a gross deviation from prescribed experimental procedure has taken place, theresultant observation should be discarded, whether or not it agrees with the rest of the data and without recourse to statistical testsfor outliers. If a reliable correction procedure, for example, for temperature, is a
18、vailable, the observation may sometimes becorrected and retained.4.1 In many cases evidence for deviation from prescribed procedure will consist primarily of the discordant value itself. In suchcases it is advisable to adopt a cautious attitude. Use of one of the criteria discussed below will someti
19、mes permit a clear-cutdecision to be made. In doubtful cases the experimenters judgment will have considerable influence. When the experimentercannot identify abnormal conditions, he should at least report the discordant values and indicate to what extent they have been usedin the analysis of the da
20、ta.An outlying observation, or “outlier,” is an extreme one in either direction that appears to deviatemarkedly from other members of the sample in which it occurs.4.2 Thus, for purposes of orientation relative to the over-all problem of experimentation, our position on the matter of screeningsample
21、s for outlying observations is precisely the following: Statistical rules test the null hypothesis of no outliers against thealternative of one or more actual outliers. The procedures covered were developed primarily to apply to the simplest kind ofexperimental data, that is, replicate measurements
22、of some property of a given material or observations in a supposedly randomsample.4.3.1 Physical Reason Known or Discovered for Outlier(s):4.3.1.1 Reject observation(s).4.3.1.2 Correct observation(s) on physical grounds.4.3.1.3 Reject it (them) and possibly take additional observation(s).4.3.2 Physi
23、cal Reason UnknownUse Statistical Test:4.3.2.1 Reject observation(s).4.3.2.2 Correct observation(s) statistically.4.3.2.3 Reject it (them) and possibly take additional observation(s).4.3.2.4 Employ truncated-sample theory for censored observations.4.3 TheA statistical test may always be used to supp
24、ort a judgment that a physical reason does actually exist for an outlier, orthe statistical criterion may be used routinely as a basis to initiate action to find a physical cause.5. Procedure5.1 In dealing with an outlier, the following alternatives should be considered:5.1.1 An outlying observation
25、 might be the result of gross deviation from prescribed experimental procedure or an error incalculating or recording the numerical value. When the experimenter is clearly aware that a deviation from prescribed experimentalprocedure has taken place, the resultant observation should be discarded, whe
26、ther or not it agrees with the rest of the data andwithout recourse to statistical tests for outliers. If a reliable correction procedure is available, the observation may sometimes becorrected and retained.5.1.2 An outlying observation might be merely an extreme manifestation of the random variabil
27、ity inherent in the data. If thisis true, the value should be retained and processed in the same manner as the other observations in the sample. Transformationof data or using methods of data analysis designed for a non-normal distribution might be appropriate.5.1.3 Test units that give outlying obs
28、ervations might be of special interest. If this is true, once identified they should besegregated for more detailed study.5.2 In many cases, evidence for deviation from prescribed procedure will consist primarily of the discordant value itself. In suchcases it is advisable to adopt a cautious attitu
29、de. Use of one of the criteria discussed below will sometimes permit a clearcutdecision to be made.5.2.1 When the experimenter cannot identify abnormal conditions, he should report the discordant values and indicate to whatextent they have been used in the analysis of the data.5.3 Thus, as part of t
30、he over-all process of experimentation, the process of screening samples for outlying observations andacting on them is the following:5.3.1 Physical Reason Known or Discovered for Outlier(s):5.3.1.1 Reject observation(s) and possibly take additional observation(s).5.3.1.2 Correct observation(s) on p
31、hysical grounds.5.3.2 Physical Reason UnknownUse Statistical Test:5.3.2.1 Reject observation(s) and possibly take additional observation(s).5.3.2.2 Transform observation(s) to improve fit to a normal distribution.5.3.2.3 Use estimation appropriate for non-normal distributions.E178 1625.3.2.4 Segrega
32、te samples for further study.6. Basis of Statistical Criteria for Outliers6.1 There are a number of criteria for testing outliers. In all of these, In testing outliers, the doubtful observation is includedin the calculation of the numerical value of a sample criterion (or statistic), which is then c
33、ompared with a critical value based onthe theory of random sampling to determine whether the doubtful observation is to be retained or rejected. The critical value isthat value of the sample criterion which would be exceeded by chance with some specified (small) probability on the assumptionthat all
34、 the observations did indeed constitute a random sample from a common system of causes, a single parent population,distribution or universe. The specified small probability is called the “significance level” or “percentage point” and can be thoughtof as the risk of erroneously rejecting a good obser
35、vation. It becomes clear, therefore, that if there exists If a real shift or changein the value of an observation that arises from nonrandom causes (human error, loss of calibration of instrument, change ofmeasuring instrument, or even change of time of measurements, etc.), and so forth), then the o
36、bserved value of the sample criterionused wouldwill exceed the “critical value” based on random-sampling theory. Tables of critical values are usually given for severaldifferent significance levels, for example, 5 %, 1 %. For statistical tests of outlying observations, it is generally recommended th
37、ata low significance level, such as 1 %, be used and that significance levels greater than 5 % should not be common practice.levels.In particular for this Practice, significance levels 10, 5, and 1% are used.NOTE 1In this practice, we will usually illustrate the use of the 5 %5% significance level.
38、Proper choice of level in probability depends on theparticular problem and just what may be involved, along with the risk that one is willing to take in rejecting a good observation, that is, if thenull-hypothesis stating “all observations in the sample come from the same normal population” may be a
39、ssumed correct.6.2 It should be pointed out that almost Almost all criteria for outliers are based on an assumed underlying normal (Gaussian)population or distribution. The null hypothesis that we are testing in every case is that all observations in the sample come fromthe same normal population. I
40、n choosing an appropriate alternative hypothesis (one or more outliers, separated or bunched, onsame side or different sides, and so forth) it is useful to plot the data as shown in the dot diagrams of the figures. When the dataare not normally or approximately normally distributed, the probabilitie
41、s associated with these tests will be different. Until suchtime as criteria not sensitive to the normality assumption are developed, the The experimenter is cautioned against interpreting theprobabilities too literally.6.3 Although our primary interest here is that of detecting outlying observations
42、, we remark that some of the statistical criteriapresented may also be used to test the hypothesis of normality or that the random sample taken did come from a normal or Gaussianpopulation. The end result is for all practical purposes the same, that is, we really wish to know whether we ought to pro
43、ceed asif we have in hand a sample of homogeneous normal observations.6.4 One should distinguish between data to be used to estimate a central value from data to be used to assess variability. Whenthe purpose is to estimate a standard deviation, it might be seriously underestimated by dropping too m
44、any “outlying“ observations.7. Recommended Criteria for Single Samples6.1 Let the sample of n observations be denoted in order of increasing magnitude by x1 x2 x3 . x n. Let xn be the doubtfulvalue, that is the largest value. The test criterion, Tn, recommended here for a single outlier is as follow
45、s:Tn 5xn 2x!/s (1)where:x = arithmetic average of all n values, ands = estimate of the population standard deviation based on the sample data, calculated as follows:s = !(i51nxi2x! 2n21 5!(i51n xi 22nx 2n21 5!(i51n xi 22S(i51n xiD 2/nn21If x1 rather than xn is the doubtful value, the criterion is as
46、 follows:T15x 2x1!/s (2)The critical values for either case, for the 1 and 5 % levels of significance, are given in Table 1. Table 1 and the following tablesgive the “one-sided” significance levels. In the previous tentative recommended practice (1961), the tables listed values ofsignificance levels
47、 double those in the present practice, since it was considered that the experimenter would test either the lowestor the highest observation (or both) for statistical significance. However, to be consistent with actual practice and in an attemptto avoid further misunderstanding, single-sided signific
48、ance levels are tabulated here so that both viewpoints can be represented.7.1 Criterion for a Single OutlierThe hypothesis that we are testing in every case is that all observations in the sample comefrom the same normal population. Let us adopt, for example, a significance level of 0.05. If we are
49、interested Let the sample ofonlyn in outliers that occur on the observations be denoted in order high side, we should always use the statistic Tn = (xn x)/sE178 163TABLE 1 Critical Values for T (One-Sided Test) When StandardDeviation is Calculated from the Same SampleANumber ofObservations,nUpper 0.1 %SignificanceLevelUpper 0.5 %SignificanceLevelUpper 1 %SignificanceLevelUpper 2.5 %SignificanceLevelUpper 5 %SignificanceLevelUpper 10 %SignificanceLevel3 1.155 1.155 1.155 1.155 1.153 1.1484 1.499 1.496 1.492 1.481 1.463 1.4255 1.780 1.764 1.749 1.715
