1、Designation: D 6589 05Standard Guide forStatistical Evaluation of Atmospheric Dispersion ModelPerformance1This standard is issued under the fixed designation D 6589; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last
2、revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon (e) indicates an editorial change since the last revision or reapproval.1. Scope1.1 This guide provides techniques that are useful for thecomparison of modeled air concentrations with observed fielddata. Su
3、ch comparisons provide a means for assessing amodels performance, for example, bias and precision oruncertainty, relative to other candidate models. Methodologiesfor such comparisons are yet evolving; hence, modificationswill occur in the statistical tests and procedures and dataanalysis as work pro
4、gresses in this area. Until the interestedparties agree upon standard testing protocols, differences inapproach will occur. This guide describes a framework, orphilosophical context, within which one determines whether amodels performance is significantly different from othercandidate models. It is
5、suggested that the first step should be todetermine which models estimates are closest on average tothe observations, and the second step would then test whetherthe differences seen in the performance of the other models aresignificantly different from the model chosen in the first step.An example p
6、rocedure is provided inAppendix X1 to illustratean existing approach for a particular evaluation goal. Thisexample is not intended to inhibit alternative approaches ortechniques that will produce equivalent or superior results. Asdiscussed in Section 6, statistical evaluation of model perfor-mance i
7、s viewed as part of a larger process that collectively isreferred to as model evaluation.1.2 This guide has been designed with flexibility to allowexpansion to address various characterizations of atmosphericdispersion, which might involve dose or concentration fluctua-tions, to allow development of
8、 application-specific evaluationschemes, and to allow use of various statistical comparisonmetrics. No assumptions are made regarding the manner inwhich the models characterize the dispersion.1.3 The focus of this guide is on end results, that is, theaccuracy of model predictions and the discernment
9、 of whetherdifferences seen between models are significant, rather thanoperational details such as the ease of model implementation orthe time required for model calculations to be performed.1.4 This guide offers an organized collection of informationor a series of options and does not recommend a s
10、pecific courseof action. This guide cannot replace education or experienceand should be used in conjunction with professional judgment.Not all aspects of this guide may be applicable in all circum-stances. This guide is not intended to represent or replace thestandard of care by which the adequacy o
11、f a given professionalservice must be judged, nor should it be applied withoutconsideration of a projects many unique aspects. The word“Standard” in the title of this guide means only that thedocument has been approved through the ASTM consensusprocess.1.5 This standard does not purport to address a
12、ll of thesafety concerns, if any, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and to determine theapplicability of regulatory limitations prior to use.2. Referenced Documents2.1 ASTM Standards:2D 1356 Terminology
13、 Relating to Sampling and Analysis ofAtmospheres3. Terminology3.1 DefinitionsFor definitions of terms used in this guide,refer to Terminology D 1356.3.2 Definitions of Terms Specific to This Standard:3.2.1 atmospheric dispersion model, nan idealization ofatmospheric physics and processes to calculat
14、e the magnitudeand location of pollutant concentrations based on fate, trans-port, and dispersion in the atmosphere. This may take the formof an equation, algorithm, or series of equations/algorithmsused to calculate average or time-varying concentration. Themodel may involve numerical methods for s
15、olution.3.2.2 dispersion, absolute, nthe characterization of thespreading of material released into the atmosphere based on acoordinate system fixed in space.1This guide is under the jurisdiction of ASTM Committee D22 on Air Qualityand is the direct responsibility of Subcommittee D22.11 on Meteorolo
16、gy.Current edition approved October 1, 2005. Published October 2005. Originallyapproved in 2000. Last previous edition approved in 2000 as D 7144 - 00.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book of ASTMStan
17、dards volume information, refer to the standards Document Summary page onthe ASTM website.1Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.3.2.3 dispersion, relative, nthe characterization of thespreading of material released into the
18、 atmosphere based on acoordinate system that is relative to the local median positionof the dispersing material.3.2.4 evaluation objective, na feature or characteristic,which can be defined through an analysis of the observedconcentration pattern, for example, maximum centerline con-centration or la
19、teral extent of the average concentration patternas a function of downwind distance, which one desires toassess the skill of the models to reproduce.3.2.5 evaluation procedure, nthe analysis steps to betaken to compute the value of the evaluation objective from theobserved and modeled patterns of co
20、ncentration values.3.2.6 fate, nthe destiny of a chemical or biological pol-lutant after release into the environment.3.2.7 model input value, ncharacterizations that must beestimated or provided by the model developer or user beforemodel calculations can be performed.3.2.8 regime, na repeatable nar
21、row range of conditions,defined in terms of model input values, which may or may notbe explicitly employed by all models being tested, needed fordispersion model calculations. It is envisioned that the disper-sion observed should be similar for all cases having similarmodel input values.3.2.9 uncert
22、ainty, nrefers to a lack of knowledge aboutspecific factors or parameters. This includes measurementerrors, sampling errors, systematic errors, and differencesarising from simplification of real-world processes. In prin-ciple, uncertainty can be reduced with further information orknowledge (1)3.3.2.
23、10 variability, nrefers to differences attributable totrue heterogeneity or diversity in atmospheric processes thatresult in part from natural random processes. Variabilityusually is not reducible by further increases in knowledge, butit can in principle be better characterized (1).4. Summary of Gui
24、de4.1 Statistical evaluation of dispersion model performancewith field data is viewed as part of a larger process thatcollectively is called model evaluation. Section 6 discusses thecomponents of model evaluation.4.2 To statistically assess model performance, one mustdefine an overall evaluation goa
25、l or purpose. This will suggestfeatures (evaluation objectives) within the observed and mod-eled concentration patterns to be compared, for example,maximum surface concentrations, lateral extent of a dispersingplume. The selection and definition of evaluation objectivestypically are tailored to the
26、models capabilities and intendeduses. The very nature of the problem of characterizing airquality and the way models are applied make one single orabsolute evaluation objective impossible to define that issuitable for all purposes. The definition of the evaluationobjectives will be restricted by the
27、 limited range conditionsexperienced in the available comparison data suitable for use.For each evaluation objective, a procedure will need to bedefined that allows definition of the evaluation objective fromthe available observations of concentration values.4.3 In assessing the performance of air q
28、uality models tocharacterize a particular evaluation objective, one shouldconsider what the models are capable of providing. As dis-cussed in Section 7, most models attempt to characterize theensemble average concentration pattern. If such models shouldprovide favorable comparisons with observed con
29、centrationmaxima, this is resulting from happenstance, rather than skill inthe model; therefore, in this discussion, it is suggested a modelbe assessed on its ability to reproduce what it was designed toproduce, for at least in these comparisons, one can be assuredthat zero bias with the least amoun
30、t of scatter is by definitiongood model performance.4.4 As an illustration of the principles espoused in thisguide, a procedure is provided in Appendix X1 for comparisonof observed and modeled near-centerline concentration values,which accommodates the fact that observed concentrationvalues include
31、a large component of stochastic, and possiblydeterministic, variability unaccounted for by current models.The procedure provides an objective statistical test of whetherdifferences seen in model performance are significant.5. Significance and Use5.1 Guidance is provided on designing model evaluation
32、performance procedures and on the difficulties that arise instatistical evaluation of model performance caused by thestochastic nature of dispersion in the atmosphere. It is recog-nized there are examples in the literature where, knowingly orunknowingly, models were evaluated on their ability to de-
33、scribe something which they were never intended to charac-terize. This guide is attempting to heighten awareness, andthereby, to reduce the number of “unknowing” comparisons. Agoal of this guide is to stimulate development and testing ofevaluation procedures that accommodate the effects of naturalva
34、riability. A technique is illustrated to provide informationfrom which subsequent evaluation and standardization can bederived.6. Model Evaluation6.1 BackgroundAir quality simulation models have beenused for many decades to characterize the transport anddispersion of material in the atmosphere (2-4)
35、. Early evalua-tions of model performance usually relied on linear least-squares analyses of observed versus modeled values, usingtraditional scatter plots of the values, (5-7). During the 1980s,attempts have been made to encourage the standardization ofmethods used to judge air quality model perfor
36、mance (8-11).Further development of these proposed statistical evaluationprocedures was needed, as it was found that the rote applica-tion of statistical metrics, such as those listed in (8), wasincapable of discerning differences in model performance (12),whereas if the evaluation results were sort
37、ed by stability anddistance downwind, then differences in modeling skill could bediscerned (13). It was becoming increasingly evident that themodels were characterizing only a small portion of the ob-served variations in the concentration values (14). To betterdeduce the statistical significance of
38、differences seen in modelperformance in the face of large unaccounted for uncertainties3The boldface numbers in parentheses refer to the list of references at the end ofthis standard.D6589052and variations, investigators began to explore the use ofbootstrap techniques (15). By the late 1980s, most o
39、f the modelperformance evaluations involved the use of bootstrap tech-niques in the comparison of maximum values of modeled andobserved cumulative frequency distributions of the concentra-tions values (16). Even though the procedures and metrics to beemployed in describing the performance of air qua
40、lity simula-tion models are still evolving (17-19), there has been a generalacceptance that defining performance of air quality modelsneeds to address the large uncertainties inherent in attemptingto characterize atmospheric fate, transport and dispersionprocesses. There also has been a consensus re
41、ached on thephilosophical reasons that models of earth science processescan never be validated, in the sense of claiming that a model istruthfully representing natural processes. No general empiricalproposition about the natural world can be certain, since therewill always remain the prospect that f
42、uture observations maycall the theory in question (20). It is seen that numerical modelsof air pollution are a form of a highly complex scientifichypothesis concerning natural processes, that can be confirmedthrough comparison with observations, but never validated.6.2 Components of Model Evaluation
43、A model evaluationincludes science peer reviews and statistical evaluations withfield data. The completion of each of these componentsassumes specific model goals and evaluation objectives (seeSection 10) have been defined.6.3 Science Peer ReviewsGiven the complexity of char-acterizing atmospheric p
44、rocesses, and the inevitable necessityof limiting model algorithms to a resolvable set, one compo-nent of a model evaluation is to review the models science toconfirm that the construct is reasonable and defensible for thedefined evaluation objectives. A key part of the scientific peerreview will in
45、clude the review of residual plots where modeledand observed evaluation objectives are compared over a rangeof model inputs, for example, maximum concentrations as afunction of estimated plume rise or as a function of distancedownwind.6.4 Statistical Evaluations with Field DataThe objectivecompariso
46、n of modeled concentrations with observed field dataprovides a means for assessing model performance. Due to thelimited supply of evaluation data sets, there are severe practicallimits in assessing model performance. For this reason, theconclusions reached in the science peer reviews (see 6.3) andth
47、e supportive analyses (see 6.5) have particular relevance indeciding whether a model can be applied for the defined modelevaluation objectives. In order to conduct a statistical compari-son, one will have to define one or more evaluation objectivesfor which objective comparisons are desired (Section
48、 10). Asdiscussed in 8.4.4, the process of summarizing the overallperformance of a model over the range of conditions experi-enced within a field experiment typically involves determiningtwo points for each of the model evaluation objectives: whichof the models being assessed has on average the smal
49、lestcombined bias and scatter in comparisons with observations,and whether the differences seen in the comparisons with theother models statistically are significant in light of the uncer-tainties in the observations.6.5 Other Tasks Supportive to Model EvaluationAs atmo-spheric dispersion models become more sophisticated, it is noteasy to detect coding errors in the implementation of the modelalgorithms. And as models become more complex, discerningthe sensitivity of the modeling results to input parametervariations becomes less clear; hence, two important tasks th
