ASTM E2617-2009a Standard Practice for Validation of Empirically Derived Multivariate Calibrations.pdf

1、Designation: E 2617 09aStandard Practice forValidation of Empirically Derived Multivariate Calibrations1This standard is issued under the fixed designation E 2617; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last re

2、vision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This practice covers requirements for the validation ofempirically derived calibrations (Note 1) such as calibrationsderived

3、by Multiple Linear Regression (MLR), Principal Com-ponent Regression (PCR), Partial Least Squares (PLS), Artifi-cial Neural Networks (ANN), or any other empirical calibra-tion technique whereby a relationship is postulated between aset of variables measured for a given sample under test and oneor mo

4、re physical, chemical, quality, or membership propertiesapplicable to that sample.NOTE 1Empirically derived calibrations are sometimes referred to as“models” or “calibrations.” In the following text, for conciseness, the term“calibration” may be used instead of the full name of the procedure.1.2 Thi

5、s practice does not cover procedures for establishingsaid postulated relationship.1.3 This practice serves as an overview of techniques usedto verify the applicability of an empirically derived multivari-ate calibration to the measurement of a sample under test andto verify equivalence between the p

6、roperties calculated fromthe empirically derived multivariate calibration and the resultsof an accepted reference method of measurement to withincontrol limits established for the prespecified statistical confi-dence level.1.4 This standard does not purport to address all of thesafety concerns, if a

7、ny, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and determine the applica-bility of regulatory limitations prior to use.2. Referenced Documents2.1 ASTM Standards:2E 131 Terminology Relating to Molecular Spectrosc

8、opyE 1655 Practices for Infrared Multivariate QuantitativeAnalysisE 1790 Practice for Near Infrared Qualitative Analysis3. Terminology3.1 For terminology related to molecular spectroscopicmethods, refer to Terminology E 131. For terminology relatedto multivariate quantitative modeling refer to Pract

9、ices E 1655.While Practices E 1655 is written in the context of multivariatespectroscopic methods, the terminology is also applicable toother multivariate technologies.3.2 Definitions of Terms Specific to This Standard:3.2.1 accuracythe closeness of agreement between a testresult and an accepted ref

10、erence value.3.2.2 biasthe arithmetic average difference between thereference values and the values produced by the analyticalmethod under test, for a set of samples.3.2.3 detection limitthe lowest level of a property in asample that can be detected, but not necessarily quantified, bythe measurement

11、 system.3.2.4 estimatethe constituent concentration, identifica-tion, or other property of a sample as determined by theanalytical method being validated.3.2.5 initial validationvalidation that is performed whenan analyzer system is initially installed or after major mainte-nance.3.2.6 Negative Frac

12、tion Identifiedthe fraction of samplesnot having a particular characteristic that is identified as nothaving that characteristic.3.2.6.1 DiscussionNegative Fraction Identified assumesthat the characteristic that the test measures either is or is notpresent. It is not applicable to tests with multipl

13、e possibleoutcomes.3.2.7 ongoing periodic revalidationthe quality assuranceprocess by which, in the case of quantitative calibrations, thebias and precision or, in the case of qualitative calibrations, thePositive Fraction Identified and Negative Fraction Identifiedperformance determined during init

14、ial validation are shown tobe sustained.3.2.8 Positive Fraction Identifiedthe fraction of sampleshaving a particular characteristic that is identified as havingthat characteristic.3.2.8.1 DiscussionPositive Fraction Identified assumesthat the characteristic that the test measures either is or is not

15、present. It is not applicable to tests with multiple possibleoutcomes.1This practice is under the jurisdiction of ASTM Committee E13 on MolecularSpectroscopy and Separation Science and is the direct responsibility of Subcom-mittee E13.11 on Multivariate Analysis.Current edition approved April 1, 200

16、9. Published April 2009. Originallyapproved in 2008. Last previous edition approved in 2009 as E 2617 09.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book of ASTMStandards volume information, refer to the standar

17、ds Document Summary page onthe ASTM website.1Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.3.2.9 precisionthe closeness of agreement between inde-pendent test results obtained under stipulated conditions.3.2.9.1 DiscussionPrecision

18、may be a measure of eitherthe degree of reproducibility or degree of repeatability of theanalytical method under normal operating conditions. In thiscontext, reproducibility refers to the use of the analyticalprocedure in different laboratories, as in a collaborative study.3.2.10 quantification limi

19、tthe lowest level of a sampleproperty which can be determined with acceptable precisionand accuracy under the stated experimental conditions.3.2.11 rangethe interval between the upper and lowerlevels of a property (including these levels) that has beendemonstrated to be determined with a suitable le

20、vel of preci-sion and accuracy using the method as specified.3.2.12 reference valuethe metric of a property as deter-mined by well-characterized method, the accuracy of whichhas been stated or defined, that is, another, already-validatedmethod.3.2.13 validationthe statistically quantified judgment t

21、hatan empirically derived multivariate calibration is applicable tothe measurement on which the calibration is to be applied andcan perform property estimates with, in the case of quantitativecalibrations, acceptable precision, accuracy and bias or, in thecase of qualitative calibrations, acceptable

22、 Positive FractionIdentified and Negative Fraction Identified, as compared withresults from an accepted reference method.4. Summary of Practice4.1 Validating an empirically derived multivariate calibra-tion (model) consists of four major procedures: validation atinitial development, revalidation at

23、initial deployment or aftera revision, ongoing periodic revalidation, and qualification ofeach measurement before using the calibration to estimate theproperty(s) of the sample being measured.5. Significance and Use5.1 This practice outlines a universally applicable procedureto validate the performa

24、nce of a quantitative or qualitative,empirically derived, multivariate calibration relative to anaccepted reference method.5.2 This practice provides procedures for evaluating thecapability of a calibration to provide reliable estimationsrelative to an accepted reference method.5.3 This practice pro

25、vides purchasers of a measurementsystem that incorporates an empirically derived multivariatecalibration with options for specifying validation requirementsto ensure that the system is capable of providing estimationswith an appropriate degree of agreement with an acceptedreference method.5.4 This p

26、ractice provides the user of a measurement systemthat incorporates an empirically derived multivariate calibra-tion with procedures capable of providing information that maybe useful for ongoing quality assurance of the performance ofthe measurement system.5.5 Validation information obtained in the

27、application ofthis practice is applicable only to the material type and propertyrange of the materials used to perform the validation and onlyfor the individual measurement system on which the practice iscompletely applied. It is the users responsibility to select theproperty levels and the composit

28、ional characteristics of thevalidation samples such that they are suitable to the applica-tion. This practice allows the user to write a comprehensivevalidation statement for the analyzer system including specificlimits for the validated range of application and specificrestrictions to the permitted

29、 uses of the measurement system.Users are cautioned against extrapolation of validation resultsbeyond the material type(s) and property range(s) used toobtain these results.5.6 Users are cautioned that a validated empirically derivedmultivariate calibration is applicable only to samples that fallwit

30、hin the subset population represented in the validation set.The estimation from an empirically derived multivariate cali-bration can only be validated when the applicability of thecalibration is explicitly established for the particular measure-ment for which the estimation is produced. Applicabilit

31、ycannot be assumed.6. Methods and Considerations6.1 When validating an empirically derived multivariatecalibration, it is the responsibility of the user to describe themeasurement system and the required level of agreementbetween the estimations produced by the calibration and theaccepted reference

32、method(s).6.2 When validating a measurement system incorporatingan empirically derived multivariate calibration, it is the respon-sibility of the user to satisfy the requirements of any applicabletests specific to the measurement system including any Instal-lation Qualification (IQ), Operational Qua

33、lification (OQ), andPerformance Qualification (PQ) requirements; which may bemandated by competent regulatory authorities, an applicableQuality Assurance (QA), or Standard Operating Procedure(SOP) or be recommended by the instrument or equipmentmanufacturer.6.3 Reference Values and Quality Controls

34、for the AcceptedReference Method:6.3.1 The reference (or true) value which is compared witheach respective estimate produced by the empirically derivedmultivariate calibration is established by applying an acceptedreference method, the characteristics of which are known andstated, to the sample from

35、 which the measurement systemderives the measurement.6.3.2 To ensure the reliability of the reference valuesprovided by an accepted reference method, appropriate qualitycontrols should be applied to the accepted reference method.7. Procedure7.1 The objective of the validation procedure is to quantif

36、ythe performance of an empirically derived multivariate calibra-tion in terms of, in the case of quantitative calibrations,precision, accuracy and bias or, in the case of qualitativecalibrations, Positive Fraction Identified and Negative FractionIdentified relative to an accepted reference method fo

37、r eachproperty of interest. The user must specify, based on theintended use of the calibration, acceptable precision and bias orPositive Fraction Identified and Negative Fraction Identifiedperformance criteria before initiating the validation. Thesecriteria will be dependent on the intended use of t

38、he analyzerand may be based, all or in part, on risk based criteria.E 2617 09a27.1.1 The acceptable performance criteria specified by theuser may be constant over the entire range of sample variabil-ity.Alternatively, different acceptable performance criteria maybe specified by the user for differen

39、t sub-ranges of the fullsample variability.7.2 Validation of calibration is accomplished by using thecalibration to estimate the property(s) of a set of validationsamples and statistically comparing the estimates for thesesamples to known reference values. Validation requires thor-ough testing of th

40、e model with a sufficient number of repre-sentative validation samples to ensure that it performs ad-equately over the entire range of possible sample variability.7.3 Initial Validation Sample Set:7.3.1 For the initial validation of a multivariate model, anideal validation sample set will:7.3.1.1 Co

41、ntain samples that provide sufficient examples ofall combinations of variation in the sample properties whichare expected to be present in the samples which are to beanalyzed using the calibration;7.3.1.2 Contain samples for which the ranges of variation inthe sample properties is comparable to the

42、ranges of variationexpected for samples that are to be analyzed using the model;7.3.1.3 Contain samples for which the respective variationsof the sample properties are uniformly and mutually indepen-dently distributed over their full respective ranges or, whenapplicable, subranges of variation; and7

43、.3.1.4 Contain a sufficient number of samples to statisti-cally test the relationships between the measured variables andthe properties that are modeled by the calibration.7.3.2 For simple systems, sufficient validation samples cangenerally be obtained to meet the criteria in 7.3.1.1-7.3.1.4. Forcom

44、plex mixtures, obtaining an ideal validation set may bedifficult if not impossible. In such cases, it may be necessary tovalidate discrete subranges of the calibration incrementally,over time as samples become available.7.3.3 The number of samples needed to validate a calibra-tion depends on the com

45、plexity of the calibration, the ranges ofproperty variation over which the calibration is to be applied,and the degree of confidence required. It is important tovalidate a calibration with as many samples as possible tomaximize the likelihood of challenging the calibration withrarely occurring, but

46、potentially troublesome samples. Thenumber and range of validation samples should be sufficient tovalidate the calibration to the statistical degree of confidencerequired for the application. In all cases, a minimum of 20validation samples is recommended. In addition, the validationsamples should:7.

47、3.3.1 Multivariately span the ranges of sample propertyvalues over which the calibration will be used; that is, the spanand the standard deviation of the ranges of sample propertyvalues for the validation samples should be at least 100 % ofthe spans of the sample property values over which thecalibr

48、ation will be used, and the sample property values for thevalidation samples should be distributed as uniformly aspossible throughout their respective ranges, and the variationsof the sample property values among the samples should be asmutually independent as possible; and7.3.3.2 Span the ranges of

49、 the independent variables overwhich the calibration will be used; that is, if the range of anindependent variable is expected to vary from a to b, and thestandard deviation of the independent variable is c, then thevariations of that independent variable in the set of validationsamples should cover at least 100 % of the range from a to b,and should be distributed as uniformly as possible across therange such that the standard deviation in that independentvariable estimated for the validation samples will be at least95 % of c.(1) When validating a calibration for w