1、Designation: E3000 17Standard Guide forMeasuring and Tracking Performance of Assessors on aDescriptive Sensory Panel1This standard is issued under the fixed designation E3000; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the yea
2、r of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This guide provides guidelines for measuring and track-ing the performance of individual assessors on a descripti
3、vesensory panel.1.2 This guide provides guidelines to assist sensory profes-sionals in measuring performance for given assessors. Measur-ing performance will form the basis for (1) determining thereliability of the results, and (2) establishing remedial actionsfor an individual assessor.1.3 This gui
4、de examines various aspects of trained assessorperformance; such as repeatability, discrimination, and agree-ment and demonstrates some ways to measure them. Theprocedures will help the sensory professional determine areasof good performance as well as those that require improve-ment.1.4 Individual
5、assessor performance is tracked using estab-lished statistical procedures. These procedures depend onwhether replicates are collected and if they are collected overmultiple sessions or within a single session.1.5 This guide provides suggested procedures, includingstatistical procedures that can be d
6、one using standard statisticalsoftware, for evaluating performance and is not meant toexclude other methods that may be effectively used for asimilar purpose.1.6 Methods for training and screening assessors are notwithin the scope of this guide. This guide does not address howto communicate performa
7、nce feedback information to indi-vidual assessors. This monitoring of panel reproducibility, ameasure of the panels ability to reproduce the results of otherpanels, is also not within the scope of this guide.1.7 This standard does not purport to address all of thesafety concerns, if any, associated
8、with its use. It is theresponsibility of the user of this standard to establish appro-priate safety, health, and environmental practices and deter-mine the applicability of regulatory limitations prior to use.1.8 This international standard was developed in accor-dance with internationally recognize
9、d principles on standard-ization established in the Decision on Principles for theDevelopment of International Standards, Guides and Recom-mendations issued by the World Trade Organization TechnicalBarriers to Trade (TBT) Committee.2. Referenced Documents2.1 ASTM Standards:2E253 Terminology Relating
10、 to Sensory Evaluation of Mate-rials and ProductsE456 Terminology Relating to Quality and Statistics2.2 Other Documents:2ASTM STP 758 Guidelines for the Selection and Training ofSensory Panel MembersASTM MNL13 Manual on Descriptive Analysis for SensoryEvaluation2.3 ISO Standards:3ISO 11132:2012 Sens
11、ory Analysis MethodologyGuide-lines for Monitoring the Performance of a QuantitativeSensory Panel3. Terminology3.1 Please refer to Terminologies E253 and E456, ASTMSTP 758, ASTM MNL13 and ISO 11132:2012 for any termsrelated to assessor performance that are not listed below.3.2 Definitions:3.2.1 agre
12、ementability of an assessor to give similarscores (rate) or to order the intensity of stimuli similarly to therest of the panel (rank) on a given attribute.3.2.2 performanceability of an assessor to make repeat-able assessments that are in agreement with other assessors onthe panel and discriminate
13、perceptible differences betweenattributes when they are present.1This guide is under the jurisdiction of ASTM Committee E18 on SensoryEvaluation and is the direct responsibility of Subcommittee E18.03 on SensoryTheory and Statistics.Current edition approved Nov. 1, 2017. Published December 2017. DOI
14、:10.1520/E3000-17.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book of ASTMStandards volume information, refer to the standards Document Summary page onthe ASTM website.3Available from International Organization
15、for Standardization (ISO), ISOCentral Secretariat, BIBC II, Chemin de Blandonnet 8, CP 401, 1214 Vernier,Geneva, Switzerland, http:/www.iso.org.Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United StatesThis international standard was developed i
16、n accordance with internationally recognized principles on standardization established in the Decision on Principles for theDevelopment of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.13.2.3 scale usagethe exte
17、nt to which the assessor(s) usesthe scale with respect to the intensities of the attributes beingmeasured.4. Summary of Practice4.1 The protocols described in this guide provide a proce-dure for quantitatively establishing the performance of indi-vidual assessors by discussing the minimum level of g
18、oodperformance, determining when a performance problem exists,and detailing specific procedures to address those problems.5. Significance and Use5.1 This guide is meant to be used with and applied toindividual trained descriptive assessors.5.2 The procedures recommended in this guide can be usedby t
19、he panel leader to periodically appraise the performance ofindividual descriptive assessors.5.3 Tracking assessor performance will provide informationas to the quality of the data being generated. Performanceinformation may be used to decide whether to use the data tointerpret product profiles.5.4 M
20、onitoring assessor performance will enable the panelleader to identify retraining needs or to identify assessors whoare not performing well enough to continue participating on apanel.6. Performance6.1 Introduction:6.1.1 This section provides sensory approaches for theassessment of assessor performan
21、ce. It is assumed that goodsensory practices are being followed in order to allow for goodassessor performance. Panel members must be motivated tocarry out the job conscientiously, be in good health bothphysically and mentally, and must be willing and able to followinstructions. Standard procedures
22、to reduce random variabilityand systematic bias, including robust experimental design toreduce order and carry over effects must be followed by thesensory professional.6.1.2 Assessor performance is the measure of the ability ofan assessor to make reliable attribute assessments across theproducts bei
23、ng evaluated. It can be measured at a given timepoint or tracked over time. Performance is compromised if anassessor cannot repeat their own results (repeatability), dis-criminate among the products (discrimination) and assessstimuli similarly to other assessors on a given attribute (agree-ment). Th
24、is guide will focus on these three key measures.These measures can allow the sensory professional to diagnosesources of poor performance such as the inability to use a scaleto correctly indicate intensity (scale usage), and failure to useattributes similarly to other assessors (lexicon usage).6.1.3
25、It is important to track panel performance as a whole,since panel data are used for decision making; however, thisguide describes how to measure and interpret performancecriteria for the individual assessors, since assessor performanceinfluences panel results.An assessor who does not discriminateamo
26、ng the samples may impact the panel data, causing themean values for a specific attribute to be close together andpreventing overall discrimination between samples. In somecases, a poorly performing assessor can cause panel data to beinconsistent and non-repeatable. All of the assessors must beusing
27、 the vocabulary in the same way and utilizing the scalesin a consistent manner in order for the panel to succeed.6.2 Individual Assessor Performance:6.2.1 In the early stages of training, performance evaluationshould be analyzed for each individual assessor prior toparticipation in a panel. During t
28、his phase, the panel leadertypically monitors panel agreement on ranking or rating stimulifor intensity and on scale usage. Specific examples for eachattribute need to be introduced, experienced and defined toensure that all assessors understand the sensory qualities andrange of intensity of the att
29、ribute. During training, attributedefinitions and references should be reviewed and possiblyrevised, to ensure that attributes are understood and usedconsistently by individual assessors across all samples. Asses-sors should be selected for continued panel participation basedupon performance.6.2.2 O
30、nce assessors are trained, they should be monitoredfor the three key measures of performance (repeatability,discrimination, agreement). It is important to evaluate assessorperformance periodically in order to detect any change inindividual performance over time and to identify an assessor orassessor
31、s who are not performing well. Assessor performanceon an individual study should be monitored if you are makinghigh value or high risk business decisions with your panel data.The panel member should be engaged for a sufficient period tohave established a history of performance which has beenmonitore
32、d.6.2.3 In cases of poor performance, initially check with theindividual assessor for any reasons they may not have beenperforming as usual. This would indicate the need to eliminatetheir responses on relevant data sets.6.2.4 Rule out the possibility that assessor variation may bedue to potential va
33、riability within the samples. Some types ofproducts such as meat, seafood, or crop-based products can bequite variable and this variability must also be understoodbefore concluding that there is an issue with assessor or panelperformance.6.2.5 Verification of test procedures, such as correct samples
34、evaluated, correct instructions given, no data transcriptionerrors, should also be done.6.2.6 Fundamental issues such as insufficient training (is-sues with scale or lexicon usage), not understanding theprocedure, boredom, over-use, or being unable to perceivecertain attributes of the stimuli (physi
35、ological differences) canalso contribute to poor assessor performance. It is important toidentify early signs of performance inconsistency and correctthe problem before the assessor has an impact on the overallpanels results. Additional training should be given as a part ofpanel maintenance to addre
36、ss these issues. By correcting theproblem of the inconsistent assessor, one can achieve the aimof having a consistent panel.6.3 Key Measures of PerformanceThere are three mainelements of poor performancelack of repeatability, inabilityto discriminate, and lack of agreementthat should be exam-ined re
37、gularly.E3000 1726.3.1 RepeatabilityLack of repeatability occurs when as-sessor(s) cannot replicate their ratings from one evaluation toanother of the same sample. It should be noted that assessmentof repeatability is only possible if assessors evaluate the samesample on at least two occasions, eith
38、er during the samesession or on different sessions.6.3.1.1 Inadequate training, inconsistent scale usage, lexi-con usage, and various psychological and physiological factorscan impair repeatability. Assessor fatigue, improper spacing ofsamples, poor instructions, inconsistent reference samples, ands
39、ample variation can also contribute to the problem. Studyredesign or retraining, or both, may be necessary to reduce thevariation in repeatability.6.3.2 DiscriminationAn assessors inability to find sig-nificant differences among samples that are found to bedifferent by the panel as a whole may occur
40、 for several reasons,such as general or specific ageusia and anosmia or differencesin lexicon usage.6.3.2.1 Using the same ratings across all samples for anattribute may indicate low sensory acuity resulting in theassessors inability to use the scale as they were trained. Poorlydiscriminating assess
41、ors may use similar ratings across allsamples in a “safe scale range” to cover their inability todiscern the attribute.6.3.2.2 The non-discriminating attributes should be identi-fied and training provided to the assessor on those attributes forwhich the samples are expected to differ. It may be nece
42、ssaryto change the reference standards to better represent theattribute if previously used references are not helpful for thepanel.6.3.3 AgreementAgreement is obtained when assessorsrate samples similarly in relation to each other. Similar ratingsindicate that the assessors are scoring the samples c
43、onsistentlyfor each attribute.6.3.3.1 The data set should be carefully examined to deter-mine which individual assessors contribute to the dissimilarityof the attribute ratings. A lack of agreement may be due to adifference in the assessors discrimination, differences in scaleor lexicon usage, or bo
44、th.6.3.3.2 Sometimes the main cause of a lack of agreementmay not be due to a poor assessor, but rather, an assessor whomay be more discriminating or more sensitive to an attribute.The identification of the origin of a disagreement is thereforeessential for identification of the appropriate correcti
45、ve action.6.3.3.3 Assessors who vary on the perceived intensity inrelation to other assessors, but still show the same sampleranking pattern as the other assessors (magnitude typeinteractions), usually differ in scale usage. A disagreement inassessor ratings may also indicate that assessors do notas
46、sociate the same sensory perception with the attributes orvary on the perceived intensity due to individual differences insensory acuity, thus causing cross-over interactions. A cross-over interaction occurs when an assessors mean score for aspecific sample is reversed in response pattern from those
47、 ofother panel members. All cross-over interactions should becarefully examined since they reduce chances of the panelfinding significant sample differences.6.3.3.4 Lexicon usage may also contribute to agreementissues when one assessor understands the attribute to meansomething different from the ot
48、her panel members.6.4 Performance DiagnosticsScale usage and lexicon us-age are two diagnostics that can be examined to understandwhat is causing issues with agreement, discrimination, andrepeatability.6.4.1 Scale UsageInconsistent scale usage occurs whendifferent assessors use different ranges of t
49、he scale and alsodifferent areas or locations of the scale while rating the samesample (note: this is an assessor effect in ANOVA). Inconsis-tent scale usage for an overall panel can be consideredacceptable to a certain degree as long as assessors are consis-tent with their own behavior (across all samples) and are inagreement with the rest of the panel (for example, rank thesamples in the same order). Poor assessor calibration, inad-equate training, insensitivity or super-sensitivity to the problemattribute, or lack of reference standards is usually the source of
