1、Ch19. Evaluation Criteria for BCI Research,Contents,Introduction Criteria for Evaluating Trial-based BCI Data Criteria for Evaluating Self-Paced BCI Data Other criteria,Introduction,The factors affect BCI performance Trial-based (system-paced) Asynchronous mode (self-paced) Type and number of EEG fe
2、ature Spectral parameter Slow cortical potentials Spatiotemporal parameters Nonlinear features Type of classifier Linear and quadratic discriminate analysis Support vector machines Neural networks Simple threshold detection Target application,Introduction,The necessity of consistent evaluation crite
3、ria For compare different BCI systems and approaches Consideration for evaluation criteria What is being evaluated Feedback loop The most frequently used evaluation criterion Error rate or accuracy Response speed of BCIs Evaluation of asynchronous BCI data,Criteria for Evaluating Trial-based BCI Dat
4、a,The Confusion Matrix Classification Accuracy and Error Rate Cohens Kappa Coefficient Mutual Information of a Discrete Output,The Confusion Matrix,TP: True Positive TN: True Negative FP: False Positive FN: False Negative FA: False Activation CR: Correct Rejection,Correctly classified,Incorrectly cl
5、assified,Classification Accuracy and Error Rate,The most widely used evaluation criteria in BCI research Denoted as ACC for classification ACCuracy, ERR(=1-ACC) for error rate Can be very easily calculated and interpreted The accuracy is 100%/M (M: number of class) The maximum accuracy can never exc
6、eed 100%Some limitation The off-diagonal values of the confusion matrix are not considered Classification accuracy of less frequent classes have small weight,Correctly classified sample,Total number of sample,Cohens Kappa Coefficient,Addresses several of the critiques on the accuracy measure Use the
7、 overall agreement po, and the chance agreement pe,Predicted classes show no correlation with actual classes,Perfect classification,Different assignment between output and the true classes,Sum of i-th column,Sum of i-th row,Standard error of kappa coefficient,Cohens Kappa Coefficient,Address several
8、 of the criticisms of the accuracy measure It considers the distribution of the wrong classifications = i.e., off-diagonal elements of the confusion matrix) Frequency of occurrence is normalized for each class = class with less samples get the same weight as class with many samples The standard erro
9、r of the kappa coefficient easily can be used for comparing whether the results of distinct classification systems have statistically significant differences,Mutual Information of a Discrete Output,Assume following things BCI system can be modeled as communication channel Communication theory of Sha
10、nnon and Weaver (1949) can be applied directly to quantify the information transfer Farwell and Donchin (1988) Information transfer for M classes can be calculated asProblms The information rate assume an error-free system This suggestion is not useful for comparing different BCI systems,Mutual Info
11、rmation of a Discrete Output,Based on Pierce (1980), Wolpaw et al. (2000a) Information transfer rate for M classes an ACC = po isThe formula has following limitations M selections (classes) are possible Each class has the same probability The specific accuracy is same for each class Each undesired s
12、election must have the same probability of selection Often these assumption are not fulfilled,It does not satisfy 3rd and 4th condition,Mutual Information of a Discrete Output,Random variable X models the user intension Random variable Y models the classifier output The entropy H(X) of a discrete ra
13、ndom variable is defined asNykopp(2001) derived the information transfer for a general confusion matrix,Mutual information,a priori probability for class xi,Probability to classify xi as yj,Criteria for Evaluating Self-paced BCI Data,Asynchronous mode BCI The BCI system is specially designed to prod
14、uce outputs in response to intentional control HF-Difference Hit False difference H : hit rate, F: false detection rate,Other Criteria,Receiver-Operator Characteristics (ROC) Correlation Coefficient Evaluation of Continuous-Input and Continuous-Output Systems Response time High accuracy is important But the response time is also important Maximum Steepness of the Mutual Information is used in BCI Competition III,t0 : time for the cue onset I(t) : continuous mutual information,
