1、Assessment embedded in step-based tutors (SBTs),CPI 494 Feb 12, 2009 Kurt VanLehn ASU,Outline,Benefits of assessment embedded in SBT Synthesis of several common ITS assessment methods Terminology Where does assessment fit into an ITS? A synthesis,Why should step-based tutors (SBT) be better than mul
2、tiple choice tests (MCT) for assessment?,Why is a 30 minute task (SBT) better than a 1 minute task (MCT)? Authenticity, validity, prediction of valued tasks “teaching to the test” becomes valuable Why is seeing the steps better than seeing just the answer? Assess problem solving strategy Assess meta
3、-cognitive strategy (e.g., use of hints),Why is embedding assessment in daily homework or seatwork better than monthly in-class testing?,Frees up time for teaching Cant be absent on day of test “doesnt test well” and “only good at tests” are no longer excuses No test anxiety No cramming Instructors
4、can see changes in competence daily,Homework is unsupervised but testing is supervised. How does that affect assessment?,What if students refer to textbook, web, friends or instructor? Sequestered problem solving is inauthentic Assessment may improve if tutor can “see” such references What about che
5、ating? Technology can thwart most copying Would have to hire a friend to do almost all of ones homework,Outline,Benefits of embedding assessments in SBTs Synthesis of several common ITS assessment methods Terminology Where does assessment fit into an ITS? A synthesis,Definition: Knowledge component,
6、A piece of knowledge Generic can be applied in to multiple tasks Any form: rule, principle, concept, fact, procedure Examples: Two angles are called “supplementary” if. To buckle your seat belt, insert the The gravitational constant of earth, g, is 9.8 m/s2.,Definition: Mastery of a knowledge compon
7、ent (KC),No more training needed E.g., The ITS need no longer pick problems that involve it Typical signs of mastery Applies the KC when should Applies the KC only when should Applies without referring to text, notes, Can explain the KC, when to apply, etc.,Main goal of assessment,Never have enough
8、data E.g., typically do not have students explanations Must estimate P(mastery) for every knowledge component given only steps as data,Definition: The response pattern of a students step,How a step was accomplished by the student E.g., 4 response patterns: (S = student; T = tutoring system) S: Corre
9、ct. S: Error; T: Hint; S: Correct. S: Help; T: Hint; S: Correct. S: Help; T: Hint; S: Help; T: Hint; S: Help; T: Bottom out hint; S: Correct,Definition: The relevant knowledge components of a students step,The knowledge components involved in deriving the correct step given the information available
10、 at the time the step was done The knowledge the student should apply to generate this step The topic of the hints (if any were given),Outline,An introduction to a common type of ITS Terminology Three examples Synthesis of several common ITS assessment methods Terminology (KC, mastery, response patt
11、ern, KCs relevant to a step) Where does assessment fit into an ITS? A synthesis,The 3 major computations of intelligent tutoring systems,The experts computation,Expert can be humans or an expert system Solve the problem in all acceptable ways Record steps taken Record knowledge components used at ea
12、ch step,The helpers computation,When the student enters a step, match it to a correct step Give feedback & hints as necessary Record response pattern,The assessors computation,Given Reponse patterns for each step taken by the student Old P(mastery) for each knowledge component Calculate New P(master
13、y),Outline,An introduction to a common type of ITS Terminology Three examples Synthesis of several common ITS assessment methods Terminology Where does assessment fit into an ITS? A synthesis,Some assessment methods,Knowledge tracing (Corbett & Anderson) - Only two response patterns: “S:Correct” & “
14、other” - Only one relevant knowledge component per step,SMART (Shute) - Multiple response patterns,Andes (VanLehn et al.) - Multiple knowledge components per step,Today: A synthesis of the above,DT tutor (Murray) - Dynamic Bayesian networks,The problem: Given step responses & prior P(mastery), calcu
15、late posterior P(mastery),KC1,KC3,KCn,KC2,KC4,Step 1 S: error; T: Hint; S: Correct,Step 2 S: Correct,Step 3 S: Help; T: Hint; S: Help; T: Hint; S: Help; T: B.O. Hint; S: Correct,.05,.05,.11,.10,.05,.75,.12,.50,.99,.05,Prior P(mastery) for each knowledge component (KC),Posterior P(mastery) for each k
16、nowledge component,A dynamic Bayesian network represents that P(mastery) changes during steps,Bayesiannetwork,Bayesian network,Step 1,Step 2,P(mastery) of knowledge component 4 at time 1,P(mastery) of knowledge component 4 at time 2,Assume P(mastery) changes only for relevant knowledge components,Ba
17、yesian network,Bayesian network,Step 1 Involves only KC2 and KC3,Step 2 Involves only KC1 and KC4,Knowledge tracing: Assumes just one relevant knowledge component per step,Bayesian network,Bayesian network,Step 1 Involves only KC2,Step 2 Involves only KC4,Knowledge tracing: Network topology and cond
18、itional probability tables,Step 1,Step 2,Step response,Step response,P(Step response not correct | KC2 mastered) Called “slip parameter”,P(Step response correct | KC2 not mastered) Called “guess parameter”,P(Mastered | Was not mastered, relevant to step) Called “acquisition parameter”,Need to allow
19、multiple relevant knowledge components per step,Step response,Step 1,Step 2,Step response,Use AND node to represent that the step response depends only on whether or not all relevant knowledge components are mastered,The students meta-cognitive strategy also affects the students step response,Step r
20、esponse,Step 1,Step 2,Step response,Meta-cognitive strategy,Conditional probability table for the first “Step response” node,P(step response was “S:Correct” | Meta-cognitive strategy was “Learn”, Not all knowledge components mastered before step,Strong tendency to abuse help when the meta-cognitive
21、strategy is “Perform” and some relevant knowledge component is not mastered yet.,Need to assume new P(mastery) depends on the step response as well as the old P(mastery),Step response,Step 1,Step 2,Step response,Meta-cognitive strategy,Conditional probability table for a “knowledge component” node,H
22、ow the network should be used,Given log file all steps taken by the student Construct the whole network all steps in all problems Enter prior P(mastery) on knowledge components at time 1 Clamp all step responses Update a huge computation Read out P(mastery) on knowledge components after last step.,H
23、ow the network is actually used,Step 1 Construct network Enter prior P(mastery) Clamp step 1 response Update Read out P(mastery) Step 2 Construct network Enter prior P(mastery) Clamp step 2 response Update Etc.,Future work,Summary,An introduction to a common type of ITS Steps replace answers as main student response A synthesis of several Bayesian assessment methods Many knowledge components Probability of mastery of each knowledge component One (or more) knowledge components per step Can use step response patterns to infer mastery Global traits e.g., meta-cognitive strategy,
