1、,California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO Educational Benchmarking (EBI) June 25, 2004,Fundamentals of Structuring and Implementing Successful First Year Seminars,The statistical analysis for this presentation was based on the Educational Benchma
2、rking (EBI) First Year Initiative (FYI) Assessment and conducted by Randy L. Swing, Ph.D. Co-Director, Policy Center on the First Year of College & Fellow, The National Resource Center on The First-Year Experience and Students in Transition in conjunction with Educational Benchmarking,Value of a Col
3、laborative Group,Shared mission and purpose Common problems and barriers Varied expertise Benchmarks for performance,Direct Quote: “The elements that research indicates are the most important to have as components of an effective first year program.”,Presentation Overview,The elements that research
4、indicates are the most important to have as components of an effective first year program. Principles for leveraging assessment to initiate and sustain improvement,FYS Goals,Retention 37% Academic Achievement 37% Participation in College Life 11%The Policy Center on the First College Year Ease trans
5、ition to college Increase “student skills” Increase persistence rates Increase graduation rates,First-Year Initiative Survey FYI,Developed by the Policy Center on the First College Year and Educational Benchmarking (EBI) 62 institutions (limited to 4-year institutions*) Over 30,000 students 7- point
6、 scale Learning Outcome Factors Administered in the last week of fall 2001,*4-year or 2-year regional campuses - a 2-year version is in development.,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Te
7、aching Assistants,FYI Assessment and Research Context,The following principles and practices provide the foundation to the research and the First Year Initiative Assessment (FYI),Research/Assessment Context,Factors: A more powerful indicator than individual question results. 2. Regression: Allows yo
8、u to identify predictors of performance Predictor Factors: Factors, if improved, have the greatest impact on improving overall effectiveness,Factors,Factors (also called “constructs”) are groupings of related questions that share a relationship. The basic assumption of factor analysis is that underl
9、ying dimensions, or factors, can be used to explain more complex phenomena.,Factor Means Course Learning Outcomes,mean Course Improved Knowledge of Campus Services . . . . . 4.72 Course Improved Knowledge of Campus Policies. . . . . 4.58 Course Improved Connections with Peers. . . . . 4.54 Course Im
10、proved Connections with Faculty. . . . 4.50 Course Improved Managing Time/Priorities. . . . 4.50 Course Improved Critical Thinking. . . . 4.40 Course Improved Study Strategies. . . .4.19 Course Improved Knowledge of Wellness Issues. . . . 3.90 Course Improved Academic/Cognitive Skills. . . . 3.74 Co
11、urse Increased Out-of-Class Engagement. . . .3.63,1.0 2.0 3.0 4.0 5.0 6.0 7.0,1/2,3/4,1/4,Factor Reliability,Once it has been determined that a certain set of questions do share a relationship and therefore constitute a factor, there is an additional (and necessary) statistical test to assess the ps
12、ychometric soundness of the factor.,A Cronbachs Alpha of zero would mean that there is no internal consistency at all, i.e., subjects are likely to respond with anything from 1 to 7 on any of the questions in a factor with no discernable pattern. An Alpha of 1 would mean that every subject answered
13、every question comprising the factor consistently (e.g., all subjects answered with all 7s, or all 1s). This is a highly unlikely event.,Factor Reliability,FYI Factor Reliabilities,Factor Descriptions Reliability Course Improved Study Strategies 0.90 Course Improved Academic and Cognitive Skills 0.8
14、9 Course Improved Critical Thinking 0.91 Course Improved Connections with Faculty 0.84 Course Improved Connections with Peers 0.90 Course Increased Out-of-Class Engagement 0.90 Course Improved Knowledge of Campus Policies 0.90 Course Improved Knowledge of Academic Services 0.87 Course Improved Manag
15、ing Time and Priorities 0.92 Course Improved Knowledge of Wellness 0.90 Sense of Belonging and Acceptance 0.90 Usefulness of Course Readings 0.90 Satisfaction with College/University 0.90 Course Included Engaging Pedagogy 0.92 Overall Course Effectiveness 0.92,Regression,A correlation establishes th
16、e relationship between two variables. Regression analysis, by contrast, allows us to determine the relationship between some dependent variable (e.g. overall effectiveness) and multiple independent variables (e.g., engaging pedagogy, etc.).,Major predictors identify, in order of descending importanc
17、e, the factors that have the greatest impact on overall effectiveness. While minor predictors are statistically significant, individually and combined they add little to the predictability of overall effectiveness.,Regression,Predictors,Therefore, the return on the investment for improving the major
18、 predictors would far outweigh the return on improving the minor predictors.,Top Priority High Impact, Low Performance,Monitor Low Impact, Low Performance,Maintain Low Impact, High Performance,Maintain or Improve High Impact, High Performance,Impact: Predictors of Overall Satisfaction,Performance,4
19、Quadrants,Priority Matrix,Top Priority High Impact, Low Performance,Monitor Low Impact, Low Performance,Maintain Low Impact, High Performance,Maintain or Improve High Impact, High Performance,Impact: Predictors of Overall Satisfaction,Performance,Priority Matrix,Engaging Pedagogy #1 Predictor of Cou
20、rse Effectiveness,Engaging Pedagogy Factor,To what degree did the course include: A variety of teaching methods Meaningful class discussions Challenging assignments Productive use of classroom time Encouragement to speak in class Encouragement for students to work together Meaningful homework,Issues
21、 Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants,Four Types of Academic Seminars,COLLEGE TRANSITION THEME orientation/study skills/managing transitions SPECIAL ACADEMIC THEME interdiscip
22、linary/problem-focused/selected topic DISCIPLINE BASED introduction to major/department/discipline REMEDIAL/STUDY SKILLS study skills for a high risk population,What is a First-Year Seminar?,A course offered specially for new students to assist with the transition into college.,Four Types of Seminar
23、s,College Transition Theme courses deal directly with orientation to college and academic skills. This is the University 101 (USC) model.Special Academic Theme courses deal with a selected topic other than college transition. These are often taught as interdisciplinary seminars where a small group o
24、f students and a model learner/teacher use a variety of methods to investigate an important theme.Discipline based are often an introduction to a major or department. They are based in individual academic departments.Remedial/Study Skills, was considered, but there were too few cases to draw any mea
25、ningful conclusions.,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,19.2 13.3 27.9 28.2 34.4 18.0 34.1 39.4 26.4 23.0 58.5 3
26、3.6 30.5,17.3 16.0 31.1 27.7 32.1 12.7 19.2 22.5 23.1 15.5 55.1 37.0 36.5,9.7 9.1 16.8 19.7 17.2 10.0 35.9 32.2 15.3 10.7 52.8 21.3 18.3,Theme Format: Percent of students with mean 5.50 or greater,Special Transition Academic Discipline,FYI Finding,Transition-theme and Special Academic-theme courses
27、were about equal on learning outcomes and student satisfaction.,Discipline-theme courses produced lower learning outcomes and student satisfaction.,Remedial courses too few in the study to draw valid conclusions.,Why? Engaging Pedagogy explains the difference. Transition 30.5% Special Academic 36.5%
28、 Discipline 18.3%,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants,Credit Hours 2001 Pilot Administration % of institutions (N=62),No credit,4 or more credits,1 credit,2 credits,3
29、credits,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,Percent of students mean 5.50 or greater,15.9 20.6 19.4 10.9 12.7 17.
30、3 23.1 26.1 34.2 24.9 28.3 29.6 26.1 35.5 37.3 14.8 18.5 18.6 33.3 35.3 30.4 35.9 41.0 35.4 24.6 27.7 25.2 18.5 24.0 23.0 55.3 58.5 60.2 28.1 34.9 35.4 24.1 30.1 37.0,Contact Hour(s) 1 Hr 2 Hrs 3 Hrs,1 contact hour courses Orientation to Services,2 contact hour courses Study Strategies Peer Connecti
31、ons Faculty Connections,3 contact hour courses Academic Skills Critical Thinking,most effective with learning outcomes associated with a basic orientation to campus services.,perform well at producing effective study strategies, peer connections, and faculty connections,significantly impact academic
32、 skills (reading, writing, oral presentation skills) and critical thinking skills,FYI Finding,The number of contact hours should match the intended goals of a first-year seminar.,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/Not Required Letter Graded/Pass-Fail Linked C
33、ourse Format Undergraduate Teaching Assistants,Required/ Not Required 2001 Pilot Administration % of institutions (N = 62),Cal FYI 42% 37% No students in any section required to enroll 32% 11% Some students in some sections required to enroll 16% 35% All/most students in every section required to en
34、roll 10% 17% Mixed formats no one format constitutes 80%,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,Required,Not Require
35、d (Elective),Percent of students mean 5.50 or greater,16.1 12.7 27.8 24.8 27.4 12.9 25.9 27.7 22.6 16.7 56.0 27.9 29.0,19.5 14.0 28.1 29.4 38.0 17.8 34.4 42.4 26.7 24.0 58.2 37.7 31.9,Required Not Required Study Strategies 4.18 4.43 Academic Skills 3.47 3.76 Critical Thinking 4.36 4.56 Faculty Conne
36、ctions 4.35 4.66 Peer Connections 4.31 4.84 Out-of-Class 3.36 3.76 Policies/Procedures 4.28 4.66 Campus Services 4.35 4.93 Time/Priorities 4.27 4.57 Wellness/Spirituality 3.59 4.14 Belonging 5.41 5.51 Course Satisfaction 4.29 4.88 Engaging Pedagogy 4.43 4.74,Factor Means for Required/not required Co
37、ntrolled for Contact Hrs, Theme-types, and Grade-formats,Required Not Required Study Strategies 4.18 4.43 Academic Skills 3.51 3.72 Critical Thinking 4.39 4.55 Faculty Connections 4.37 4.65 Peer Connections 4.34 4.83 Out-of-Class 3.36 3.77 Policies/Procedures 4.21 4.72 Campus Services 4.29 4.98 Time
38、/Priorities 4.26 4.59 Wellness/Spirituality 3.55 4.18 Belonging 5.43 5.51 Course Satisfaction 4.32 4.86 Engaging Pedagogy 4.47 4.70,Factor Means for Required/not required Controlled for student characteristics (gender, race/ethnic, HS Grades, and commuter/residential),FYI Finding Courses that are NO
39、T REQUIRED . . . produced greater outcomes than required courses. WHY?,Engaging Pedagogy Required Not Required 4.48* 4.71* * after controlling for differences in grading and contact hours,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fai
40、l Linked Course Format Undergraduate Teaching Assistants,Grading 2001 Pilot Administration % of institutions (N = 62),82% Letter Graded 16% Pass/Fail 2% Mixed,Cal,47%,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Servi
41、ces Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,Graded,Pass/Fail,Mean Scores * controlling for Contact Hrs, Required, UGTAs, & Themes,4.25 4.33 3.63 3.45 4.40 4.39 4.49 4.54 4.56 4.45 3.68 3.44 4.51 4.79 4.70 4.79 4.38 4.50 3.93 3.81 5.48 5.41 4.56 4.62 4.58
42、 4.43,FYI Finding,Overall, grading format produces mixed results, even when controlled for Contact Hours, Theme-types, Required, and UGTAs.,Key Finding: Graded courses are associated with higher scores on Engaging Pedagogy,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/
43、Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants,Linked Courses 2001 Pilot Administration % of institutions (N = 62),Few or no sections linked,Some Linked,Most Linked,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Ou
44、t-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,Linked,Not Linked,Mean Scores controlling for Required, Grading, Contact, & Theme,4.46 4.38 3.78 3.64 4.62 4.50 4.60 4.65 5.02 4.63 3.63 3.81 4.55 4.70 4.75 4.91 4.60
45、4.57 4.02 4.15 5.55 5.50 4.89 4.72 4.76 4.66,FYI Finding Linking the seminar to other courses produced greater learning outcomes for: Academic Skills Study Skills Critical Thinking Engaging Pedagogy,Limitations in this study reduce the capability to make definitive statements about the impact of lin
46、king courses.,Issues Addresses by FYI Findings,Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants,Use of Undergraduate Teaching Assistants,Less than 80% of sections have UTAs,No sections have UTAs,More than 80% of s
47、ections have UTAs,Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy,UGTA,No UGTA,Mean Scores controlling for Required, Grading,
48、 Contact, & Theme,4.28 4.31 3.56 3.74 4.40 4.58 4.54 4.44 4.66 4.42 3.73 3.23 4.51 4.21 4.83 4.23 4.47 4.30 4.04 3.49 5.48 5.41 4.57 4.51 4.57 4.60,FYI Finding,Undergraduate Teaching Assistants are associated with higher mean learning outcomes - except for academic skills, critical thinking skills,
49、and engaging pedagogy.even after controlling for:Required/Not Grading format Theme Contact Hours,Summary: Engaging Pedagogy best predicts the learning outcomes and student satisfaction with the seminar,Student rating of Engaging Pedagogy: No significant difference by gender African-Americans, Latino, and Native Americans gave higher ratings than Whites and Asians “A” high school students gave lower ratings,Credible Confidential Comparative Comprehensive Continuous,
