Analyzing COSF Data in Support of System Validity.ppt

1、Analyzing COSF Data in Support of System Validity,Charles R. Greenwood & Dale Walker,Some of these data are published in Greenwood, C. R., Walker, D., Hornback, M., Nelson, C., Hebbeler, K., & Spiker, D. (2007). Progress developing the Kansas Early Childhood Special Education Accountability System:

2、Initial findings using the ECO Child Outcome Summary Form (COSF). Topics Early Childhood Special Education, 27(1), 2-18.,Margy Hornback (KS) Birth-5 Marybeth Wells (ID) Section 619,http:/www.fpg.unc.edu/ECO/,2,Purpose of this Presentation,Demonstrate analyses of COSF data Point out how analyses info

3、rm the validity of the States OSEP accountability system Help States conduct similar analyses,3,Validity of an Accountability System,An accountability system can be said to have validity when evidence is judged to be strong enough to support inferences that: The components of the system are aligned

4、to the purposes, and are working in harmony to help the system accomplish those purposes The system is accomplishing what was intended (and not what was not intended) (Marion et al., 2002, pg. 105),4,The Validity of an Accountability System,Requires answers to a number of logical questions demonstra

5、ting that the parts of the system are working in harmony as planned Validity is improved by improving the quality and integrity of the parts in the system Validity requires continued monitoring and improvement,5,COSF Validity Questions and Evidence,1. The Anchor Indicators used in the COSF Process a

6、re Mapped via a Cross-walk to the OSEP Outcomes Have the Anchor Indicators been cross-walked to the 3 OSEP outcomes? Do the Anchor Indicators have evidence of validity and reliability?,6,COSF Validity Questions and Evidence,2. The COSF Process Involves Multiple Participants and Sources of Evidence A

7、re multiple adults participating in the process? Are parents participating in the process? Are multiple sources of evidence being used?,7,Team Roles Make Up Roles less than 3% Collapsed to Other,8,Parents and Ratings,How many children have a parent providing the rating 731 out of 2388 (31%) Other Fa

8、mily Members? Foster Parent = 10 Grandparent = 12 Advocate = 5 Baby Sitter = 2,9,Evidence Sources Reported in One District (Part B),10,COSF Validity Questions and Evidence,3. COSF Ratings Should Display Differences Between Childrens Performance Is the distribution of COSF ratings normally distribute

9、d? Are fewer children scored 1 and 7, and more children scored 3, 4, and 5?,11,Part B Social Knowledge & Skills Meets Needs,Kansas Part B,12,Idaho Part B,13,Part C Social Knowledge & Skills Meets Needs,Kansas Part C,14,COSF Validity Questions and Evidence,4. OSEP Outcomes are Defined Functionally, T

10、herefore, They Should be Highly Inter-correlated Are the three outcomes highly inter-correlated? Does each outcome contribute unique information?,Correlations Between Entry Outcomes,Entry Correlations When Controlling for the Third Outcome,What is the commonality of shared variance in the entry Part

11、 B Social Outcome in ID?,Formulas for Unique and Commonality Components of Shared Variance:U1 = R2(12) R2(2); U2 = R2(12) R2(1); C12 = R2(1) + R2(2) R2(12) (Thompson, 2006 (pg 279),18,COSF Validity Questions and Evidence,5. COSF ratings should be at least moderately (not strongly) correlated with th

12、e anchor-primary assessment measure What is the concurrent validity correlation with the primary assessment measure? Is there a linear, increasing relationship between ratings and mean test scores?,19,Correlation between COSF Outcome Ratings And BDI Domain Scores Social vs. PerSocial = .65 Knowledge

13、 vs. Cognitive = .62 Meets Needs vs. Adaptive = .61,BDI Domain Means by COSF Rating,20,By Anchor Test (ID),21,By Anchor Tests in KS,22,COSF Validity Questions and Evidence,6. COSF Ratings Should Not Be Affected by Conditions in the States COSF Process Are there differences by region or program? Are

14、there differences due to use of different Anchor tests? Are there differences due to quality or intensity of training/fidelity in the COSF process?,23,District Comparison,24,District Comparison,25,Evidence Use Profiles for 3 largest Part C Entities,C0016,C0033,Z0042,26,COSF Validity Questions and Ev

15、idence,7. Theoretically, We Might Expect COSF Ratings to be Influenced by Differences in Sociodemographics Are there differences in COSF ratings due to type of disability? Are boys rated lower than girls on the Social Outcome? (boys tend to have more behavior problems than girls) Are English Languag

16、e Learners rated lower on the Knowledge and Skills Outcome? Do these variables explain significant variance in COSF Outcome at Entry and Exit?,27,By Gender (ID),28,By Disability (ID),29,By Race (ID),30,How much variance in entry rating do demographic variables explain?,31,COSF Validity Questions and

17、 Evidence,8. Theoretically, We Expect COSF Ratings Will Be Sensitive to Growth and Early Intervention Over Time Are COSF exit rating distributions skewed to the right, indicating children scoring higher at exit compared to entry? Are there gains in COSF ratings when comparing entry to exit? Are thes

18、e gains statistically significant, and what are the effect sizes?,32,Sample KS Entry and Exit Data,Sample ID Entry and Exit Data,33,What growth is evident? KS ID,34,What growth is evident: KS?,What growth is evident: ID?,35,What GAIN in Rating: KS?,What GAIN in Rating: ID?,36,What growth is evident:

19、 KS?,37,What growth is evident: ID?,38,Change in Social Distribution: KS,39,Change in Social Distribution: ID,40,Change in Knowledge Distribution: KS,41,Change in Knowledge Distribution: ID,42,Change in Meets Needs Distribution: KS,43,Change in Meets Needs Distribution: ID,44,What do the OSEP outcom

20、e category results look like?,Note: There were no Cat a. children,45,A States OSEP Outcome Distributions,46,COSF New Skills Coding Error to Check,Yes or No and the New Skills Question? No Means no new skills acquired, no can not be associated with ratings that go up from entry to exit (e.g., 3 to 4

21、always = yes) Yes Means new skills were acquired and in COST 7 to 6 (child means child is still typical) 2 to 2, 3 to 3, etc (staying the same rating in COSF = yes, new skills acquired) http:/www.fpg.unc.edu/ECO/pdfs/Summary_of_Rules_COSF_to_OSEP_8-9-07.pdf,47,COSF Validity Questions and Evidence (F

22、uture Inquiry),9 Theoretically, We Expect Gains in COSF Exit Ratings to be Explained by Early Intervention Factors Are gains in COSF ratings explained by length of service? Are gains in COSF ratings explained by intervention/program quality features (e.g., models, evidence-based practice, etc.)? Are

23、 gains in COSF ratings explained by family outcomes?,48,Conclusion,9 validity questions and supporting COSF evidence were discussed Such analyses help establish and maintain a states OSEP accountability system Evidence from two states appears to support the COSF process as a valid approach More work is needed, we need to know more from more states!,For More Information see: http:/www.fpg.unc.edu/ECO/,