Classroom Quality in the Age of Accountability: Using a Comprehensive Multidimensional Rasch Approach to Investigate the Validity of the Early Childhood Environment Rating Scale-Revised - Foster, Brandon. (2017)
Abstract: The Early Childhood Environment Rating Scale-Revised (ECERS-R) is the most widely used measure of classroom quality, and has been implemented into numerous states' Quality Rating and Improvement Systems (QRISs). However, the ECERS-R was never designed for implementation into these systems or to be used for the purposes of accountability. As such, its validity for determining benchmarks on key indicators of classroom quality and suitability for identifying low or high performing preschools has yet to be established. Findings from a handful of validity studies which utilized classical test theory (CTT) have highlighted information pertaining to the factor structure of the measure. However, the CTT approach provides very limited diagnostic information about the functioning of the ECERS-R (e.g., Fan, 1998; Kieffer, 1998; Macdonald & Paunonen, 2002; Traub, 1997; Schumacker & Smith, 2007), and studies investigating the psychometric properties of the ECERS-R using more rigorous methods from item-response theory have hardly emerged (Gordon et al., 2013; 2015). This dissertation provides the first comprehensive account of the psychometric properties of the ECERS-R, utilizing the ECLS-B dataset, a nationally representative sample of early childhood environments. In particular, three proposed dimensional specifications of the measure that have emerged from the literature were examined. Psychometric analyses were carried out using Andrich Rating Scale Models and Multidimensional Random Coefficient Multinomial Logit (MRCML) models. To organize this information, this dissertation adopted Wolfe and Smith's (2007) Rasch validity framework, which ensured that analyses provided psychometric information about each component of Messick's seminal (1989) construct validity framework. Results illuminate the psychometric trade-offs that occur when different specifications of the ECERS-R are utilized to measure classroom quality. Additional information, which builds off the work of Gordon et al. (2013; 2015b), is provided to guide policymakers in the use of total scores from the measure in policy applications. Further, a promising new dimensional structure of the ECERS-R is posited and confirmed. Results from these analyses suggested that the ECERS-R functioned as a risk assessment, capable of precisely measuring classrooms with low levels of classroom quality, but functioned poorly for classrooms with average or above average levels of classroom quality. These results raise serious concerns about the use of the ECERS-R in most states' early-childhood assessment systems, and call into question some basic assumptions for how this measure is assumed to have performed in the prior research studies.; Thesis (Ph.D.)--Tufts University, 2017.; Submitted to the Dept. of Child Development.; Advisor: Christine McWayne.; Committee: Tama Leventhal, Shelagh Peoples, and Eric Dearing.; Keywords: Early childhood education, Quantitative psychology, and Public policy.