This paper describes the current evaluation of the Physics and Astronomy New Faculty Workshop (NFW) as a case-study in evaluation of professional development workshops. We describe a The-ory of Action (ToA) for the workshop, and the evaluation methods and measures. The evaluation suggests that the ToA of the workshop is only partially fulfilled: workshop experiences are posi-tive, and participants gain knowledge of active learning, but participants have room for additional growth in skill, self-efficacy and social support in their use of active learning. We discuss the im-plications of these results for the NFW program and evaluation.
The process of setting and evaluating student learning objectives (SLOs) has become increasingly popular as an example where classroom assessment is intended to fulfill the dual purpose use of informing instruction and holding teachers accountable. A concern is that the high‐stakes purpose may lead to distortions in the inferences about students and teachers that SLOs can support. This concern is explored in the present study by contrasting student SLO scores in a large urban school district to performance on a common objective external criterion. This external criterion is used to evaluate the extent to which student growth scores appear to be inflated. Using 2 years of data, growth comparisons are also made at the teacher level for teachers who submit SLOs and have students that take the state‐administered large‐scale assessment. Although they do show similar relationships with demographic covariates and have the same degree of stability across years, the two different measures of growth are weakly correlated.
We report on our investigation of a retrospective pre-test to measure faculty attitude change towards the use of active learning after the Physics and Astronomy New Faculty Workshop (NFW). The purpose of the study is to explore alternative methods of evaluating the effectiveness of educational interventions aimed at attitude change. In the current study, we focus on faculty attitudes that would support change in teaching practice. Using traditional pre/post surveys, we find that only knowledge of and skill using active learning are substantively increased by the workshop. We administered a retrospective pre-test, where participants retrospectively rate their pre-workshop attitudes on the post-workshop survey. The rationale for this approach is that participants do not start with a common understanding of what "active learning" entails, and the workshop provides a normalizing experience so participants shift their understanding of active learning (termed response shift bias) as well as potentially generating gains in positive attitudes towards active learning. Using the retrospective pre-test, we see attitudinal gains for most items, but pre-test and retrospective pre-test results are poorly and inconsistently correlated. Preliminary interviews are suggestive of response shift bias, but only for some items. We can conclude that the validity of pre-workshop attitude ratings is questionable, but because of a conflation of response shift bias with other reporting biases (such as social desirability) and respondent characteristics, further research is needed to indicate whether retrospective pre-testing is an improved approach.
This study examines the relationship between English language proficiency outcomes and the use of a Computer-Assisted Language Learning (CALL) tool that is designed to strengthen the academic English and oral proficiency skills of young English Learners (ELs). We compare scores on a standardized English language proficiency assessment for 2,034 ELs from kindergarten through Grade 5 in the US who either used the CALL tool during the 2020-2021 academic school year (n=1,478) or did not (n=556). Descriptive analyses show larger scores for students who used the program than those who did not, and this was seen across student demographic subgroups. Statistical analyses reveal that this difference is significant even when accounting for student demographics and enrolled school and that greater program use is related to higher proficiency scores. The largest effects are seen for oral proficiency scores. The study raises implications for use of CALL tools in contexts where English is the language of education.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.