Exploring the Relationship Between Scoring Accuracy and Predictive Validity in Risk Assessment Using the Service Planning Instrument (SPIn)

Abstract

Accurate scoring is an essential component of a risk assessment's reliability, which in turn contributes to its predictive validity. This study tested a new method for evaluating the accuracy of completed risk assessments, described as intra-rater consistency. Pairs of cross-consistency items were identified in the Service Planning Instrument (SPIn), which were used to flag errors in completed assessments. Participants were 31,460 adults (20.5% female; 20.7% Indigenous) on community supervision in Alberta. The base rate of error was low; however, Indigenous participants had significantly higher rates of error. Contrary to my hypothesis, in the full sample assessments with errors had slightly higher predictive accuracy than assessments without errors. This pattern was repeated among non-Indigenous participants, but no significant differences in predictive accuracy based on errors were found among Indigenous participants. Differences in predictive accuracy based on sex and Indigenous status were observed in assessments with errors, but not assessments without errors. SCORING ACCURACY AND PREDICTIVE VALIDITY IN RISK ASSESSMENT iii AcknowledgementsFirst and foremost, I would like to express my gratitude for my phenomenal supervisor, Dr. Shelley Brown. It is not often that you come across someone who has such a clear, measurable impact on your development. I have felt myself grow, as both a researcher and a person, under your mentorship. Your unique ability to be caring and compassionate, while also encouraging your students to push themselves to produce the best work possible, is truly incredible. I feel unbelievably privileged to have been mentored by such a brilliant person through both my undergraduate and master's theses. Thank you for seeing my potential, even when I couldn't see it myself. I would also like to thank my committee members, Dr. Ralph Serin and Dr. Craig Leth-Steensen. Your insight and feedback were absolutely essential to this research. I would also like to thank my external examiner, Dr. Steven Prus, and chair, Dr. Kevin Nunes for agreeing to participate in my graduate committee. I sincerely appreciate the time and effort you invested in my defence. A very special thank you to Dr. David Robinson of Orbis Partners, who provided the original inspiration for this study. I am so thankful to you for sharing your ideas with me and for investing so much time and energy into providing your insight and feedback throughout this process. This study would most certainly not have been possible without your help. I would also like to extend my thanks to Christie Nicholson and the Alberta Solicitor General for supporting this research and providing the dataset. Needless to say, this study would not have been possible without you. I would also like to express my gratitude for the incredibly supportive Gender and Crime lab, and specifically to my friend Colleen. Your friendship, support, and encouragement made SCORING ACCURACY AND PREDICTIVE VALIDITY IN RISK ASSESSMENT iv what could have been a very difficult and...