The Performance Assessment for Leaders: Construct Validity and Reliability Evidence

Abstract

The Performance Assessment for Leaders (PAL) was developed by a team of nationally recognized experts in response to a Massachusetts requirement to determine and evaluate the leadership abilities of candidates seeking initial school principal licensure. This article describes and evaluates research conducted on all aspects of a 2014-2015 statewide field trial of PAL. Findings suggest that this assessment is a valid and reliable measure of individual candidate competence for granting initial school leader licensure, and is a positive, educative experience for candidates. It concludes with implications for use elsewhere.

The validity of a performance based assessment for aspiring school leaders

“…For CPA scholars, the choice of theory is the third main concern; theory influences the identification of the research problem, the manner in which the researcher thinks about the problem, the kinds of questions raised, and the lens through which data are analyzed. As questions about the validity of the test began to emerge, the author turned first to the standards and theoretical framework employed by the authors of the technical reports on the PAL pilot and field trial (Orr, Pecheone, Hollingworth, et al, 2017;Orr, Pecheone, Snyder, et al, 2017): namely, the Standards for Educational and Psychological Testing (2014). This resource was developed jointly by the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education, and updated periodically since 1974.…”

“…Each stage began with document analysis of the DESE literature, such as the handbook, field guides, scorer's manual, and standards alignment piece, supplemented by notes from state meetings and memorandums in order to best understand the authors' understanding and intentions. A consideration of the technical reports by the design team (Orr, Pecheone, Hollingworth, et al, 2017;Orr, Pecheone, Snyder, et al, 2017) often followed. The researcher then compared this information with material gathered from faculty and students plus PAL test scoring, examining everything in light of the literature and personal experience in school leadership.…”

“…The team addressed construct validity from four different angles, looking at the measurement quality of each indicator and examining the correlations among indicators, dimensions, and tasks for internal coherence and differentiation. For example, the researchers found strong internal consistency among the leadership dimensions for each task as shown in Table 1, which were balanced by distinctiveness (as none were higher than 0.77): Source: (Orr, Pecheone, Hollingworth, et al, 2017).…”

“…These seemingly contradictory conclusions-inflate or deflate-for score clustering are analyzed in the results section below. Source: (Orr, Pecheone et al, 2016) Using a stratified coefficient alpha, the team calculated the potential reliability of a total score (the average of all four task scores) and found an overall reliability coefficient of 0.844 for two scorers (Orr, Pecheone, Hollingworth, et al, 2017).…”

“…Racial disaggregation was unavailable due to insufficient numbers. Surveys of candidates and faculty concluded that the PAL offered ease of use and feasibility for principal preparation programs and flexibility for different types of settings (Orr, Pecheone, Snyder, et al, 2017).…”