How performance assessment for leaders (PAL) influences preparation program quality and effectiveness

Orr, Margaret Terry; Hollingworth, Liz

doi:10.1080/13632434.2018.1439464

Cited by 8 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several papers from the PAL design team offered a preliminary examination of validity evidence and were presented in the literature review (Orr & Hollingworth, 2018;Orr, Pecheone, Hollingworth, et al, 2017;Orr, Pecheone, Snyder, et al, 2017). The results presented here offer a critique of that literature and additional evidence that challenges the validity of the PAL.…”

Section: Resultsmentioning

confidence: 94%

“…However, each validity source called for a different mixture of evidence. For example, the consideration of test content validity began with the standards alignment document , followed by the state leadership standards and the research literature on school leadership; in contrast, the final stage (consequences of testing) began with the design team's analysis of programmatic effects (Orr & Hollingworth, 2018) followed by examination of internal evidence from faculty and students. Each stage of analysis involved extensive reading, reflection, and writing, which often prompted new insights on previous stages already written; a recursive process of consideration, reflection, writing, reexamination, and rewriting ensued.…”

Section: Discussionmentioning

confidence: 99%

“…Program quality and effectiveness. More recently, PAL designers surveyed program faculty (n = 12, representing about half the MA preparation programs) and students (n = 53) to assess the influence of the PAL on program quality and effectiveness (Orr & Hollingworth, 2018). Since they gathered responses during difficult adjustment years (the field trial and following implementation year), the authors assumed that results would be likely to over-represent those who were aggravated by the test (Orr & Hollingworth, 2018).…”

Section: Field Trial Evaluationmentioning

confidence: 99%

“…More recently, PAL designers surveyed program faculty (n = 12, representing about half the MA preparation programs) and students (n = 53) to assess the influence of the PAL on program quality and effectiveness (Orr & Hollingworth, 2018). Since they gathered responses during difficult adjustment years (the field trial and following implementation year), the authors assumed that results would be likely to over-represent those who were aggravated by the test (Orr & Hollingworth, 2018). Based upon largely favorable responses from participants, the authors concluded "PAL has the potential to serve as both a high-quality summative assessment tool for licensure as well as a learning opportunity and barometer for programs to judge the quality of their preparation, both in Massachusetts and in other states" (Orr & Hollingworth, 2018, p. 18).…”

Section: Field Trial Evaluationmentioning

confidence: 99%

“…In a follow-up study on the implementation year (2016-2017), PAL researchers explored the impact of the PAL on program effectiveness. They asked program faculty how effective their program was in preparing candidates to complete the requirements for each task and they asked candidates to rate how challenging they found the tasks (Orr & Hollingworth, 2018). See Table 4: Source: (Orr & Hollingworth, 2018) There are obvious validity challenges when the inventor of a product also conducts the follow-up impact study.…”

Section: Evidence For Validity and Consequences Of Testingmentioning

confidence: 99%

See 4 more Smart Citations

The validity of a performance based assessment for aspiring school leaders

Leonard

2018

EPAA

View full text Add to dashboard Cite

This paper introduces the new Massachusetts Performance Assessment for Leaders (PAL) and uses critical policy analysis to re-examine the validity evidence (using the 2014 Standards for Educational and Psychological Testing and a theory of multicultural validity) for the use and interpretation of the PAL in regards to emerging school leadership. Data sources include two years of PAL test documentation plus candidate surveys and interviews with program directors. The author’s role as a test user, faculty instructor, and certified test scorer afforded access to student work, student communications, scorer network training, and state department of education communications and meetings. The paper challenges the content validity, raises questions in regards to evidence based on response processes, internal structure, relation to other variables, consequences, and multicultural validity particularly when the PAL is used as a stand-alone, high-stakes licensure test and offers suggestions to improve the test as a formative assessment.

show abstract

Section: Resultsmentioning

confidence: 94%

Section: Discussionmentioning

confidence: 99%

Section: Field Trial Evaluationmentioning

confidence: 99%

Section: Field Trial Evaluationmentioning

confidence: 99%

Section: Evidence For Validity and Consequences Of Testingmentioning

confidence: 99%

See 3 more Smart Citations

The validity of a performance based assessment for aspiring school leaders

Leonard

2018

EPAA

View full text Add to dashboard Cite

show abstract

Programmatic Effects of the Massachusetts Performance Assessment for Leaders

Leonard

2019

Leadership and Policy in Schools

View full text Add to dashboard Cite

Performance assessment for school leaders: comparing field trial and implementation results

Orr

Hollingworth

Beaudin

2019

JEA

Self Cite

View full text Add to dashboard Cite

Purpose The purpose of this paper is to compare two years of results for one state’s performance-based assessments for principal licensure Performance Assessment for Leaders (PAL). This includes the field trial (2014–2015) and first year of statewide implementation (2015–2016) when passing score requirements and fees were added. Survey results on candidates’ career aspirations provide concurrent validation. Design/methodology/approach Two years of PAL submissions (n=569 candidates) were scored by trained, certified scorers. Task and total score results were compared by year, preparation pathway and gender. Online feedback survey results on career aspirations for (n=146 candidates) were compared by year. Findings The results show that PAL assessments measure independent dimensions of leadership, differentiate candidates on leadership knowledge and skills, and confirm PAL’s internal validity. Implementation year scores were higher than field trial scores, and preparation program candidates scored better than non-program candidates did. Candidate career aspirations were stronger in the implementation year than during the field trial. Research limitations/implications The study is limited to one state’s candidates, but findings are generalizable based on the wide range among candidates’ districts (demographically and economically). Practical implications The results are promising for the leadership preparation and assessment field, demonstrating the effectiveness of performance assessment for authentic evaluation of leadership candidates’ knowledge and skill and overall readiness for initial leadership work. Originality/value This is the first large scale performance assessment for aspiring leaders designed for state licensure decisions. It is being replicated in another state and shown promise for both formative and summative leadership assessment.

show abstract

How performance assessment for leaders (PAL) influences preparation program quality and effectiveness

Cited by 8 publications

References 20 publications

The validity of a performance based assessment for aspiring school leaders

The validity of a performance based assessment for aspiring school leaders

Programmatic Effects of the Massachusetts Performance Assessment for Leaders

Performance assessment for school leaders: comparing field trial and implementation results

Contact Info

Product

Resources

About