Estimating reliability of school-level scores using multilevel and generalizability theory models

Jeon, Min-Jeong; Lee, Guemin; Hwang, Jeong Yeon; Kang, Sung Hoon

doi:10.1007/s12564-009-9014-3

Cited by 23 publications

(19 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Data for the 12 short-form items were entered in a three-level variance components model, with item responses nested within respondents within wards. 51 Variance estimates at each level were obtained using MLwiN v2.36 (MLwiN, Centre for Multilevel Modelling, Bristol, UK).…”

Section: After Francis Preliminary Analysesmentioning

confidence: 99%

An evaluation of a near real-time survey for improving patients’ experiences of the relational aspects of care: a mixed-methods evaluation

Graham

Käsbauer

Cooper

et al. 2018

Health Serv Deliv Res

View full text Add to dashboard Cite

Background The Francis Report (of 2013) provided many recommendations to improve compassionate care in NHS organisations, including more widespread use of real-time feedback (RTF) to collect patient experience data. This research directly addressed these recommendations and aimed to provide an evidence-based toolkit to support NHS quality improvements. Objectives To develop and validate a survey of compassionate care for use in near real time on elderly care wards and accident and emergency (A&E) departments. This research also evaluated the effectiveness of the RTF approach for improving relational aspects of care and provides suggestions for how the approach can be used by other hospitals to strengthen compassionate care. Design The research utilised a mixed-methods design, using quantitative, qualitative and participatory research approaches to collect patients’ experiences of relational care and the views of NHS staff in an effort to evaluate the processes and impacts of near real-time feedback (NRTF) data collection. Data sources included a NRTF patient experience survey, weekly volunteer diaries, staff interviews and surveys, workshops and meetings with case study sites. Setting The research was carried out across six case study sites across England, in wards that predominantly serve elderly patients and in A&E departments. Participants The 3928 participants in the patient experience survey were inpatients on elderly care wards, or persons who had sought medical care in A&E. Frontline staff, service leads, senior management and volunteers also took part in surveys (n = 274) and interviews (n = 82) designed to understand the staff perspectives and opinions of collecting patient experience data. Interventions A patient experience survey was implemented using a tablet computer-based methodology, facilitated by trained volunteers. Responses were used alongside feedback from staff to evaluate the use of a NRTF approach as a method for improving patient experiences of relational aspects of care. Main outcome measures The patient experience survey measured relational aspects of care. Another outcome measure was improvements to care as planned, implemented and reported by staff. Results A small but statistically significant improvement (p = 0.044) in relational aspects of care over the course of the study was noted overall. Staff implemented a variety of improvements to enhance communication with patients. Limitations Maintaining volunteer and staff engagement throughout the study was difficult. Few surveys were completed per ward or department each week. This made examining trends in patient experiences over time challenging. Conclusions Near real-time feedback offers an effective approach for monitoring and improving relational aspects of care. Future work Staff frequently expressed a view that volunteers’ interactions with patients while administering the survey were themselves beneficial to patients. Future research should examine the impact of volunteer interactions with patients on their experiences of relational aspects of care. Study registration The project is registered on the Clinical Research Network portfolio under the primary trial identification number 18449. Funding The National Institute for Health Research Health Services and Delivery Research programme.

show abstract

Section: After Francis Preliminary Analysesmentioning

confidence: 99%

An evaluation of a near real-time survey for improving patients’ experiences of the relational aspects of care: a mixed-methods evaluation

Graham

Käsbauer

Cooper

et al. 2018

Health Serv Deliv Res

View full text Add to dashboard Cite

show abstract

“…Since the individual-level reliability estimates, such as Cronbach's alpha, are not appropriate for assessing the reliability of aggregate-level variables (see Jeon et al 2008;LeBreton and Senter 2008;Lüdtke et al 2006), we used the intraclass correlations ICC(1) and ICC(2) to determine whether or not aggregated student-level ratings were reliable indicators of school-level constructs (Bliese 2000;Raudenbush and Bryk 2002). The ICC(1) and ICC (2) indices are based on a one-way analysis of variance (ANOVA) with random effects.…”

Section: School-level Measuresmentioning

confidence: 99%

Influence of motivation, self-beliefs, and instructional practices on science achievement of adolescents in Canada

2010

View full text Add to dashboard Cite

This study examined the effects of motivation to learn science, science self-beliefs, and science instructional practices on science achievement of 13,985 15-year-old students from 431 schools across Canada. Hierarchical linear modeling (HLM) analyses, while controlling for student-and school-level demographic characteristics, revealed the substantial predictive effects of motivation to learn science, science self-beliefs, and science instructional practices on science achievement of adolescents. Motivational beliefs-self-efficacy and self-concept-and enjoyment of science had substantial positive predictive effects on science achievement. In contrast, general interest in science had a negative predictive effect on science achievement in the context of other variables. Whereas science teaching using hands-on activities had a substantial positive predictive effect on science achievement, science teaching using student investigations had a substantial negative predictive effect in the context of other variables. The final HLM model indicated that only 8% of the variance in science achievement was between schools and 92% of the variance involved students within schools.

show abstract

“…By extension, we propose that specific aspects of observer bias should be investigated as detailed above. Here, multi‐level modelling techniques can account for the hierarchical nature of these data, within‐ and between‐assessor effects, lack of independence in the data and potential problems of auto‐correlation 18,19 . Most importantly, the rationale and justification for why certain attributes are targeted in selection, and matching these to an appropriate selection method, require greater exposition in the research literature.…”

Section: Resultsmentioning

confidence: 99%

Testing non‐cognitive attributes in selection centres: how to avoid being reliably wrong

Patterson¹,

Ferguson

2012

Medical Education

View full text Add to dashboard Cite

Estimating reliability of school-level scores using multilevel and generalizability theory models

Cited by 23 publications

References 25 publications

An evaluation of a near real-time survey for improving patients’ experiences of the relational aspects of care: a mixed-methods evaluation

An evaluation of a near real-time survey for improving patients’ experiences of the relational aspects of care: a mixed-methods evaluation

Influence of motivation, self-beliefs, and instructional practices on science achievement of adolescents in Canada

Testing non‐cognitive attributes in selection centres: how to avoid being reliably wrong

Contact Info

Product

Resources

About