?I?m pickin' up good regressions': the governance of generalisability analyses

Abstract

We make the following recommendations. Ensure that all relevant factors are sampled, and that the sampling meets the theory's assumption that the conditions represent a random and representative sample of the factor's 'universe'. Research evaluations will require large samples of each factor if they are to generalise adequately. Where feasible, conduct 2 separate studies (pilot and evaluation, or Generalisability and Decision studies). For unbalanced data, use either urgenova, or 1 of the procedures minimum norm quadratic unbiased estimator, (minque), maximum likelihood (ml) or restricted maximum likelihood (reml) in spss or sas if the data are too complex. State which mathematical procedure was used and the degrees of freedom (d.f.) of the effect estimates. If the procedure does not report d.f., re-analyse with type III sum of squares anova (anova ss III) and report these d.f. Describe and justify the regression model used. Present the raw variance components. Describe the effects that they represent in plain, non-statistical language. If standard error of measurement (SEM) or Reliability coefficients are presented, give the equations used to calculate them. Make sure that the method of reporting reliability (precision or discrimination) is appropriate to the purpose of the assessment. This will usually demand a precision indicator such as SEM. Consider a graphical presentation to combine precision and discrimination.

Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre

“…Reliability was explored using generalizability theory 12 , in line with published expert consensus 13 . Variance components were calculated using MINQUE in SPSS  .…”

Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

“…We selected the MINQUE method because of its superior handling of unbalanced data. 122 Our regression model assumes that each factor contains a random sample from an infinite universe and estimates the factors in Table 4.…”

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

“…The MINQUE method was chosen because it requires no distributional assumptions and is recommended for unbalanced designs (Baltagi et al 2002); ANOVA III to estimate the degrees of freedom (Crossley et al 2007); and REML to confirm the MINQUE results (Baltagi et al 2002). Since it is possible that some variance components may have negative estimates by MINQUE, the REML method was also used to determine if the competency items having negative variance components were redundant.…”

“…Reducing rater variation and other forms of error will potentially increase the ability of assessments to discriminate between assessment performances (Wherry and Bartlett 1982) and also improve the integrity of the assessment so as to identify true trainee competency achievement (Sadler 2009). Methods exist to help identify such potential sources of variation (Wherry and Bartlett 1982;Cronbach et al 1972;Saal et al 1980;Crossley et al 2007). …”