Relative and Absolute Standards in Assessing Medical Knowledge

Muijtjens, Arno; Hoogenboom, Ron; Verwijnen, G. M.; Vleuten, Cees P.M. van der

doi:10.1007/978-94-011-4886-3_200

Cited by 32 publications

(47 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, a factor that also should be taken into consideration when looking at the student scores is that 'tests and examinations drive student learning', a statement first coined by Muijtjens et al 31 Both student groups where primed as they were aware that they were being graded on their performance, unlike the pharmacy staff that were aware that they would be partaking in the mystery shopping scheme, but didn't know when the mystery shopper was going to enter the pharmacy. It has been found that testing students leads to an increase in the amount of study time and improvements in study strategies, thus student scores are expected to be higher compared to the pharmacy staff as they were more prepared.…”

Section: R a F T A J P E A C C E P T E D D R A F Tmentioning

confidence: 99%

Ability of Pharmacy Students, Pharmacists and Pharmacy Support Staff to Manage Childhood Fever via Simulation

Wigmore

Collins

Schneider

et al. 2018

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objective. Fever is a common childhood symptom of illness often mismanaged by caregivers.Community pharmacies can provide an influential role in the education and management of fever.This study aimed to ascertain how pharmacy students (novices) and pharmacy staff (experts) respond to a childhood fever scenario.Methods. Data were collected from 65 second year students and 51 fourth (final) year students in an over-the-counter fever scenario during assessment tasks. Data from pharmacy staff were collected via mystery shopping conducted over nine weeks between March-October 2015. All encounters were immediately scored by the trained simulated client, and immediate feedback was provided for pharmacy staff and fourth year students. Questioning scores (WHAT) and proportions of competent participants were collected in all groups. Statistical comparative analyses were made between fourth year students and pharmacy staff. Pharmacy staff scores were also tracked over time.Results. Second year students performed well when questioning, achieving a median WHAT score of 100%. Conversely, pharmacy staff scored 22%. A large proportion of the fourth year students and pharmacy staff achieved appropriate outcomes (92% and 65% respectively), however a smaller proportion of second years performed well (52%). The pharmacy staff achieved statistical improvements over time for median WHAT scores (p<0.001). Conclusions.Protocol compliant questioning appears to decline with experience. However, experienced counsellors are more likely to provide appropriate patient advice. Further improvements in outcomes can also occur when staff are provided feedback and coaching. Mystery shopping simulations can be used as a valuable educational tool.

show abstract

Section: R a F T A J P E A C C E P T E D D R A F Tmentioning

confidence: 99%

Ability of Pharmacy Students, Pharmacists and Pharmacy Support Staff to Manage Childhood Fever via Simulation

Wigmore

Collins

Schneider

et al. 2018

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

show abstract

“…For relative standards the pass mark is set to fail a defined proportion of students whatever their attainment in the assessment. The use of absolute methods is perceived to be fairer as it takes in to account differences in difficulty between papers and also helps to ensure maintenance of standards between cohorts (Lowry, 1993;Muijtjens et al, 1998). Such methods can be divided into test-centred (or rational) if they rely on expert judgements based on the assessment content or as examinee-centred (or empirical) if expert judgement is based on examinee performance on the test.…”

Section: Introductionmentioning

confidence: 98%

“…Previous authors have provided useful examples of standard setting in the medical school context (Morrison et al, 1996;Muijtjens et al, 1998;Verhoeven et al, 1999;Wilkinson et al, 2001;Downing et al, 2003). However the resource issue of the number of judges required to obtain a reliable standard for small-scale assessments has not been specifically addressed.…”

Section: Introductionmentioning

confidence: 98%

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Fowell

Fewtrell

2006

Adv in Health Sci Educ

View full text Add to dashboard Cite

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.

show abstract

“…Because of this problem, most teachers consider normreferenced standards as not acceptable. The second problem is that, regardless of the group's ability, a fixed percentage of examinees will always fail (Muijtjens et al 1998), which seems rather unfair and compromises the credibility of this method.…”

Section: Finding the Right Balancementioning

confidence: 99%

A standard setting method with the best performing students as point of reference: Practical and affordable

Cohen-Schotanus

Vleuten

2010

Medical Teacher

View full text Add to dashboard Cite

A standard setting method with the best performing students as point of reference Schotanus, Janke; van der Vleuten, Cees P. M. Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum. AbstractBackground: Teachers involved in test development usually prefer criterion-referenced standard setting methods using panels. Since expert panels are costly, standards are often set by a pre-fixed percentage of questions answered correctly or normreferenced methods aimed at ranking examinees. Aim: To discuss the (dis)advantages of commonly used criterion and norm-referenced methods and present a new compromise method: standards based on a fixed cut-off score using the best scoring students as reference point. Methods: Historical data from 54 Maastricht (norm-referenced) and 52 Groningen (criterion-referenced) tests were used to demonstrate huge discrepancies and variability in cut-off scores and failure rates. Subsequently, the compromise model -known as Cohen's method -was applied to the Groningen tests.Results: The Maastricht norm-referenced method led to a large variation in required cut-off scores (15-46%), but a stable failure rate (about 17%). The Groningen method with a conventional, pre-fixed standard of 60% led to a large variation in failure rates (17-97%). The compromise method reduced variation in required cut-off scores as well as failure rates. Conclusion: Both the criterion and norm-referenced standards, used in practice, have disadvantages. The proposed compromise model reduces the disadvantages of both methods and is considered more acceptable. Last but not least, compared to standard setting methods using panels, this method is affordable.

show abstract

Relative and Absolute Standards in Assessing Medical Knowledge

Cited by 32 publications

References 6 publications

Ability of Pharmacy Students, Pharmacists and Pharmacy Support Staff to Manage Childhood Fever via Simulation

Ability of Pharmacy Students, Pharmacists and Pharmacy Support Staff to Manage Childhood Fever via Simulation

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

A standard setting method with the best performing students as point of reference: Practical and affordable

Contact Info

Product

Resources

About