Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

Abstract

Margaret MacDougall, Gregory E Stone

The veracity of Objective Standard Setting (OSS) as a modern approach to criterion-referenced standard setting has been reported for healthcare student assessment in the USA, while in other countries, OSS remains unrecognized. OSS upholds the foundational principle for itemized tests that judges should base their decisions on test item content. Moreover, it presents judges with a conceptually transparent decision procedure. This contrasts with the predictions concerning a hypothetical borderline candidate which typify Angoff procedures. Furthermore, the iterative process involved in the Angoff standard setting task incurs financial and administrative burdens, thus creating the potential to cut corners through recruiting fewer judges. The underlying objective of homogenizing the test standard undermines its validity, while circumventing reputable standard setting principles. While the Rasch model offers an objective approach to predicting successful outcomes, combining Rasch and Angoff procedures does not resolve the validity problem for Angoff-based pass marks. This commentary highlights the virtues of OSS relative to the modified Angoff approach in the standard setting of itemized tests. It also identifies gaps in the research literature that should be addressed to strengthen the case for using OSS on an international scale for high-stakes assessments within healthcare disciplines as a testing ground for other disciplines.