Accuracy of faculty evaluations of residents??? medical knowledge: comparison of subjective summative evaluations to in-training examination scores

Abstract

Matthew Stenerson, Amy Weaver, Jason Homme

Although faculty evaluations are an integral component in the assessment of residents’ medical knowledge, there is conflicting evidence about the accuracy of such evaluations. In contrast, the intraining examination (ITE) is an objective measure of residents’ medical knowledge that has proven to predict future performance. This study investigates the accuracy of faculty evaluations by comparing scores of medical knowledge on faculty completed evaluations to in-training examination scores. Subjects included categorical pediatrics and pediatric neurology residents at Mayo Clinic from 2006-2010. For each year of training the standard ITE score was compared to the corresponding composite faculty evaluation score for medical knowledge and patient care competencies using the Spearman’s rank correlation coefficient. Forty-two residents were analyzed for the first year of training and 35 for the second. For both years, there was only a mild correlation between ITE score and either the medical knowledge (r=0.26 and 0.19, respectively) or patient care competency (r=0.27 and 0.29, respectively). Faculty assessments of residents’ medical knowledge did not correlate well with a standard objective measure. Training programs should consider targeted faculty development to improve evaluation skills and supplementing existing evaluation methods with additional assessment tools.