Reliability of faculty evaluations on dental students' grades

“Objectives:” Students' scores and subsequent grades for dental courses should reflect their performance. However, the scores are dependent in part on the reliability of faculty evaluations and variation in faculty staff scorings which could unduly affect students' final grades. This study investigated the effect of staff members' variability on students' scores in assigned topics in fixed prosthodontics.

“Methods:” The study population comprised fifth-year dental students at faculty of Dentistry, King Abdulaziz University, Saudi Arabia. At the Division of Fixed Prosthodontics, a total of 72 computer-assigned interactive tutorials during 2003 and 2004 were selected as a method of continuing assessment. Each presentation was graded on the basis of the criteria described in the handout distributed to each evaluator. Three senior faculty members participated in the evaluation session. Four level of grading process, rating presentation as highly acceptable, acceptable, marginally acceptable, or unacceptable were used. Data were collected and analyzed.

“Results:” A one-way multivariate analysis of variance showed significant difference among the evaluators (p<.001). The results also indicated that staff member variability was responsible for 35 percent of the variance in students' final scores. The remainder of the differences resulted from differences in student performance. More than one half of the students' grades would have been either higher or lower if the scores assigned by one individual instructor had been excluded. There is also a significant variability in the scoring patterns between individual evaluators. Nevertheless, the measurement of Kappa is .507 indicating moderately accurate evaluations.

“Conclusion:” The students' grades were reliable and that individual differences among staff members adversely affect the students' final scores