Assessment Outcomes Can Be Skewed by Marking Schemes and Examiners

Objectives: The aim of this project was to explore the impact of a marking scheme and examiner assignment of scores on summative assessment in a UK dental school using a mixed methods approach.
Methods: As educational assessment incorporates both score scales and professional judgement a mixed methods approach was used to collect both quantitative and qualitative data. The un-transformed and linearly transformed scores were compared at each stage of the score aggregation to determine any impact on the score and grade assignment. Think-aloud interviews of examiners were used to gain insight into how examiners set about marking essays and assigning scores.
Results: We found that there was a significant (paired t-test, p<0.01) skewing effect resulting in misclassification of candidate achievement by 1 or 2 grade categories. We also found that the practice of capping scores of failed candidates further skewed the outcomes resulting in rank changes. Think-aloud interviews revealed differing interpretations of assessment corresponding to individual examiner practices which were also contributing to skewing of scores. Common to all examiners was the tendency primarily to rank essays (norm-referencing) rather than to adhere primarily to criterion-referencing. Whether examiners mark holistically or analytically, essays presented a problem to inexperienced examiners who needed more guidance and were reluctant to make negative judgements.
Conclusions: The researchers argue the following: 1. If score aggregation is to be employed in assessment, the mathematics underpinning the calculation need to be modified to minimise skewing of scores 2. Or, aggregation needs to be avoided across independent components of an assessment and replaced by assessment design that requires examiner decisions about quality based on criteria rather than percentage scores.