Effect of Faculty Development Programme on Multiple-Choice Item Writing Quality

Objectives: The use of multiple choice questions (MCQs) is a well-accepted method to evaluate the professional competencies in medical science education. Nevertheless, assessment items written by faculty without training are lacking in quality. Their improvement might endorse with the validity of the examination, better achievement of the students, and discrimination in the competency level of the borderline students.
The aim of this study is to determine the effectiveness of a faculty development programme on MCQ item writing. This programme is a part of a dental school wider quality assurance protocol, and included workshops, a detailed item construction and a blueprint guide, in addition to a personalised guidance to improve items, before and after the tests.
Methods: Several workshops about assessment principles and how to write high quality MCQ items were dictated by the Education Directorate during 2015-2017. After that, there was an implementation of a MCQ quality assurance protocol.
Values for mean score, reliability (Cronbach Alfa), item-discrimination index and item-difficulty index were registered for 87 different MCQ tests done before (27 tests in 2014) and after (27 tests in 2016, 33 tests in 2017) the quality assurance protocol was introduced, analyzing a total of 5265 items.
Results: Mean students score during 2014 was 4.5 (scale1-7), during 2016 was 4.3 and in 2017 was 4.4. Cronbach alpha reliability increased 25% between 2014 and 2016 but decreased slightly (4%) between 2016 and 2017. The same behaviour was observed for item discrimination index. Difficulty index decreased from 0.67 to 0.58 between 2014-2016, but then increased 0.06 points in 2017.
Conclusions: The implementation of a faculty development programme on MCQ item writing improves their quality. However, in this study the most significant improvement was in 2016, after the first faculty development programme was implemented.