Modified Angoff vs Ebel for Standard-Setting: Judge Perspectives

Objectives: Criterion-referenced standard-setting is used to determine passing marks in high-stakes examinations; however, it is not widely used in dental schools in Malaysia. This study aimed to explore the experience and thought process of Malaysian dental faculty when applying Modified Angoff (MA) and Ebel techniques in the final-year multidisciplinary exam.
Methods: A convergent mixed-methods design collected data through quantitative feedback questionnaires and qualitative focus group discussions during a three-day standard setting workshop at the National University of Malaysia. Twenty Single Best Answer (SBA) items and ten Short Answer Questions (SAQs) were analysed. Questionnaires evaluated judges' understanding, confidence, and preference of the defensibility of the standard-setting methods. Focus group explored judges' experiences and thought processes during standard-setting and inductive thematic analysis was used to interpret the result.
Results: Twelve participants were recruited as judges in the standard-setting study. The mean and standard deviation (SD) of experience in teaching dentistry subjects was 16.3 years (8.6). A range of specialisms were represented. No differences were observed in feedback evaluations between the MA and Ebel methods within each round of ratings. Judges reported improved confidence levels from Round 1 to 3 in both MA and Ebel methods. Reasons for the preferences for each method varied among judges. The themes identified were: confidence in standard setting: continuous vs. category scoring, the role of diverse expertise in decision-making, aligning with course learning outcomes (CLOs), barriers to effective standard setting, and training as a catalyst for effective standard setting.
Conclusions: The study identified the strengths and challenges of the MA and Ebel methods in standard-setting for dental examinations. Judges showed differing thought processes but increased confidence and understanding through iterative rounds and group discussions. Comprehensive training, familiarity with CLOs, and diverse judge panels are critical for minimising biases and ensuring fair pass scores.