Assessing radiographic grading system reproducibility when no gold standard available

Method: 200 standardized digital bitewing radiographs pairs were sampled from those taken during a caries clinical trial in adolescents.  All radiographs were scored using the Pitts grading system, which allowed both enamel and dentine lesions to be scored at two depths and included a method for scoring caries when overlaps were present.  Three experienced dentists (clinical trial radiograph reviewer, clinical trial examiner, paediatric dentist) were trained and calibrated in the grading system prior to its use.  Scoring was undertaken in a dedicated room with controlled lighting, a calibrated monitor and custom-built database.  Sensitivity, specificity and the area (A_z) under the ROC curve were calculated at D₁ (enamel) and D₃ (dentine) thresholds.  Consensus grading by all examiners was used as a gold standard.

Result: 4800 proximal surfaces (4d to 7m) were assessed.  Prevalence of enamel and dentine lesions were 12.6% and 2.3%, respectively, < 1% had fillings and 3.6% had unreadable overlap.  Kappa values between each examiner and the consensus were 0.84, 0.85 and 0.83.  At the D₁ threshold, sensitivity values were 0.75, 0.76 and 0.89; specificity values were 0.99, 0.99 and 0.96 and A_z values were 0.87, 0.88 and 0.93.  At the D₃ threshold, sensitivity values were 0.70, 0.82 and 0.75; specificity values were 1.00, 1.00 and 1.00; and A_z values were 0.85, 0.90 and 0.88.

Conclusion: The reproducibility of radiographic scoring for approximal caries using the Pitts grading system was excellent at both the D1 and D3 thresholds. Random effects modelling approach which utilizes the ordinal scale of the grading system and can other covariates will also be discussed.