IADR Abstract Archives
Evaluation of Different Predictive Modeling Strategies for Tooth Loss
Objectives: Prediction models learn patterns from available data (training) and are then validated on new data (testing). Prediction modelling is increasingly common in dental research. We aimed to evaluate how different model development and model validation steps impact on the predictive performance of tooth loss prediction models in periodontitis patients.
Methods: Two independent cohorts (627 patients; 11651 teeth) had been followed over mean (SD) 18.2 (5.6) years (Kiel cohort) and 6.6 (2.9) years (Greifswald cohort). Tooth loss and nine patient- or tooth-level predictors had been recorded. The impact of different model development and validation steps were evaluated: (1) Model complexity (logistic regression/logR, recursive partitioning/RPA; random forest/RFO, and extreme gradient boosting/XGB), (2) sample size (full dataset or 10%/25%/75% cases dropped at random) (3) prediction periods (max. 10/15/20 years or uncensored), (5) validation schemes (internal, or external by centers/time periods).
Results: Tooth loss was generally a rare event (880 teeth were lost). All models showed limited sensitivity, but high specificity. More complex models (RFO/XGB) had no consistent advantages over simpler ones (logR/RPA). Internal (in-sample) validation over-estimated the predictive power (the area-under-the-curve AUC was up to 0.89), while external (out-of-sample) validation found lower AUCs (ranging between 0.51 and 0.82). Reducing the sample size decreased the predictive power, in particular for more complex models. Censoring the prediction period had only limited impact. Internal validation yielded higher accuracy than external (cross-center) validation training (AUC dropped to 0.51 in the worst case). When training the model in one time period and testing it in another, model outcomes were similar to the base-case, indicating temporal validation being a valid option.
Conclusions: In conclusion, none of the developed models would be useful in a clinical setting, despite high “apparent” accuracy. Various modeling steps had relevant impact on model performance. During modelling, rigorous development and external validation should be applied and reported accordingly.
Methods: Two independent cohorts (627 patients; 11651 teeth) had been followed over mean (SD) 18.2 (5.6) years (Kiel cohort) and 6.6 (2.9) years (Greifswald cohort). Tooth loss and nine patient- or tooth-level predictors had been recorded. The impact of different model development and validation steps were evaluated: (1) Model complexity (logistic regression/logR, recursive partitioning/RPA; random forest/RFO, and extreme gradient boosting/XGB), (2) sample size (full dataset or 10%/25%/75% cases dropped at random) (3) prediction periods (max. 10/15/20 years or uncensored), (5) validation schemes (internal, or external by centers/time periods).
Results: Tooth loss was generally a rare event (880 teeth were lost). All models showed limited sensitivity, but high specificity. More complex models (RFO/XGB) had no consistent advantages over simpler ones (logR/RPA). Internal (in-sample) validation over-estimated the predictive power (the area-under-the-curve AUC was up to 0.89), while external (out-of-sample) validation found lower AUCs (ranging between 0.51 and 0.82). Reducing the sample size decreased the predictive power, in particular for more complex models. Censoring the prediction period had only limited impact. Internal validation yielded higher accuracy than external (cross-center) validation training (AUC dropped to 0.51 in the worst case). When training the model in one time period and testing it in another, model outcomes were similar to the base-case, indicating temporal validation being a valid option.
Conclusions: In conclusion, none of the developed models would be useful in a clinical setting, despite high “apparent” accuracy. Various modeling steps had relevant impact on model performance. During modelling, rigorous development and external validation should be applied and reported accordingly.