IADR Abstract Archives
Annotation Accuracy and Calculus Detection on Radiographs Using Deep Learning
Objectives: Labeling of medical data for developing deep learning-based software is time consuming and cost intensive, and the more exact labels need to be, the more efforts will be required for labeling. This becomes even more challenging when the data needs to be labeled by multiple experts. We aimed to assess the impact of annotation accuracy on the performance of a deep learning object detection model. Our case focused on the detection of dental calculus on bitewing radiographs.
Methods: In total 6,840 bitewing radiographs had been labeled for dental calculus by 2 annotators using bounding boxes (BBs). To assess the impact of annotation accuracy, we simulated the behavior of three hypothetical experts, by dividing the dataset into thirds. For the first third of the dataset, we kept the original BBs to represent an accurate annotator. For the second and third third, we increased and decreased BB respectively, to simulate too small and too large labeling. In- and decreases were done to varying degrees, 25% and 50% relative to the original BB area. In a second experiment all BBs in the dataset were increased and decreased by 50%, respectively. For each experiment, an object detection model, YoloV5, was trained up to 300 epochs. Performance was evaluated using mean average precision with an intersection over union threshold of 50% (mAP50). A train/validation/test split of 80%/10%/10% was used.
Results: The model trained on the original dataset achieved a mAP50 of 0.77, showing the feasibility of detecting dental calculus. Simulating different annotators with a deviation of 25% and 50% yielded mAP50 of 0.70 and 0.14, respectively. Increasing and decreasing all BBs by 50% resulted in mAP50 of 0.82 and 0.69, respectively.
Conclusions: Consistently smaller or larger BBs for labeling did only limitedly affect model performance, while inconsistent BB sizes had significant negative impact. Calibration of annotators is highly important.
Methods: In total 6,840 bitewing radiographs had been labeled for dental calculus by 2 annotators using bounding boxes (BBs). To assess the impact of annotation accuracy, we simulated the behavior of three hypothetical experts, by dividing the dataset into thirds. For the first third of the dataset, we kept the original BBs to represent an accurate annotator. For the second and third third, we increased and decreased BB respectively, to simulate too small and too large labeling. In- and decreases were done to varying degrees, 25% and 50% relative to the original BB area. In a second experiment all BBs in the dataset were increased and decreased by 50%, respectively. For each experiment, an object detection model, YoloV5, was trained up to 300 epochs. Performance was evaluated using mean average precision with an intersection over union threshold of 50% (mAP50). A train/validation/test split of 80%/10%/10% was used.
Results: The model trained on the original dataset achieved a mAP50 of 0.77, showing the feasibility of detecting dental calculus. Simulating different annotators with a deviation of 25% and 50% yielded mAP50 of 0.70 and 0.14, respectively. Increasing and decreasing all BBs by 50% resulted in mAP50 of 0.82 and 0.69, respectively.
Conclusions: Consistently smaller or larger BBs for labeling did only limitedly affect model performance, while inconsistent BB sizes had significant negative impact. Calibration of annotators is highly important.