Evaluating ChatGPT’s Performance for Classifying CAD/CAM Materials in Abstracts

Objectives: To evaluate the performance of Chat Generative Pre-trained Transformer (ChatGPT) for classifying CAD/CAM materials used in studies’ abstracts.
Methods: Abstracts containing CAD/CAM restorative materials published until July 2021 were collected from PubMed. Two independent evaluators manually classified the studies according to whether resin, ceramic, composite and polymer were used. We then instructed ChatGPT to classify the abstracts in the same way and evaluated ChatGPT’s performance, using metrics such as F1 score and accuracy relative to the independent evaluators' results.
Results: The independent evaluators classified the 526 abstracts as follows: 222 ceramic, 72 composite, 102 polymer, and 145 resin. By comparison, ChatGPT classified the materials used in the studies as follows: 369 ceramic, 273 composite, 166 polymer, and 297 resin. ChatGPT performed best at classifying abstracts that studied polymer materials (0.75 F1 score) and worst at classifying abstracts that studied composites (0.41 F1 score). We also analyzed ChatGPT’s performance to determine if a material was not used in the study. In comparison to the previous task, ChatGPT performed better in classifying a study that did not use polymer materials (0.92 F1 score) or composites (0.41 F1 score) and worse in classifying studies that did not use ceramics (0.51 F1 score).
Conclusions: ChatGPT seems to be able to assist with the classification of polymer-based CAD/CAM materials. However, its performance at classifying non-polymer-based CAD/CAM materials does not seem very reliable. Within the limitations of this study, ChatGPT seems to perform better at classifying whether a study does not use a material than using it.