Can Chatbots Mimic Humans to Generate Systematic Review Abstracts?

Objectives: Different chatbots are increasingly being used to assist with academic writing. The purpose of this study was to determine if ChatGPT can generate abstracts for systematic reviews (SRs) in implant-fixed prosthodontics, one of the most commonly published topics in prosthodontics, which cannot be differentiated by humans.
Methods: Six SRs on implant prosthodontics published between January 2019 and October 2024 were randomly selected, with 2 from each of the 3 prosthodontic journals (genuine group). ChatGPT-4o was asked to generate 6 systematic review abstracts (experimental group) in implant-fixed prosthodontics using paraphrased titles of the genuine abstracts following the author guidelines of the respective journals. Participants who were attending international prosthodontic conferences were invited to judge, “Do you think this abstract is AI-generated?”. Participants were not informed about the number of AI-generated or genuine abstracts that were evaluated. Data were tabulated and analyzed using a chi-square test with the level of significance set at 0.05.
Results: Thirty participants completed this study. A significantly higher percentage of genuine abstracts (50%) were correctly identified by participants than the AI abstracts (32.2%) (P=0.02). Out of the six abstracts, only 36.7% (n=11) of participants correctly identified three or more AI-generated/genuine abstracts.
Conclusions: ChatGPT has demonstrated the capability to generate systematic review abstracts indistinguishable from those written by humans. The importance of employing chatbots responsibly and ethically should not be underestimated.