Karen Ebenezer

and 4 more

Purpose: A standardized set of clinical practice guidelines from the Musculoskeletal Tumor Society (MSTS) was released in 2018 to aid non-specialist physicians in diagnosing bone and soft-tissue lesions. Artificial intelligence (AI) models like ChatGPT are increasingly prevalent. Our objectives were to determine how closely ChatGPT recommendations align with MSTS guidelines for imaging bone and soft tissue lesions. Methods: We developed questions to assess ChatGPT’s alignment with MSTS guidelines. Answers from ChatGPT were double-blinded and evaluated for concordance. Answers were scored using four categories: accuracy, overconclusiveness, supplementary information, and incompleteness. Chi-square test was used with a statistically significant p-value of less than 0.05. Results: A total of 14 questions were generated from the 12 guidelines. Results showed alignment between the accuracy of ChatGPT’s responses and the guidelines. Of the 14 questions posed to ChatGPT, 10 were deemed accurate. However, responses to 13 questions were deemed overconclusive or contained supplementary information (p<0.05). Additionally, 9 responses were deemed incomplete when compared to guidelines (p<0.05). Conclusions: Imaging methods recommended by MSTS to front-line practitioners are generally aligned with AI-based modeling recommendations. However, ChatGPT is not yet sufficient as a standalone modality, often providing supplementary or incomplete information. The value of AI-based predictions in healthcare are an evolving modality.