loading page

Evaluation of the Quality and Reliability of ChatGPT-4's Responses on Allergen Immunotherapy Using Validated Tools
  • +17
  • Ivan Cherrez-Ojeda,
  • Torsten Zuberbier,
  • Gabriela Rodas-Valero,
  • Jorge Sanchez,
  • Michael Rudenko,
  • Stephanie Dramburg,
  • Pascal Demoly,
  • Davide Caimmi,
  • Maximiliano Gómez,
  • German Ramón,
  • Ghada Fouda E,
  • Kim Quimby R,
  • Herberto Chong-Neto,
  • Oscar Calderón,
  • Jose Ignacio Larco,
  • Olga Patricia Monge Ortega,
  • Marco Faytong-Haro,
  • Oliver Pfaar,
  • Jean Bousquet,
  • Karla Robles-Velasco
Ivan Cherrez-Ojeda
Charite - Universitatsmedizin Berlin

Corresponding Author:ivancherrez@gmail.com

Author Profile
Torsten Zuberbier
Charite - Universitatsmedizin Berlin
Author Profile
Gabriela Rodas-Valero
Universidad de Especialidades Espiritu Santo
Author Profile
Jorge Sanchez
Hospital “Alma Mater de Antioquia
Author Profile
Michael Rudenko
London Allergy and Immunology Center
Author Profile
Stephanie Dramburg
Charite - Universitatsmedizin Berlin
Author Profile
Pascal Demoly
University Hospital of Montpellier
Author Profile
Davide Caimmi
University Hospital of Montpellier
Author Profile
Maximiliano Gómez
Catholic University of Salta
Author Profile
German Ramón
Hospital Italiano Regional del Sur
Author Profile
Ghada Fouda E
Food and Drug Allergy Center
Author Profile
Kim Quimby R
Caribbean Institute for Health Research The University of the West Indies
Author Profile
Herberto Chong-Neto
Universidade Federal do Parana Hospital de Clinicas
Author Profile
Oscar Calderón
Clínica SANNA el Golf
Author Profile
Jose Ignacio Larco
Clinica San Felipe SA
Author Profile
Olga Patricia Monge Ortega
San Juan de Dios Hospital
Author Profile
Marco Faytong-Haro
Universidad Estatal de Milagro
Author Profile
Oliver Pfaar
University Hospital Marburg
Author Profile
Jean Bousquet
Charite - Universitatsmedizin Berlin
Author Profile
Karla Robles-Velasco
Universidad de Especialidades Espiritu Santo
Author Profile

Abstract

Background: Artificial Intelligence (AI) technologies could potentially change many aspects of clinical practice. While Allergen Immunotherapy (AIT) can change the course of allergic diseases providing relief of symptoms that extend for many years after treatment completion, it can also bring uncertainty to patients, who turn to readily available resources such as ChatGPT-4 to address these doubts. The aim of this study was to use validated tools to evaluate the information provided by ChatGPT-4 regarding AIT in terms of quality, reliability and readability. Methods: In accordance with AIT clinical guidelines, 24 questions were selected and introduced in ChatGPT-4. Answers were evaluated by a panel of allergists, using validated tools DISCERN, JAMA Benchmark and Flesch Reading Ease Score and Grade Level. Results: Questions were sorted into 6 categories. ChatGPT provided bad quality information according to DISCERN medians scores in the “Definition”, “Standardization and Efficacy”, and “Safety and Adverse Reactions” categories. It provided insufficient information according to JAMA Benchmark across all categories. Finally, ChatGPT-4 answers required a “college graduate” level of education to be understood as they were very difficult to read. Conclusions: ChatGPT-4 exhibits potential as a valuable complement to healthcare; however, it requires further refinement. The information it provides should be approached with caution regarding its quality, as significant details may be omitted or may not be fully comprehensible. Artificial intelligence models continue to evolve, and medical professionals should participate in this process, given that AI impacts various aspects of life, including health, to ensure the availability of optimal information.