Both Patients and Plastic Surgeons Prefer Artificial Intelligence–Generated Microsurgical Information

Abstract

Background With the growing relevance of artificial intelligence (AI)-based patient-facing information, microsurgical-specific online information provided by professional organizations was compared with that of ChatGPT (Chat Generative Pre-Trained Transformer) and assessed for accuracy, comprehensiveness, clarity, and readability.

Methods Six plastic and reconstructive surgeons blindly assessed responses to 10 microsurgery-related medical questions written either by the American Society of Reconstructive Microsurgery (ASRM) or ChatGPT based on accuracy, comprehensiveness, and clarity. Surgeons were asked to choose which source provided the overall highest-quality microsurgical patient-facing information. Additionally, 30 individuals with no medical background (ages: 18–81, μ = 49.8) were asked to determine a preference when blindly comparing materials. Readability scores were calculated, and all numerical scores were analyzed using the following six reliability formulas: Flesch–Kincaid Grade Level, Flesch–Kincaid Readability Ease, Gunning Fog Index, Simple Measure of Gobbledygook Index, Coleman–Liau Index, Linsear Write Formula, and Automated Readability Index. Statistical analysis of microsurgical-specific online sources was conducted utilizing paired t-tests.

Results Statistically significant differences in comprehensiveness and clarity were seen in favor of ChatGPT. Surgeons, 70.7% of the time, blindly choose ChatGPT as the source that overall provided the highest-quality microsurgical patient-facing information. Nonmedical individuals 55.9% of the time selected AI-generated microsurgical materials as well. Neither ChatGPT nor ASRM-generated materials were found to contain inaccuracies. Readability scores for both ChatGPT and ASRM materials were found to exceed recommended levels for patient proficiency across six readability formulas, with AI-based material scored as more complex.

Conclusion AI-generated patient-facing materials were preferred by surgeons in terms of comprehensiveness and clarity when blindly compared with online material provided by ASRM. Studied AI-generated material was not found to contain inaccuracies. Additionally, surgeons and nonmedical individuals consistently indicated an overall preference for AI-generated material. A readability analysis suggested that both materials sourced from ChatGPT and ASRM surpassed recommended reading levels across six readability scores.

Keywords

artificial intelligence - accuracy - comprehensiveness - clarity - readability - quality - online resources - American Society of Reproductive Medicine

Author Contributions

C.E.B., A.Z.F., and D.C.W. lead conception and design, collection and assembly of data, data analysis and interpretation, and manuscript writing for this project. All authors assisted with administrative support, provision of study materials or patients, and final review and approval of the manuscript.

^* These authors contributed equally to this work.

Publication History

Received: 09 September 2023

Accepted: 15 February 2024

Accepted Manuscript online:
21 February 2024

Article published online:
26 March 2024

• References

• 1 Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 2023; 6: 1169595
• 2 Nguyen J, Pepping CA. The application of ChatGPT in healthcare progress notes: a commentary from a clinical and research perspective. Clin Transl Med 2023; 13 (07) e1324
• 3 Issom DZ, Hardy-Dessources MD, Romana M. et al. Toward a Conversational Agent to Support the Self-Management of Adults and Young Adults With Sickle Cell Disease: Usability and Usefulness Study. Front Digit Health 2021; 3: 600333
• 4 Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel) 2023; 11 (06) 887
• 5 Patel SB, Lam K. ChatGPT: the future of discharge summaries?. Lancet Digit Health 2023; 5 (03) e107-e108
• 6 Revolutionizing healthcare: the top 14 uses of ChatGPT in medicine and wellness. Accessed July 17, 2023 at: https://www.forbes.com/sites/bernardmarr/2023/03/02/revolutionizing-healthcare-the-top-14-uses-of-chatgpt-in-medicine-and-wellness/?sh=43160a546e54
• 7 Daraz L, Morrow AS, Ponce OJ. et al. Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet. J Gen Intern Med 2019; 34 (09) 1884-1891
• 8 Shahsavar Y, Choudhury A. User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study. JMIR Human Factors 2023; 10: e47564
• 9 97+ ChatGPT Statistics & User Numbers in July 2023 (New Data). Nerdy Nav. December 13, 2022. Accessed July 17, 2023 at: https://nerdynav.com/chatgpt-statistics/
• 10 Javaid M, Haleem A, Singh RP. ChatGPT for healthcare services: an emerging stage for an innovative perspective. BenchCouncil Trans Benchmarks Stand Eval 2023; 3 (01) 100105
• 11 Vargas CR, Chuang DJ, Lee BT. Assessment of patient health literacy: a national survey of plastic surgeons. Plast Reconstr Surg 2014; 134 (06) 1405-1414
• 12 Chen DH, Johnson AR, Ayyala H, Lee ES, Lee BT, Tran BNN. A multimetric health literacy analysis of autologous versus implant-based breast reconstruction. Ann Plast Surg 2020; 85 (S1, Suppl 1) S102-S108
• 13 Berry CE, Fazilat AZ, Churukian AA. et al. Quality assessment of online resources for gender-affirming surgery. Plast Reconstr Surg Glob Open 2023; 11 (10) e5306
• 14 Padula WV, Armstrong DG, Goldman DP. Complexity bias in the prevention of iatrogenic injury: why specific harms may inhibit performance. Mayo Clin Proc 2022; 97 (02) 221-224
• 15 Shah YB, Ghosh A, Hochberg AR. et al. Comparison of ChatGPT and traditional patient education materials for men's health. Urol Pract 2024; 11 (01) 87-94
• 16 Abi-Rafeh J, Xu HH, Kazan R, Tevlin R, Furnas H. Large language models and artificial intelligence: a primer for plastic surgeons on the demonstrated & potential applications, promises, and limitations of ChatGPT. Aesthet Surg J 2024; 44 (03) 329-343
• 17 Johnson D, Goodman R, Patrinely J. et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res Sq 2023; rs.3.rs-2566942
• 18 Li H, Moon JT, Iyer D. et al. Decoding radiology reports: potential application of OpenAI ChatGPT to enhance patient understanding of diagnostic reports. Clin Imaging 2023; 101: 137-141
• 19 Moosvi N, Kovarik C. Readability, accuracy, and appropriateness of ChatGPT 4.0 responses for use in patient education materials for Condyloma acuminatum . Clin Dermatol 2024; 42 (01) 87-88
• 20 Taloni A, Borselli M, Scarsi V. et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep 2023; 13 (01) 18562
• 21 Homolak J. Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma. Croat Med J 2023; 64 (01) 1-3