Assessing the Quality and Readability of Online Patient Information: ENT UK Patient Information e-Leaflets versus Responses by a Generative Artificial Intelligence

Abstract

Background The evolution of artificial intelligence has introduced new ways to disseminate health information, including natural language processing models like ChatGPT. However, the quality and readability of such digitally generated information remains understudied. This study is the first to compare the quality and readability of digitally generated health information against leaflets produced by professionals.

Methodology Patient information leaflets from five ENT UK leaflets and their corresponding ChatGPT responses were extracted from the Internet. Assessors with various degrees of medical knowledge evaluated the content using the Ensuring Quality Information for Patients (EQIP) tool and readability tools including the Flesch-Kincaid Grade Level (FKGL). Statistical analysis was performed to identify differences between leaflets, assessors, and sources of information.

Results ENT UK leaflets were of moderate quality, scoring a median EQIP of 23. Statistically significant differences in overall EQIP score were identified between ENT UK leaflets, but ChatGPT responses were of uniform quality. Nonspecialist doctors rated the highest EQIP scores, while medical students scored the lowest. The mean readability of ENT UK leaflets was higher than ChatGPT responses. The information metrics of ENT UK leaflets were moderate and varied between topics. Equivalent ChatGPT information provided comparable content quality, but with reduced readability.

Conclusion ChatGPT patient information and professionally produced leaflets had comparable content, but large language model content required a higher reading age. With the increasing use of online health resources, this study highlights the need for a balanced approach that considers both the quality and readability of patient education materials.

Keywords

ChatGPT - patient information leaflets - rhinology leaflets - facial plastic surgery leaflets - patient information

Authors' Contributions

Study conception and design: E.S., T.K.K., K.S.F., J.S-B.

Acquisition of data: E.S., T.K.K., K.S.F., J.S-B., M.J., S.K., N.E-L., A.D.S., P.A.

Analysis and interpretation of data: K.S.F., T.K.K., E.S.

Drafting of manuscript: T.K.K., K.S.F., E.S., J.S-B., M.J., S.K., N.E-L., A.D.S., P.A.

Critical revision: E.S., K.S.F., T.K.K., J.S-B., M.J., S.K., N.E-L., A.D.S., P.A.

Publikationsverlauf

Accepted Manuscript online:
11. September 2024

Artikel online veröffentlicht:
15. Oktober 2024

• References

• 1 Goldie JGS. Connectivism: a knowledge learning theory for the digital age?. Med Teach 2016; 38 (10) 1064-1069
• 2 Hoffman DL, Novak TP, Venkatesh A. Has the Internet become indispensable?. Commun ACM 2004; 47 (07) 37-42
• 3 Dobransky K, Hargittai E. Inquiring minds acquiring wellness: uses of online and offline sources for health information. Health Commun 2012; 27 (04) 331-343
• 4 Siddique S, Chow JCL. Machine learning in healthcare communication. 2021; 1: 220-239
• 5 Tang PC, Smith MD. Democratization of health care. JAMA 2016; 316 (16) 1663-1664
• 6 Fan KS, Ghani SA, Machairas N. et al. COVID-19 prevention and treatment information on the internet: a systematic analysis and quality assessment. BMJ Open 2020; 10 (09) e040487
• 7 Shamil E, North AS, Fan KS, D'Souza H, Kaladjiska M, D'Souza A. The quality of online information on cosmetic injectable fillers: current status. Facial Plast Surg 2022; 38 (02) 124-130
• 8 Shamil E, Scenza GD, Ghani SA. et al. A quality assessment of online patient information regarding rhinoplasty. Facial Plast Surg 2022; 38 (05) 530-538
• 9 Kwan LY, Yip HCA, Tan S, Fan KS. A quality assessment of online patient information regarding tonsillitis using the EQIP tool. Int J Pediatr Otorhinolaryngol 2022; 159: 111224
• 10 Tan DJY, Ko TK, Fan KS. The readability and quality of web-based patient information on nasopharyngeal carcinoma: quantitative content analysis. JMIR Form Res 2023; 7 (01) e47762
• 11 Ko TK, Tan DJY, Fan KS. An evaluation of the quality and readability of online information regarding foreign bodies of the ear, nose and throat (preprint). JMIR Form Res 2023 December 15. Accessed September 18, 2024 at: https://preprints.jmir.org/preprint/55535
• 12 Ghani S, Fan KS, Fan KH, Lenti L, Raptis D. Using the ensuring quality information for patients tool to assess patient information on appendicitis websites: systematic search and evaluation. J Med Internet Res 2021; 23 (03) e22618
• 13 ENT UK. ENT Conditions. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions
• 14 Shamil E, Jaafar M, Fan KS. et al. The use of large language models like ChatGPT on delivering patient information relating to surgery. Facial Plast Surg 2024
• 15 Goodman RS, Patrinely Jr JR, Osterman T, Wheless L, Johnson DB. On the cusp: considering the impact of artificial intelligence language models in healthcare. Med (N Y) 2023; 4 (03) 139-140
• 16 Chiesa-Estomba CM, Lechien JR, Vaira LA. et al. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur Arch Otorhinolaryngol 2024; 281 (04) 2081-2086
• 17 Garg RK, Urs VL, Agarwal AA, Chaudhary SK, Paliwal V, Kar SK. Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: a systematic review. Health Promot Perspect 2023; 13 (03) 183-191
• 18 Moult B, Franck LS, Brady H. Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect 2004; 7 (02) 165-175
• 19 Charvet-Berard AI, Chopard P, Perneger TV. Measuring quality of patient information documents with an expanded EQIP scale. Patient Educ Couns 2008; 70 (03) 407-411
• 20 Heydari P. The validity of some popular readability formulas. Mediterr J Soc Sci 2012; 3 (02) 423
• 21 Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm 2013; 9 (05) 503-516
• 22 Readability Test - WebFX. Accessed September 18, 2024 at: https://www.webfx.com/tools/read-able/
• 23 Świeczkowski D, Kułacz S. The use of the Gunning Fog Index to evaluate the readability of Polish and English drug leaflets in the context of health literacy challenges in medical linguistics: an exploratory study. Cardiol J 2021; 28 (04) 627-631
• 24 Hughes F, Robbins I, Bryan K. An analysis of information available to relatives in intensive care. Intensive Crit Care Nurs 2004; 20 (04) 200-205
• 25 Paasche-Orlow MK, Taylor HA, Brancati FL. Readability standards for informed-consent forms as compared with actual readability. N Engl J Med 2003; 348 (08) 721-726
• 26 Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel. Accessed September 18, 2024 at: http://library.ucf.edu
• 27 Wrigley Kelly NE, Murray KE, McCarthy C, O'Shea DB. An objective analysis of quality and readability of online information on COVID-19. Health Technol (Berl) 2021; 11 (05) 1093-1099
• 28 Open AI. ChatGPT. Accessed September 18, 2024 at: https://help.openai.com/en/collections/3742473-chatgpt
• 29 ENT UK. Septal surgery | ENTUK. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions/62/septal_surgery_new/
• 30 ENT UK. Sinus surgery for chronic rhinosinusitis with nasal polyps | ENTUK. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions/57/sinus_surgery_for_chronic_rhinosinusitis_with_nasal_polyps_new/
• 31 Nosebleeds ENTUK. (Epistaxis) | ENTUK. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions/42/nosebleeds_epistaxis_update/
• 32 ENT UK. Septorhinoplasty | ENTUK. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions/61/septorhinoplasty_new/
• 33 ENT UK. Facial Skin Lesions | ENTUK. Accessed September 18, 2024 at: https://www.entuk.org/patients/conditions/19/facial_skin_lesions
• 34 Kridel RWH. Considerations in the etiology, treatment, and repair of septal perforations. Facial Plast Surg Clin North Am 2004; 12 (04) 435-450 , vi
• 35 Ostherr K. The shifting aesthetics of expertise in the sharing economy of scientific medicine. Sci Context 2018; 31 (01) 107-127
• 36 Wouda JC, van de Wiel HBM. Education in patient-physician communication: how to improve effectiveness?. Patient Educ Couns 2013; 90 (01) 46-53
• 37 Hamaguchi R, Nematollahi S, Minter DJ. Picture of a pandemic: visual aids in the COVID-19 crisis. J Public Health (Oxf) 2020; 42 (03) 483-485
• 38 Woolf K, Potts HWW, McManus IC. Ethnicity and academic performance in UK trained doctors and medical students: systematic review and meta-analysis. BMJ 2011; 342 (7797): d901
• 39 Werner A, Holderried F, Schäffeler N. et al. Communication training for advanced medical students improves information recall of medical laypersons in simulated informed consent talks–a randomized controlled trial. BMC Med Educ 2013; 13 (01) 15
• 40 Hors-Fraile S, Rivera-Romero O, Schneider F. et al. Analyzing recommender systems for health promotion using a multidisciplinary taxonomy: a scoping review. Int J Med Inform 2018; 114: 143-155
• 41 Lee DA, Wu G, Tien K. et al. Can ChatGPT^TM, an intelligent chatbot, be used to educate our glaucoma patients?. Invest Ophthalmol Vis Sci 2023; 64 (08) 379-379
• 42 Kim J, Shin S, Bae K, Oh S, Park E, del Pobil AP. Can AI be a content generator? Effects of content generators and information delivery methods on the psychology of content consumers. Telemat Inform 2020; 55: 101452
• 43 Biswas S. ChatGPT and the Future of Medical Writing. Radiology 2023; 307 (02) e223312
• 44 Lovett J, Gordon C, Patton S, Chen CX. Online information on dysmenorrhoea: An evaluation of readability, credibility, quality and usability. J Clin Nurs 2019; 28 (19-20): 3590-3598
• 45 Zaki HA, Mai M, Abdel-Megid H. et al. Using ChatGPT to improve readability of interventional radiology procedure descriptions. Cardiovasc Intervent Radiol 2024; 47 (08) 1134-1141
• 46 Liu C, Wang D, Liu C. et al. What is the meaning of health literacy? A systematic review and qualitative synthesis. Fam Med Community Health 2020; 8 (02) 351
• 47 BBC. Google AI search tells users to glue pizza and eat rocks - BBC News. 2024 . Accessed September 18, 2024 at: https://www.bbc.co.uk/news/articles/cd11gzejgz4o
• 48 Walker HL, Ghani S, Kuemmerli C. et al. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. J Med Internet Res 2023; 25: e47479