Readability of Chatbot Responses in Prostate Cancer and Urological Care: Objective Metrics Versus Patient Perceptions.
Simple Summary: Large language models (LLMs) are being used more often in healthcare, but we still know little about how well they help patients understand medical information. Until now, no study has asked patients directly how clear they find information generated by LLMs. To address this, we cond...
| Publicado en: | Current Oncology Vol. 32; no. 10; pp. 582 - 595 |
|---|---|
| Autores principales: | , , , , , , , , |
| Formato: | Journal Article |
| Publicado: |
MDPI
Oct2025
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=188957488&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 188957488 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 11980052 5EKK jtl: Current Oncology issn: 11980052 maglogo: N pubinfo: dt: Oct2025 vid: 32 iid: 10 pid: 97109 pub: MDPI artinfo: ui: 188957488 10.3390/curroncol32100582 188957488 ppf: 582 ppct: 13 formats: fmt: – @attributes: type: T – @attributes: type: P tig: atl: Readability of Chatbot Responses in Prostate Cancer and Urological Care: Objective Metrics Versus Patient Perceptions. aug: au: Maywald, Lasse Nguyen, Lisa Winterstein, Jana Theres Hetz, Martin Joachim Mangold, Maurin Helen Renner, Luisa Vivienne Brinker, Titus Josef Wessels, Frederik Carl, Nicolas affil: Department of Urology, University Medical Center Mannheim, 68167 Mannheim, Germany sug: ab: Simple Summary: Large language models (LLMs) are being used more often in healthcare, but we still know little about how well they help patients understand medical information. Until now, no study has asked patients directly how clear they find information generated by LLMs. To address this, we conducted a survey comparing how readable chatbot responses were with how easy patients said they were to understand. Knowledge retention was not assessed. Urology patients used a voice-based chatbot to ask medical questions during regular clinic visits. We analyzed transcripts from 231 conversations. While chatbot answers were written at a level higher than typically recommended for patient education, most patients still found the information understandable. We believe this may be due to how patients received the information, by both reading and listening, which might make it easier to understand. These findings suggest the benefit of multimodal designs in health tools to meet patients' diverse needs. Large language models (LLMs) are increasingly explored as chatbots for patient education, including applications in urooncology. Since only 12% of adults have proficient health literacy and most patient information materials exceed recommended reading levels, improving readability is crucial. Although LLMs could potentially increase the readability of medical information, evidence is mixed, underscoring the need to assess chatbot outputs in clinical settings. Therefore, this study evaluates the measured and perceived readability of chatbot responses in speech-based interactions with urological patients. Urological patients engaged in unscripted conversations with a GPT-4-based chatbot. Transcripts were analyzed using three readability indices: Flesch–Reading-Ease (FRE), Lesbarkeitsindex (LIX) and Wiener-Sachtextformel (WSF). Perceived readability was assessed using a survey covering technical language, clarity and explainability. Associations between measured and perceived readability were analyzed. Knowledge retention was not assessed in this study. A total of 231 conversations were evaluated. The most frequently addressed topics were prostate cancer (22.5%), robotic-assisted prostatectomy (19.9%) and follow-up (18.6%). Objectively, responses were classified as difficult to read (FRE 43.1 ± 9.1; LIX 52.8 ± 6.2; WSF 11.2 ± 1.6). In contrast, perceived readability was rated highly for technical language, clarity and explainability (83–90%). Correlation analyses revealed no association between objective and perceived readability. Chatbot responses were objectively written at a difficult reading level, exceeding recommendations for optimized health literacy. Nevertheless, most patients perceived the information as clear and understandable. This discrepancy suggests that perceived comprehensibility is influenced by factors beyond measurable linguistic complexity. pubtype: Academic Journal doctype: Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|