| Sumario: | Simple Summary: Large language models (LLMs) are being used more often in healthcare, but we still know little about how well they help patients understand medical information. Until now, no study has asked patients directly how clear they find information generated by LLMs. To address this, we conducted a survey comparing how readable chatbot responses were with how easy patients said they were to understand. Knowledge retention was not assessed. Urology patients used a voice-based chatbot to ask medical questions during regular clinic visits. We analyzed transcripts from 231 conversations. While chatbot answers were written at a level higher than typically recommended for patient education, most patients still found the information understandable. We believe this may be due to how patients received the information, by both reading and listening, which might make it easier to understand. These findings suggest the benefit of multimodal designs in health tools to meet patients' diverse needs. Large language models (LLMs) are increasingly explored as chatbots for patient education, including applications in urooncology. Since only 12% of adults have proficient health literacy and most patient information materials exceed recommended reading levels, improving readability is crucial. Although LLMs could potentially increase the readability of medical information, evidence is mixed, underscoring the need to assess chatbot outputs in clinical settings. Therefore, this study evaluates the measured and perceived readability of chatbot responses in speech-based interactions with urological patients. Urological patients engaged in unscripted conversations with a GPT-4-based chatbot. Transcripts were analyzed using three readability indices: Flesch–Reading-Ease (FRE), Lesbarkeitsindex (LIX) and Wiener-Sachtextformel (WSF). Perceived readability was assessed using a survey covering technical language, clarity and explainability. Associations between measured and perceived readability were analyzed. Knowledge retention was not assessed in this study. A total of 231 conversations were evaluated. The most frequently addressed topics were prostate cancer (22.5%), robotic-assisted prostatectomy (19.9%) and follow-up (18.6%). Objectively, responses were classified as difficult to read (FRE 43.1 ± 9.1; LIX 52.8 ± 6.2; WSF 11.2 ± 1.6). In contrast, perceived readability was rated highly for technical language, clarity and explainability (83–90%). Correlation analyses revealed no association between objective and perceived readability. Chatbot responses were objectively written at a difficult reading level, exceeding recommendations for optimized health literacy. Nevertheless, most patients perceived the information as clear and understandable. This discrepancy suggests that perceived comprehensibility is influenced by factors beyond measurable linguistic complexity.
|
|---|