A Pilot Study on Generative Artificial Intelligence's Reliability in Qualitative Research Quality Appraisal Using CASP and JBI Checklists.
Generative artificial intelligence (genAI) tools are transforming workflows, with growing interest in their potential applications in qualitative research. While the use of genAI in facilitating the systematic review process has been explored, its application in the quality appraisal of qualitative...
| Publicado en: | Inquiry (00469580) Vol. 62; pp. 1 - 10 |
|---|---|
| Autores principales: | , , , |
| Formato: | research tables/charts Journal Article |
| Publicado: |
Sage Publications Inc.
11/29/2025
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=189855744&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 189855744 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 00469580 INQ jtl: Inquiry (00469580) issn: 00469580 maglogo: Y pubinfo: dt: 11/29/2025 vid: 62 pid: 344 pub: Sage Publications Inc. place: Thousand Oaks, California artinfo: ui: 189855744 189855744 189855744 10.1177/00469580251399374 189855744 ppf: 1 ppct: 9 formats: fmt: – @attributes: type: T – @attributes: type: P tig: atl: A Pilot Study on Generative Artificial Intelligence's Reliability in Qualitative Research Quality Appraisal Using CASP and JBI Checklists. aug: au: Shereefdeen, Hisba Thaivalappil, Abhinand Young, Ian MacKay, Melissa affil: University of Guelph, ON, Canada sug: subj: Artificial Intelligence, Generative Qualitative Studies Checklists Utilization Quality Assessment Reproducibility of Results Human Male Female Pilot Studies Consensus Computer-Assisted Instruction Interrater Reliability Professional Practice, Evidence-Based Study Design Research Ethics Bias (Research) Judgment Voice User-Computer Interface Male Female ab: Generative artificial intelligence (genAI) tools are transforming workflows, with growing interest in their potential applications in qualitative research. While the use of genAI in facilitating the systematic review process has been explored, its application in the quality appraisal of qualitative research remains to be understood. This pilot study aims to evaluate the degree to which ChatGPT appraises qualitative research using popular appraisal tools compared to human assessments. Two reviewers applied the Critical Appraisal Skills Program (CASP) and Joanna Briggs Institute (JBI) checklists for qualitative research to studies identified through a previously published review (n = 21). Next, iteratively developed prompts along with a copy of each study were uploaded to ChatGPT to instruct it to appraise each article. Interrater reliability measures and crude agreements were conducted to estimate the level of agreement between human and genAI assessments. Interrater reliability assessments between human and ChatGPT (GPT-5) revealed no agreement to moderate agreement for CASP checklist items (kappa: <.00-.46; crude agreement: 23.8%-100%) and from none to substantial for JBI items (kappa: <.00-.83; crude agreement: 4.8%-95.2%). Agreement was highest for reporting-based elements such as study aims, ethics approval, value of research (CASP), and participant voices and conclusions (JBI). Disagreements were greatest for interpretive and context-dependent items such as research design, researcher–participant relationships, and worldview–methodology congruity. Findings demonstrate that ChatGPT (GPT-5) can reliably identify objective components yet performs inconsistently when assessing items requiring nuance and contextual understanding across both checklists. Currently, any adoption of genAI for quality appraisal of qualitative research must be carefully applied only alongside human assessments and uphold principles of transparency and data privacy. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|