Considerations in the use of reflective writing for student assessment: issues of reliability and validity.
Context Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. Th...
| Publicado en: | Medical Education Vol. 49; no. 9; pp. 901 - 909 |
|---|---|
| Autores principales: | , , , , , |
| Formato: | research tables/charts Journal Article |
| Publicado: |
Wiley-Blackwell
Sep2015
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=109836865&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 109836865 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 03080110 ESF jtl: Medical Education issn: 03080110 maglogo: Y pubinfo: dt: Sep2015 vid: 49 iid: 9 pid: 480 pub: Wiley-Blackwell place: Malden, Massachusetts artinfo: ui: 109836865 109076201 10.1111/medu.12771 NLM26296406 109836865 ppf: 901 ppct: 8 formats: fmt: @attributes: type: P tig: atl: Considerations in the use of reflective writing for student assessment: issues of reliability and validity. aug: au: Moniz, Tracy Arntfield, Shannon Miller, Kristina Lingard, Lorelei Watling, Chris Regehr, Glenn affil: Department of Communication Studies, Mount Saint Vincent University sug: subj: Student Performance Appraisal Reflection Writing Reliability and Validity Education, Medical Human Descriptive Statistics Coefficient alpha Interrater Reliability Scales Educational Measurement Criterion-Related Validity Correlation Coefficient One-Way Analysis of Variance Repeated Measures Canada Funding Source ab: Context Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking. Methods Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [ REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination ( OSCE) and Year 2 multiple-choice question ( MCQ) examinations to examine, respectively, convergent and divergent validity. Results Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores ( r = 0.13) and MCQ examination scores ( r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance. Conclusions Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|