The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability.
Purpose The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inte...
| Published in: | Journal of Occupational Rehabilitation Vol. 24; no. 3; pp. 511 - 525 |
|---|---|
| Main Authors: | , , , , , |
| Format: | research tables/charts Journal Article |
| Published: |
Springer Nature
Sep2014
|
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=103980644&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 103980644 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 10530487 JOR jtl: Journal of Occupational Rehabilitation issn: 10530487 maglogo: N pubinfo: dt: Sep2014 vid: 24 iid: 3 pid: 237 pub: Springer Nature place: New York, New York artinfo: ui: 103980644 97287695 10.1007/s10926-013-9486-1 NLM24338285 PMC4118042 103980644 ppf: 511 ppct: 14 formats: fmt: @attributes: type: P tig: atl: The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability. aug: au: Turner-Stokes, Lynne Fadyl, Joanna Rose, Hilary Williams, Heather Schlüter, Philip McPherson, Kathryn affil: Person Centred Research Centre, Health and Rehabilitation Research Institute, AUT University, Auckland New Zealand sug: subj: Instrument Validation Clinical Assessment Tools Evaluation Instrument Construction Needs Assessment Methods Rehabilitation, Vocational Vignettes New Zealand United Kingdom Validation Studies Interrater Reliability Intrarater Reliability Intraclass Correlation Coefficient Confidence Intervals Data Analysis Software kappa Statistic Reproducibility of Results Descriptive Statistics Human Clinical Assessment Tools Scales ab: Purpose The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inter- and intra-rater reliability and scoring accuracy, using vignettes. The impact of different methodological approaches to analysis of reliability is highlighted. Methods Following preliminary evaluation using case-histories, six occupational therapists scored vignettes, first individually and then together in two teams. Scoring was repeated blind after 1 month. Scoring accuracy was tested against agreed 'reference standard' vignette scores using intraclass correlation coefficients (ICCs) for total scores and linear-weighted kappas (kw) for individual items. Item-by-item inter- and intra-rater reliability was evaluated for both individual and team scores, using two different statistical methods. Results ICCs for scoring accuracy ranged from 0.95 (95 % CI 0.78-0.98) to 0.96 (0.89-0.99) for Part A, and from 0.78 (95 % CI 0.67-0.85) to 0.84 (0.69-0.92) for Part B. Item by item analysis of scoring accuracy, inter- and intra-rater reliability all showed 'substantial' to 'almost perfect' agreement (kw ≥ 0.60) for all Part-A and 8/12 Part-B items, although multi-rater kappa (Fleiss) produced more conservative results (mK = 0.34-0.79). Team rating produced marginal improvements for Part-A but not Part-B. Four problematic contextual items were identified, leading to adjustment of the scoring manual. Conclusion This vignette-based study demonstrates generally acceptable levels of scoring accuracy and reliability for the WSS. Further testing in real-life situations is now warranted. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|