Quantifying error in OSCE standard setting for varying cohort sizes: A resampling approach to measuring assessment quality.
Background:The use of the borderline regression method (BRM) is a widely accepted standard setting method for OSCEs. However, it is unclear whether this method is appropriate for use with small cohorts (e.g. specialist post-graduate examinations). Aims and methods:This work uses an innovative applic...
| Publicado en: | Medical Teacher Vol. 38; no. 2; pp. 181 - 189 |
|---|---|
| Autores principales: | , , , |
| Formato: | research tables/charts Journal Article |
| Publicado: |
Taylor & Francis Ltd
Feb2016
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=112902419&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 112902419 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 0142159X MCH jtl: Medical Teacher issn: 0142159X maglogo: Y pubinfo: dt: Feb2016 vid: 38 iid: 2 pid: 377 pub: Taylor & Francis Ltd place: Philadelphia, Pennsylvania artinfo: ui: 112902419 112902419 112902419 10.3109/0142159X.2015.1029898 112902419 ppf: 181 ppct: 8 formats: tig: atl: Quantifying error in OSCE standard setting for varying cohort sizes: A resampling approach to measuring assessment quality. aug: au: Homer, Matt Pell, Godfrey Fuller, Richard Patterson, John affil: University of Leeds, UK sug: subj: Sampling Error Educational Measurement Coefficient alpha Human Descriptive Statistics Data Analysis Software Confidence Intervals Checklists Spearman's Rank Correlation Coefficient Sensitivity and Specificity ab: Background:The use of the borderline regression method (BRM) is a widely accepted standard setting method for OSCEs. However, it is unclear whether this method is appropriate for use with small cohorts (e.g. specialist post-graduate examinations). Aims and methods:This work uses an innovative application of resampling methods applied to four pre-existing OSCE data sets (number of stations between 17 and 21) from two institutions to investigate how the robustness of the BRM changes as the cohort size varies. Using a variety of metrics, the ‘quality’ of an OSCE is evaluated for cohorts of approximatelyn = 300 down ton = 15. Estimates of the standard error in station-level and overall pass marks,R2coefficient, and Cronbach’s alpha are all calculated as cohort size varies. Results and conclusion: For larger cohorts (n > 200), the standard error in the overall pass mark is small (less than 0.5%), and for individual stations is of the order of 1–2%. These errors grow as the sample size reduces, with cohorts of less than 50 candidates showing unacceptably large standard error. Alpha andR2also become unstable for small cohorts. The resampling methodology is shown to be robust and has the potential to be more widely applied in standard setting and medical assessment quality assurance and research. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|