Some common errors of experimental design, interpretation and inference in agreement studies.
We signal and discuss common methodological errors in agreement studies and the use of kappa indices, as found in publications in the medical and behavioural sciences. Our analysis is based on a proposed statistical model that is in line with the typical models employed in metrology and measurement...
| Publicado en: | Statistical Methods in Medical Research Vol. 24; no. 6; pp. 920 - 936 |
|---|---|
| Autores principales: | , , |
| Formato: | equations & formulas research tables/charts Journal Article |
| Publicado: |
Sage Publications Inc.
Dec2015
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=111290220&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 111290220 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 09622802 31F jtl: Statistical Methods in Medical Research issn: 09622802 maglogo: Y pubinfo: dt: Dec2015 vid: 24 iid: 6 pid: 344 pub: Sage Publications Inc. place: Thousand Oaks, California artinfo: ui: 111290220 111290220 NLM22232301 111290220 10.1177/0962280211433597 NLM22232301 111290220 ppf: 920 ppct: 16 formats: tig: atl: Some common errors of experimental design, interpretation and inference in agreement studies. aug: au: Erdmann, T. P. De Mast, J. Warrens, M. J. affil: Institute for Business and Industrial Statistics of the University of Amsterdam, Plantage Muidergracht 12, 1018 TV Amsterdam, The Netherlands. sug: subj: Study Design Standards Data Analysis, Statistical Bias (Research) Observer Bias Reproducibility of Results Probability Sample Models, Statistical ab: We signal and discuss common methodological errors in agreement studies and the use of kappa indices, as found in publications in the medical and behavioural sciences. Our analysis is based on a proposed statistical model that is in line with the typical models employed in metrology and measurement theory. A first cluster of errors is related to nonrandom sampling, which results in a potentially substantial bias in the estimated agreement. Second, when class prevalences are strongly nonuniform, the use of the kappa index becomes precarious, as its large partial derivatives result in typically large standard errors of the estimates. In addition, the index reflects rather one-sidedly in such cases the consistency of the most prevalent class, or the class prevalences themselves. A final cluster of errors concerns interpretation pitfalls, which may lead to incorrect conclusions based on agreement studies. These interpretation issues are clarified on the basis of the proposed statistical modelling. The signalled errors are illustrated from actual studies published in prestigious journals. The analysis results in a number of guidelines and recommendations for agreement studies, including the recommendation to use alternatives to the kappa index in certain situations. pubtype: Academic Journal doctype: equations & formulas research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|