Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence.
Weighted inter-rater agreement allows for differentiation between levels of disagreement among rating categories and is especially useful when there is an ordinal relationship between categories. Many existing weighted inter-rater agreement coefficients are either extensions of weighted Kappa or are...
| Publicado en: | Educational & Psychological Measurement Vol. 86; no. 3; pp. 484 - 525 |
|---|---|
| Autor principal: | |
| Formato: | Artículo |
| Publicado: |
Sage Publications Inc.
Jun2026
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ssf&AN=193622810&site=ehost-live header: @attributes: shortDbName: ssf uiTerm: 193622810 longDbName: Social Sciences Full Text (H.W. Wilson) uiTag: AN controlInfo: bkinfo: jinfo: jid: 00131644 EPM jtl: Educational & Psychological Measurement issn: 00131644 maglogo: Y pubinfo: dt: Jun2026 vid: 86 iid: 3 pid: 344 pub: Sage Publications Inc. artinfo: ui: 193622810 10.1177/00131644251376553 ppf: 484 ppct: 41 formats: tig: atl: Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence. aug: au: Almehrizi, Rashid Saif affil: Sultan Qaboos University, Muscat, Oman su: Conflict (Psychology) Statistical correlation Statistical models Mathematics Disease prevalence Simulation methods in education Statistics Hypothesis Conceptual structures Confidence intervals Inter-observer reliability Sampling errors sug: subj: Conflict (Psychology) Statistical correlation Statistical models Mathematics Disease prevalence Simulation methods in education Statistics Hypothesis Conceptual structures Confidence intervals Inter-observer reliability Sampling errors keyword: prevalence-agreement effect weighted disagreement weighted Kappa weighted Lambda prevalence-agreement effect weighted disagreement weighted Kappa weighted Lambda ab: Weighted inter-rater agreement allows for differentiation between levels of disagreement among rating categories and is especially useful when there is an ordinal relationship between categories. Many existing weighted inter-rater agreement coefficients are either extensions of weighted Kappa or are formulated as Cohen's Kappa-like coefficients. These measures suffer from the same issues as Cohen's Kappa, including sensitivity to the marginal distributions of raters and the effects of category prevalence. They primarily account for the possibility of chance agreement or disagreement. This article introduces a new coefficient, weighted Lambda, which allows for the inclusion of varying weights assigned to disagreements. Unlike traditional methods, this coefficient does not assume random assignment and does not adjust for chance agreement or disagreement. Instead, it modifies the observed percentage of agreement while taking into account the anticipated impact of prevalence-agreement effects. The study also outlines techniques for estimating sampling standard errors, conducting hypothesis tests, and constructing confidence intervals for weighted Lambda. Illustrative numerical examples and Monte Carlo simulations are presented to investigate and compare the performance of the new weighted Lambda with commonly used weighted inter-rater agreement coefficients across various true agreement levels and agreement matrices. Results demonstrate several advantages of the new coefficient in measuring weighted inter-rater agreement. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: N holdings: @attributes: islocal: N |
|---|