Reconceptualizing Scoring Reliability Through Linguistic Similarity.
Conventional cross-country scoring reliability in international large-scale assessments often depends on double scoring, which typically involves relatively small samples of multilingual responses. To extend the reach of reliability estimation, this study introduces the Linguistic-integrated Reliabi...
| Publicado en: | Educational & Psychological Measurement Vol. 86; no. 4; pp. 738 - 769 |
|---|---|
| Autores principales: | , , |
| Formato: | Artículo |
| Publicado: |
Sage Publications Inc.
Aug2026
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ssf&AN=195218307&site=ehost-live header: @attributes: shortDbName: ssf uiTerm: 195218307 longDbName: Social Sciences Full Text (H.W. Wilson) uiTag: AN controlInfo: bkinfo: jinfo: jid: 00131644 EPM jtl: Educational & Psychological Measurement issn: 00131644 maglogo: Y pubinfo: dt: Aug2026 vid: 86 iid: 4 pid: 344 pub: Sage Publications Inc. artinfo: ui: 195218307 10.1177/00131644251397428 ppf: 738 ppct: 31 formats: tig: atl: Reconceptualizing Scoring Reliability Through Linguistic Similarity. aug: au: Jung, Ji Yoon Bezirhan, Ummugul von Davier, Matthias affil: Boston College, Chestnut Hill, MA, USA su: Reading Task performance Multilingualism Linguistics Semantics Language acquisition Reliability (Personality trait) Generative artificial intelligence Research methodology evaluation Intelligibility of speech Research methodology Data analysis software Algorithms sug: subj: Reading Task performance Multilingualism Linguistics Semantics Language acquisition Reliability (Personality trait) Generative artificial intelligence Research methodology evaluation Intelligibility of speech Research methodology Data analysis software Algorithms keyword: cross-country scoring consistency ILSAs scoring reliability semantic similarity weighted majority voting cross-country scoring consistency ILSAs scoring reliability semantic similarity weighted majority voting ab: Conventional cross-country scoring reliability in international large-scale assessments often depends on double scoring, which typically involves relatively small samples of multilingual responses. To extend the reach of reliability estimation, this study introduces the Linguistic-integrated Reliability Audit (LiRA), a novel method that measures scoring reliability using an entire dataset in a large-scale, multilingual context. LiRA automatically generates a second score for each response by analyzing its semantic alignment within a neighborhood of similar responses, then applies a weighted majority voting to determine a consensus score. Results demonstrate that LiRA provides a more comprehensive and systematic estimation of scoring reliability at the item, country, and language levels, while preserving the fundamental concepts of traditional reliability. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: N holdings: @attributes: islocal: N |
|---|