Determinants of grader agreement: an analysis of multiple short answer corpora.
The 'short answer' question format is a widely used tool in educational assessment, in which students write one to three sentences in response to an open question. The answers are subsequently rated by expert graders. The agreement between these graders is crucial for reliable analysis, both in term...
| Publicado en: | Language Resources & Evaluation Vol. 56; no. 2; pp. 387 - 417 |
|---|---|
| Autores principales: | , |
| Formato: | Artículo |
| Publicado: |
Springer Nature
Jun2022
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=157410215&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 157410215 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Jun2022 vid: 56 iid: 2 pid: 237 pub: Springer Nature artinfo: ui: 157410215 10.1007/s10579-021-09547-3 ppf: 387 ppct: 30 formats: fmt: – @attributes: type: T – @attributes: type: P size: 360KB tig: atl: Determinants of grader agreement: an analysis of multiple short answer corpora. aug: au: Padó, Ulrike Padó, Sebastian affil: Hochschule für Technik, Stuttgart, Germany Universität Stuttgart, Stuttgart, Germany su: Corpora Language ability testing Educational planning Educational evaluation Open-ended questions sug: subj: Corpora Language ability testing Educational planning Educational evaluation Open-ended questions keyword: Annotation reliability Educational applications of NLP Hierarchical generalized linear models Short answer grading ab: The 'short answer' question format is a widely used tool in educational assessment, in which students write one to three sentences in response to an open question. The answers are subsequently rated by expert graders. The agreement between these graders is crucial for reliable analysis, both in terms of educational strategies and in terms of developing automatic models for short answer grading (SAG), an active research topic in NLP. This makes it important to understand the properties that influence grader agreement (such as question difficulty, answer length, and answer correctness). However, the twin challenges towards such an understanding are the wide range of SAG corpora in use (which differ along a number of dimensions) and the hierarchical structure of potentially relevant properties (which can be located at the corpus, answer, or question levels). This article uses generalized mixed effects models to analyze the effect of various such properties on grader agreement in six major SAG corpora for two main assessment tasks (language and content assessment). Overall, we find broad agreement among corpora, with a number of properties behaving similarly across corpora (e.g., shorter answers and correct answers are easier to grade). Some properties show more corpus-specific behavior (e.g., the question difficulty level), and some corpora are more in line with general tendencies than others. In sum, we obtain a nuanced picture of how the major short answer grading corpora are similar and dissimilar from which we derive suggestions for corpus development and analysis. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2022. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2022 holdings: @attributes: islocal: N |
|---|