AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek.
The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benc...
| Publicado en: | Digital Scholarship in the Humanities Vol. 39; no. 1; pp. 373 - 393 |
|---|---|
| Autores principales: | , , |
| Formato: | Artículo |
| Publicado: |
Oxford University Press / USA
Apr2024
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=176806345&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 176806345 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 2055768X JEO9 jtl: Digital Scholarship in the Humanities issn: 2055768X maglogo: N pubinfo: dt: Apr2024 vid: 39 iid: 1 pid: 622 pub: Oxford University Press / USA artinfo: ui: 176806345 10.1093/llc/fqad087 ppf: 373 ppct: 20 formats: fmt: – @attributes: type: T – @attributes: type: P size: 868KB tig: atl: AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek. aug: au: Stopponi, Silvia Peels-Matthey, Saskia Nissim, Malvina affil: Centre for Language and Cognition Groningen, University of Groningen , Postbus 716, 9700 AS Groningen, The Netherlands su: Language models Natural language processing Native language Semantics sug: subj: Language models Natural language processing Native language Semantics keyword: ancient Greek ancient languages computational digital humanities historical language linguistics natural language processing semantics ab: The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benchmark for intrinsic evaluation of semantic models of ancient Greek created from expert judgements. In the absence of native speakers, eliciting expert judgements to create a gold standard is a way to leverage a competence that is the closest to that of natives. Moreover, this method allows for collecting data in a uniform way and giving precise instructions to participants. Human judgements about word relatedness were collected via two questionnaires: in the first, experts provided related lemmas to some proposed seeds, while in the second, they assigned relatedness judgements to pairs of lemmas. AGREE was built from a selection of the collected data. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: © 2019 EADH: The European Association for Digital Humanities. item: Digital Scholarship in the Humanities holder: Oxford University Press / USA dt: @attributes: year: 2024 holdings: @attributes: islocal: N |
|---|