A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing.
We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired...
| Published in: | Language Resources & Evaluation Vol. 56; no. 4; pp. 1075 - 1103 |
|---|---|
| Main Authors: | , , , |
| Format: | Article |
| Published: |
Springer Nature
Dec2022
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=159971995&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 159971995 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Dec2022 vid: 56 iid: 4 pid: 237 pub: Springer Nature artinfo: ui: 159971995 10.1007/s10579-022-09591-7 ppf: 1075 ppct: 28 formats: fmt: – @attributes: type: T – @attributes: type: P size: 2.1MB tig: atl: A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing. aug: au: Buljan, Maja Nivre, Joakim Oepen, Stephan Øvrelid, Lilja affil: Department of Informatics, Language Technology Group, University of Oslo, Oslo, Norway Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden RISE Research Institutes of Sweden, Kista, Sweden su: Diagnostic errors Natural languages Pilot projects Quantitative research Video coding sug: subj: Diagnostic errors Natural languages Pilot projects Quantitative research Video coding keyword: Contrastive evaluation Data-driven parsing Diagnostics Meaning representation parsing Sentence semantics ab: We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena—by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2022. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2022 holdings: @attributes: islocal: N |
|---|