CsFEVER and CTKFacts: acquiring Czech data for fact verification.
In this paper, we examine several methods of acquiring Czech data for automated fact-checking, which is a task commonly modeled as a classification of textual claim veracity w.r.t. a corpus of trusted ground truths. We attempt to collect sets of data in form of a factual claim, evidence within the g...
| Publicado en: | Language Resources & Evaluation Vol. 57; no. 4; pp. 1571 - 1606 |
|---|---|
| Autores principales: | , , , , |
| Formato: | Artículo |
| Publicado: |
Springer Nature
Dec2023
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=173723396&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 173723396 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Dec2023 vid: 57 iid: 4 pid: 237 pub: Springer Nature artinfo: ui: 173723396 10.1007/s10579-023-09654-3 ppf: 1571 ppct: 35 formats: fmt: – @attributes: type: T – @attributes: type: P size: 2.2MB tig: atl: CsFEVER and CTKFacts: acquiring Czech data for fact verification. aug: au: Ullrich, Herbert Drchal, Jan Rýpar, Martin Vincourová, Hana Moravec, Václav affil: https://ror.org/03kqpb082 Artificial Intelligence Center, Faculty of Electrical Engineering, Czech Technical University in Prague, Charles Square 13, 120 00, Prague 2, Czech Republic https://ror.org/024d6js02 Department of Journalism, Faculty of Social Sciences, Charles University, Smetanovo nábřeží 6, 110 01, Prague 1, Czech Republic su: Wikipedia Machine translating Natural languages Trust News agencies Deception sug: subj: Wikipedia Machine translating Natural languages Trust News agencies Deception keyword: Automated fact-checking Czech Document retrieval FEVER Natural language inference ab: In this paper, we examine several methods of acquiring Czech data for automated fact-checking, which is a task commonly modeled as a classification of textual claim veracity w.r.t. a corpus of trusted ground truths. We attempt to collect sets of data in form of a factual claim, evidence within the ground truth corpus, and its veracity label (supported, refuted or not enough info). As a first attempt, we generate a Czech version of the large-scale FEVER dataset built on top of Wikipedia corpus. We take a hybrid approach of machine translation and document alignment; the approach and the tools we provide can be easily applied to other languages. We discuss its weaknesses, propose a future strategy for their mitigation and publish the 127k resulting translations, as well as a version of such dataset reliably applicable for the Natural Language Inference task—the CsFEVER-NLI. Furthermore, we collect a novel dataset of 3,097 claims, which is annotated using the corpus of 2.2 M articles of Czech News Agency. We present an extended dataset annotation methodology based on the FEVER approach, and, as the underlying corpus is proprietary, we also publish a standalone version of the dataset for the task of Natural Language Inference we call CTKFactsNLI. We analyze both acquired datasets for spurious cues—annotation patterns leading to model overfitting. CTKFacts is further examined for inter-annotator agreement, thoroughly cleaned, and a typology of common annotator errors is extracted. Finally, we provide baseline models for all stages of the fact-checking pipeline and publish the NLI datasets, as well as our annotation platform and other experimental data. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2023. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2023 holdings: @attributes: islocal: N |
|---|