LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI.
Natural Language Inference (NLI) is considered a representative task to test natural language understanding (NLU). In this work, we propose an extensible framework to collectively yet categorically test diverse Logical reasoning capabilities required for NLI (and, by extension, NLU). Motivated by be...
| Publicado en: | Language Resources & Evaluation Vol. 58; no. 2; pp. 427 - 459 |
|---|---|
| Autores principales: | , , |
| Formato: | Artículo |
| Publicado: |
Springer Nature
Jun2024
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=178064689&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 178064689 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Jun2024 vid: 58 iid: 2 pid: 237 pub: Springer Nature artinfo: ui: 178064689 10.1007/s10579-023-09691-y ppf: 427 ppct: 32 formats: fmt: – @attributes: type: T – @attributes: type: P size: 1.6MB tig: atl: LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI. aug: au: Tarunesh, Ishan Aditya, Somak Choudhury, Monojit affil: Samsung Electronics, Suwon, Korea https://ror.org/03w5sq511 CSE, IIT Kharagpur, Kharagpur, India https://ror.org/02w7f3w92 Microsoft Research, 9 Lavelle Road, Bangalore, India su: Natural languages Linguistic complexity Language ability testing sug: subj: Natural languages Linguistic complexity Language ability testing keyword: Benchmarking Logic NLI Reasoning ab: Natural Language Inference (NLI) is considered a representative task to test natural language understanding (NLU). In this work, we propose an extensible framework to collectively yet categorically test diverse Logical reasoning capabilities required for NLI (and, by extension, NLU). Motivated by behavioral testing, we create a semi-synthetic large test bench (363 templates, 363k examples) and an associated framework that offers the following utilities: (1) individually test and analyze reasoning capabilities along 17 reasoning dimensions (including pragmatic reasoning); (2) design experiments to study cross-capability information content (leave one out or bring one in); and (3) the synthetic nature enables us to control for artifacts and biases. We extend a publicly available framework of automated test case instantiation from free-form natural language templates (CheckList) and a well-defined taxonomy of capabilities to cover a wide range of increasingly harder test cases while varying the complexity of natural language. Through our analysis of state-of-the-art NLI systems, we observe that our benchmark is indeed hard (and non-trivial even with training on additional resources). Some capabilities stand out as harder. Further, fine-grained analysis and fine-tuning experiments reveal more insights about these capabilities and the models – supporting and extending previous observations; thus showing the utility of the proposed testbench. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2024. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2024 holdings: @attributes: islocal: N |
|---|