A comprehensive evaluation of semantic relation knowledge of pretrained language models and humans.
Recently, much work has concerned itself with the enigma of what exactly pretrained language models (PLMs) learn about different aspects of language, and how they learn it. One stream of this type of research investigates the knowledge that PLMs have about semantic relations. However, many aspects o...
| Published in: | Language Resources & Evaluation Vol. 59; no. 4; pp. 3799 - 3850 |
|---|---|
| Main Authors: | , , , |
| Format: | Article |
| Published: |
Springer Nature
Dec2025
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=189912030&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 189912030 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Dec2025 vid: 59 iid: 4 pid: 237 pub: Springer Nature artinfo: ui: 189912030 10.1007/s10579-025-09858-9 ppf: 3799 ppct: 51 formats: fmt: – @attributes: type: T – @attributes: type: P size: 2.7MB tig: atl: A comprehensive evaluation of semantic relation knowledge of pretrained language models and humans. aug: au: Cao, Zhihan Yamada, Hiroaki Teufel, Simone Tokunaga, Takenobu affil: https://ror.org/05dqf9946 School of Computing, Institute of Science Tokyo, Tokyo, Japan https://ror.org/013meh722 Department of Computer Science and Technology, University of Cambridge, Cambridge, UK su: Language models Natural language processing Evaluation methodology Semantics Cognitive ability Generalization sug: subj: Language models Natural language processing Evaluation methodology Semantics Cognitive ability Generalization keyword: Human-model Comparison Lexical Semantics Pretrained Language Models Prompt-based Probing Semantic Relations ab: Recently, much work has concerned itself with the enigma of what exactly pretrained language models (PLMs) learn about different aspects of language, and how they learn it. One stream of this type of research investigates the knowledge that PLMs have about semantic relations. However, many aspects of semantic relations were left unexplored. Generally, only one relation has been considered, namely hypernymy. Furthermore, previous work did not measure humans' performance on the same task as that performed by the PLMs. This means that at this point in time, there is only an incomplete view of the extent of these models' semantic relation knowledge. To address this gap, we introduce a comprehensive evaluation framework covering five relations beyond hypernymy, namely hyponymy, holonymy, meronymy, antonymy, and synonymy. We use five metrics (two newly introduced here) for recently untreated aspects of semantic relation knowledge, namely soundness, completeness, symmetry, prototypicality, and distinguishability. Using these, we can fairly compare humans and models on the same task. Our extensive experiments involve six PLMs, four masked and two causal language models. The results reveal a significant knowledge gap between humans and models for all semantic relations. In general, causal language models, despite their wide use, do not always perform significantly better than masked language models. Antonymy is the outlier relation where all models perform reasonably well. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2025. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2025 holdings: @attributes: islocal: N |
|---|