The ambiguity of BERTology: what do large language models represent?
The field of “BERTology” aims to locate linguistic representations in large language models (LLMs). These have commonly been interpreted as representing structural descriptions (SDs) familiar from theoretical linguistics, such as abstract phrase-structures. However, it is unclear how such claims sho...
| Publicado en: | Synthese Vol. 203; no. 1; pp. 1 - 33 |
|---|---|
| Autor principal: | |
| Formato: | Artículo |
| Publicado: |
Springer Nature
Jan2024
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=174455825&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 174455825 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 00397857 4LI jtl: Synthese issn: 00397857 maglogo: N pubinfo: dt: Jan2024 vid: 203 iid: 1 pid: 237 pub: Springer Nature artinfo: ui: 174455825 10.1007/s11229-023-04435-5 ppf: 1 ppct: 32 formats: fmt: – @attributes: type: T – @attributes: type: P size: 1MB tig: atl: The ambiguity of BERTology: what do large language models represent? aug: au: Buder-Gröndahl, Tommi affil: https://ror.org/040af2s02 Department of Digital Humanities, University of Helsinki, Yliopistonkatu 3, 00014, Helsinki, Finland sug: keyword: BERTology Deep learning Language model Linguistic representation ab: The field of “BERTology” aims to locate linguistic representations in large language models (LLMs). These have commonly been interpreted as representing structural descriptions (SDs) familiar from theoretical linguistics, such as abstract phrase-structures. However, it is unclear how such claims should be interpreted in the first place. This paper identifies six possible readings of “linguistic representation” from philosophical and linguistic literature, concluding that none has a straight-forward application to BERTology. In philosophy, representations are typically analyzed as cognitive vehicles individuated by intentional content. This clashes with a prevalent mentalist interpretation of linguistics, which treats SDs as (narrow) properties of cognitive vehicles themselves. I further distinguish between three readings of both kinds, and discuss challenges each brings for BERTology. In particular, some readings would make it trivially false to assign representations of SDs to LLMs, while others would make it trivially true. I illustrate this with the concrete case study of structural probing: a dominant model-interpretation technique. To improve the present situation, I propose that BERTology should adopt a more “LLM-first” approach instead of relying on pre-existing linguistic theories developed for orthogonal purposes. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Synthese is a copyright of Springer, 2024. All Rights Reserved. item: Synthese holder: Springer Nature dt: @attributes: year: 2024 holdings: @attributes: islocal: N |
|---|