MarIA and BETO are sexist: evaluating gender bias in large language models for Spanish.
The study of bias in language models is a growing area of work, however, both research and resources are focused on English. In this paper, we make a first approach focusing on gender bias in some freely available Spanish language models trained using popular deep neural networks, like BERT or RoBER...
| Published in: | Language Resources & Evaluation Vol. 58; no. 4; pp. 1387 - 1418 |
|---|---|
| Main Authors: | , , |
| Format: | Article |
| Published: |
Springer Nature
Dec2024
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=180627302&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 180627302 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Dec2024 vid: 58 iid: 4 pid: 237 pub: Springer Nature artinfo: ui: 180627302 10.1007/s10579-023-09670-3 ppf: 1387 ppct: 31 formats: fmt: – @attributes: type: T – @attributes: type: P size: 1.8MB tig: atl: MarIA and BETO are sexist: evaluating gender bias in large language models for Spanish. aug: au: Garrido-Muñoz, Ismael Martínez-Santiago, Fernando Montejo-Ráez, Arturo affil: https://ror.org/0122p5f64 CEATIC, Universidad de Jaén, Campus Las Lagunillas, 23071, Jaén, Spain su: Artificial neural networks Language models Spanish language Deep learning Sexism Algorithmic bias sug: subj: Artificial neural networks Language models Spanish language Deep learning Sexism Algorithmic bias keyword: BERT Bias evaluation Gender bias Language model RoBERTa ab: The study of bias in language models is a growing area of work, however, both research and resources are focused on English. In this paper, we make a first approach focusing on gender bias in some freely available Spanish language models trained using popular deep neural networks, like BERT or RoBERTa. Some of these models are known for achieving state-of-the-art results on downstream tasks. These promising results have promoted such models' integration in many real-world applications and production environments, which could be detrimental to people affected for those systems. This work proposes an evaluation framework to identify gender bias in masked language models, with explainability in mind to ease the interpretation of the evaluation results. We have evaluated 20 different models for Spanish, including some of the most popular pretrained ones in the research community. Our findings state that varying levels of gender bias are present across these models.This approach compares the adjectives proposed by the model for a set of templates. We classify the given adjectives into understandable categories and compute two new metrics from model predictions, one based on the internal state (probability) and the other one on the external state (rank). Those metrics are used to reveal biased models according to the given categories and quantify the degree of bias of the models under study. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2024. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2024 holdings: @attributes: islocal: N |
|---|