GPT models for text annotation: An empirical exploration in public policy research.
Text annotation, the practice of labeling text following a predetermined scheme, is essential to qualitative public policy research. Despite its importance, annotating large qualitative data faces challenges of high labor and time costs. Recent developments in large language models (LLMs), specifica...
| Publicado en: | Policy Studies Journal Vol. 54; no. 1; pp. 1 - 18 |
|---|---|
| Autores principales: | , , , |
| Formato: | Artículo |
| Publicado: |
Wiley-Blackwell
Feb2026
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ssf&AN=192436099&site=ehost-live header: @attributes: shortDbName: ssf uiTerm: 192436099 longDbName: Social Sciences Full Text (H.W. Wilson) uiTag: AN controlInfo: bkinfo: jinfo: jid: 0190292X PSJ jtl: Policy Studies Journal issn: 0190292X maglogo: Y pubinfo: dt: Feb2026 vid: 54 iid: 1 pid: 480 pub: Wiley-Blackwell artinfo: ui: 192436099 10.1111/psj.70034 ppf: 1 ppct: 17 formats: fmt: – @attributes: type: T – @attributes: type: C – @attributes: type: P size: 1.8MB tig: atl: GPT models for text annotation: An empirical exploration in public policy research. aug: au: Churchill, Alexander Pichika, Shamitha Xu, Chengxin Liu, Ying affil: Albers School of Business and Economics, Seattle University, Seattle Washington,, USA Department of Computer Science, Seattle University, Seattle Washington,, USA School of Public Affairs and Nonprofit Leadership, Seattle University, Seattle Washington,, USA School of Public Affairs and Administration, Rutgers University, Newark New Jersey,, USA su: Contextual analysis Nursing care facilities Government policy Generative pre-trained transformers Annotations Language models Prompt engineering sug: subj: Contextual analysis Nursing care facilities Government policy Nursing Care Facilities (Skilled Nursing Facilities) Community care facilities for the elderly Generative pre-trained transformers Annotations Language models Prompt engineering keyword: content analysis GPT qualitative method análisis de contenido Método cualitativo 内容分析 定性方法 content analysis GPT qualitative method análisis de contenido Método cualitativo 内容分析 定性方法 ab: Text annotation, the practice of labeling text following a predetermined scheme, is essential to qualitative public policy research. Despite its importance, annotating large qualitative data faces challenges of high labor and time costs. Recent developments in large language models (LLMs), specifically models with generative pretrained transformers (GPTs), show a potential approach that may alleviate the burden of manual text annotation. In this report, we first introduce a small sample pretest strategy for researchers to decide whether to use Open AI's GPT models for text annotation. In addition, we test if GPT models can substitute human coders by comparing the results of two GPT models with different prompting strategies against human annotation. Using email messages collected from a national corresponding experiment in the US nursing home market as an example, on average, we demonstrate 86.25% percentage agreement between GPT and human annotations. We also show that GPT models possess context‐based limitations. Our report ends with reflections and suggestions for readers who are interested in using GPT models for text annotation. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: N holdings: @attributes: islocal: N |
|---|