Experts, coders and crowds: An analysis of substitutability.
Political scientists increasingly use crowdworkers to produce data, predominantly in the context of coding researcher-curated text or to retrieve simple data from the internet. In this article, we provide a theoretical and empirical basis for understanding when crowdworkers can provide data of suffi...
| Published in: | International Political Science Review Vol. 46; no. 4; pp. 622 - 639 |
|---|---|
| Main Authors: | , , , , , , , |
| Format: | Article |
| Published: |
Sage Publications Inc.
Sep2025
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ssf&AN=187256437&site=ehost-live header: @attributes: shortDbName: ssf uiTerm: 187256437 longDbName: Social Sciences Full Text (H.W. Wilson) uiTag: AN controlInfo: bkinfo: jinfo: jid: 01925121 IPC jtl: International Political Science Review issn: 01925121 maglogo: Y pubinfo: dt: Sep2025 vid: 46 iid: 4 pid: 344 pub: Sage Publications Inc. artinfo: ui: 187256437 10.1177/01925121241293459 ppf: 622 ppct: 17 formats: tig: atl: Experts, coders and crowds: An analysis of substitutability. aug: au: Marquardt, Kyle L Pemstein, Daniel Sanhueza Petrarca, Constanza Seim, Brigitte Wilson, Steven Lloyd Bernhard, Michael Coppedge, Michael Lindberg, Staffan I affil: University of Bergen, Norway North Dakota State University, USA Australian National University, Australia University of Minnesota, USA Brandeis University, USA University of Florida, USA University of Notre Dame, USA University of Gothenburg, Sweden su: Political science Crowdsourcing Information retrieval Data quality Statistical reliability Professional employees Data extraction Computer programmers sug: subj: Political science Crowdsourcing Custom Computer Programming Services Computer systems design and related services (except video game design and development) Other Computer Related Services Information retrieval Data quality Statistical reliability Professional employees Data extraction Computer programmers keyword: cross-national research democratic institutions expert surveys survey experiments cross-national research democratic institutions expert surveys survey experiments ab: Political scientists increasingly use crowdworkers to produce data, predominantly in the context of coding researcher-curated text or to retrieve simple data from the internet. In this article, we provide a theoretical and empirical basis for understanding when crowdworkers can provide data of sufficient quality to substitute for other types of coders. First, we introduce a typology of data-producing actors – experts, trained coders and crowds – and hypothesize factors that affect the substitutability of crowdworkers. We then examine how crowdworkers perform across coding tasks that vary along multiple dimensions of difficulty: information verifiability, availability and complexity. The results provide scope conditions bounding the substitutability of crowdworkers in political science applications. Although crowds can substitute for trained coders in the context of relatively simple information retrieval tasks, there is little evidence that crowdworkers can substitute for experts, whose tasks require both information retrieval and data synthesis. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: N holdings: @attributes: islocal: N |
|---|