Uncontrolled Control Variables: The Extent That a Researcher's Degrees of Freedom With Control Variables Increases Various Types of Statistical Errors.
Researchers have increasingly warned about "p-hacking" and the improper use of control variables. This paper considers the risk that a "researcher's degrees of freedom" with respect to the use of control variables has on the probability of Type I and Type II errors. We also examine the extent that c...
| Published in: | Journal of Applied Psychology Vol. 107; no. 1; pp. 9 - 23 |
|---|---|
| Main Authors: | , , |
| Format: | Article |
| Published: |
American Psychological Association
Jan2022
|
| Subjects: | |
| Online Access: | View this record in EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ssf&AN=161853700&site=ehost-live header: @attributes: shortDbName: ssf uiTerm: 161853700 longDbName: Social Sciences Full Text (H.W. Wilson) uiTag: AN controlInfo: bkinfo: jinfo: jid: 00219010 JAY jtl: Journal of Applied Psychology issn: 00219010 maglogo: N pubinfo: dt: Jan2022 vid: 107 iid: 1 pid: 34 pub: American Psychological Association artinfo: ui: 161853700 10.1037/apl0000849 ppf: 9 ppct: 14 formats: tig: atl: Uncontrolled Control Variables: The Extent That a Researcher's Degrees of Freedom With Control Variables Increases Various Types of Statistical Errors. aug: au: Sturman, Michael C. Sturman, A. J. Sturman, Caleb J. affil: School of Management and Labor Relations, Rutgers University Cayuga Medical Center, Ithaca, New York, USA School of International Service, American University su: Degrees of freedom Statistical errors False positive error Bivariate analysis Multivariate analysis sug: subj: Degrees of freedom Statistical errors False positive error Bivariate analysis Multivariate analysis keyword: computer simulation techniques ethics in research field research p-hacking computer simulation techniques ethics in research field research p-hacking ab: Researchers have increasingly warned about "p-hacking" and the improper use of control variables. This paper considers the risk that a "researcher's degrees of freedom" with respect to the use of control variables has on the probability of Type I and Type II errors. We also examine the extent that control variables can make marginal effect sizes (i.e., nonzero effects but less than should be statistically significant) appear significant (which we refer to as Type III errors) and how much control variable use can increase effect sizes. We report the results of two computer simulations that include up to 10 control variables. We find that the inappropriate use of control variables is not really a risk for Type I errors, given that the chance of there being truly a null effect in a typical multivariate analysis is very low. We also show that the use of control variables does not have a large effect on Type II errors, and that the practice of running analyses both with and without control variables will most often yield the same conclusion in both analyses. That said, we did find that p-hacking substantially increases the probability of inappropriately being able to detect statistical significance and can notably increase effect sizes. The practice of running analyses both with and without control variables can indeed reveal the potential for p-hacking, and discrepant results between bivariate and multivariate analyses suggest that authors need to carefully and clearly explain why the noted differences are theoretically and logically appropriate. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: N holdings: @attributes: islocal: N |
|---|