How to Be a Statistical Detective. - Sainani, Kristin L. (PM & R: Journal of Injury, Function & Rehabilitation, 2020)
Statistical errors are surprisingly common in the biomedical literature and contribute to the reproducibility crisis.[1] Some errors are impossible to spot without access to the underlying dataset, but many are detectable from information available in the paper. Statcheck automatically extracts these numbers from the text and then checks whether an F statistic of 2.791 with numerator degrees of freedom of 1 and denominator degrees of freedom of 50 actually yields a I P i value less than.05. When I uploaded the fun vs exercise paper, statcheck extracted 11 test statistics, 3 of which it flagged as inconsistent, including the F(1,50) = 2.791, I P i < .05 statistic. It can only check statistics that are reported in the text (not tables) and in which the test statistic, degrees of freedom, and I P i value are all available.