Brian Haig, Professor Emeritus
Department of Psychology
University of Canterbury
Christchurch, New Zealand
What do editors of psychology journals think about tests of statistical significance? Questionable editorial directives from Psychological Science
Deborah Mayo’s (2021) recent editorial in Conservation Biology addresses the important issue of how journal editors should deal with strong disagreements about tests of statistical significance (ToSS). Her commentary speaks to applied fields, such as conservation science, but it is relevant to basic research, as well as other sciences, such as psychology. In this short guest commentary, I briefly remark on the role played by the prominent journal, Psychological Science (PS), regarding whether or not researchers should employ ToSS. PS is the flagship journal of the Association for Psychological Science, and two of its editors-in-chief have offered explicit, but questionable, advice on this matter.
In the May 2005 issue of PS, the experimental psychologist, Peter Killeen (2005), published an article on a new statistic, that he maintained overcame some important deficiencies of null hypothesis significance testing. The alternative statistic, ‘prep’, he understood as the probability of replicating an experimental effect. In the same issue of PS, the editor-in chief, James Cutting, opined that Killeen’s article “may change how all psychologists report their statistics”, and he promptly informed prospective contributors to use prep rather than p values when analysing their data. Within a few years, a majority of empirical articles published in PS employed prep, along with effect sizes. This quick rise to local prominence of prep was immediately followed by the publication of a number of articles that were highly critical of the statistic. Among other things, Killeen’s article was criticized for containing mathematical errors, and for not actually being a replication probability.
Significantly, none of the articles critical of prep were published in PS, despite the fact that the journal decided at the time to devote more space to commentaries. One might reasonably fault Cutting’s editorial decision to accord prep favored status before statisticians and research methodologists had time to evaluate its soundness. To this end, he might have used PS as a forum for scrutiny of Killeen’s article. After a few years, and in the face of strong criticism, PS quietly dropped its recommendation that researchers use prep.
In 2014, the first issue of PS contained a tutorial article by Geoff Cumming (2014) on the “new statistics” that was commissioned by the incoming editor-in chief, Erich Eich. In his accompanying editorial (Eich, 2014) explicitly discouraged prospective authors from using null hypothesis significance testing, and invited them to consider using the new statistics of effect sizes, estimation, and meta-analysis. Cumming, now with Bob Calin-Jageman, continues to assiduously promote the new statistics in the form of textbooks, articles, workshops, symposia, tutorials, and a dedicated website. It is fair to say that the new statistics has become the quasi-official position of the Association for Psychological Science, and that PS continues to play a role in the uptake of the new statistics (Giofrè, et al., 2017).
To my knowledge, PS has published no major critical evaluations of the new statistics, nor presented alternatives to them for consideration. In keeping with this uncritical, one-sided attitude, the major proponents of the new statistics have been reluctant to engage with published criticisms of their position. However, a strong methodological pluralism is required for the advancement of knowledge. In particular, the regular critical interplay of alternative perspectives on ToSS is crucial for their ongoing development and understanding. By promoting two questionable alternatives to ToSS (prep and the new statistics), and shunning well-founded alternatives to them (notably, the error-statistical and Bayesian perspectives; see Mayo, 2018; Haig, 2020), the attitudes to ToSS highlighted here can fairly be interpreted as forms of editorial negligence. Although journal editors cannot be expected to solve major statistical controversies, the directives they issue to prospective authors about statistical practice should be properly informed by relevant debates in the statistics wars.
See the previous commentary by Daniel Lakens.
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29.
Eich, E. (2014). Business not as usual. Psychological Science, 25, 3-6.
Giofrè, D., et al. (2017). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PLoS ONE 12 (4): e0175583. https://doi.org/10.1731/joirnal.pone. 0175583
Haig, B. D. (2020). What can psychology’s statistics reformers learn from the error-statistical perspective. Methods in Psychology. https://doi.org/10.1016/j.metip.2020.100020
Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345-352.
Mayo, D. G. (2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.
All commentaries on Mayo (2021) editorial until Jan 31, 2022 (more to come*)
*Let me know if you wish to write one