Guest Post: Ron Kenett: What’s happening in statistical practice since the “abandon statistical significance” call

Posted on July 11, 2024 by Mayo

.

Ron S. Kenett
Chairman of the KPA Group;
Senior Research Fellow, the Samuel Neaman Institute, Technion, Haifa;
Chairman, Data Science Society, Israel

 

What’s happening in statistical practice since the “abandon statistical significance” call

This is a retrospective view from experience gained by applying statistics to a wide range of problems, with an emphasis on the past few years. The post is kept at a general level in order to provide a bird’s eye view of the points being made. Continue reading →

Categories: abandon statistical significance, Wasserstein et al 2019 | 26 Comments

Guest Post (part 2 of 2): Daniël Lakens: “How were we supposed to move beyond  p < .05, and why didn’t we?”

Posted on July 4, 2024 by Mayo

.

Professor Daniël Lakens
Human Technology Interaction
Eindhoven University of Technology

[Some earlier posts by D. Lakens on this topic are at the end of this post]*

This continues Part 1:

4: Most do not offer any alternative at all

At this point, it might be worthwhile to point out that most of the contributions to the special issue do not discuss alternative approaches to p < .05 at all. They discuss general problems with low quality research (Kmetz, 2019), the importance of improving quality control (D. W. Hubbard & Carriquiry, 2019), results blind reviewing (Locascio, 2019), or the role of subjective judgment (Brownstein et al., 2019). There are historical perspectives on how we got to this point (Kennedy-Shaffer, 2019), ideas about how science should work instead, many stressing the importance of replication studies (R. Hubbard et al., 2019; Tong, 2019). Note that Trafimow both recommends replication as an alternative (Trafimow, 2019), but also co-authors a paper stating we should not expect findings to replicate (Amrhein et al., 2019), thereby directly contradicting himself within the same special issue. Others propose not simply giving up on p-values, but on generalizable knowledge (Amrhein et al., 2019). The suggestion is to only report descriptive statistics. Continue reading →

Categories: abandon statistical significance, D. Lakens, Wasserstein et al 2019 | 13 Comments

Guest Post: “Daniël Lakens: How were we supposed to move beyond  p < .05, and why didn’t we? “(part 1 of 2):

Posted on July 1, 2024 by Mayo

.

Professor Daniël Lakens
Human Technology Interaction
Eindhoven University of Technology

*[Some earlier posts by D. Lakens on this topic are listed at the end of part 2, forthcoming this week]

How were we supposed to move beyond  p < .05, and why didn’t we?

It has been 5 years since the special issue “Moving to a world beyond p < .05” came out (Wasserstein et al., 2019). I might be the only person in the world who has read all 43 contributions to this special issue. [In part 1] I will provide a summary of what the articles proposed we should do instead of p < .05, and [in part 2] offer some reflections on why they did not lead to any noticeable change. Continue reading →

Categories: abandon statistical significance, D. Lakens, Wasserstein et al. (2019) | 23 Comments

Guest Post: Christian Hennig: “Statistical tests in five random research papers of 2024, and related thoughts on the ‘don’t say significant’ initiative”

Posted on June 25, 2024 by Mayo

.

Professor Christian Hennig
Department of Statistical Sciences “Paolo Fortunati”
University of Bologna

[An earlier post by C. Hennig on this topic:  Jan 9, 2022: The ASA controversy on P-values as an illustration of the difficulty of statistics]

Statistical tests in five random research papers of 2024, and related thoughts on the “don’t say significant” initiative

This text follows an invitation to write on “abandon statistical significance 5 years on”, so I decided to do a tiny bit of empirical research. I had a look at five new papers listed on May 17 on the “Research Articles” site of Scientific Reports. I chose the most recent five papers when I looked without being selective. As I “sampled” papers for a general impression, I don’t want this to be a criticism of particular papers or authors, however in the interest of transparency, the doi addresses of the papers are: Continue reading →

Categories: 5-year memory lane, abandon statistical significance, Christian Hennig | 7 Comments

Guest Post: Andrea Saltelli: Analytic flexibility: a badly kept secret? (thoughts on “abandon statistical significance 5 years on”)

Posted on June 15, 2024 by Mayo
.

.

Professor Andrea Saltelli
UPF Barcelona School of Management, Barcelona, Spain, Centre for the Study of the Sciences and the Humanities, University of Bergen, Bergen, Norway

[An earlier post by A. Saltelli on this topic: Nov 22, 2019: A. Saltelli (Guest post): What can we learn from the debate on statistical significance?]

Analytic flexibility: a badly kept secret?

In a previous post in this blog I expressed concern about a loss of trust that could incur the activity of scientific quantification – as practiced in several discipline – unless some technical and normative element of crisis could be managed. The piece warned that the phenomenon could lead to “a decline of public trust in the findings of science”. Five years and one pandemic later, we may wonder if the danger has indeed materialized. Continue reading →

Categories: abandon statistical significance | Tags: , , , | 11 Comments

2-4 yr review: Commentaries on my Editorial: several are published

Posted on June 4, 2024 by Mayo

I’m reblogging reader commentaries on my editorial, “The statistics wars and intellectual conflicts of interest“. 3 are published in Conservation Biology; a 4th, by Lakens, is in the Journal of the International Society of Physiotherapy.  This post was first published on May 15, 2022. Thus, “soon to be” refers to the past. Share your remarks in the comments. Continue reading →

Categories: 4 years ago!, statistical significance tests, The Statistics Wars and Their Casualties | Leave a comment

2-4 year review: The Statistics Wars and Intellectual Conflicts of Interest

Posted on May 31, 2024 by Mayo

.

Before posting new reflections on where we are 5 years after the ASA P-value controversy–both my own and readers’–I will reblog some reader commentaries from 2022 in connection with my (2022) editorial in Conservation Biology: “The Statistical Wars and Intellectual Conflicts of Interest”. First, here are excerpts from my editorial: Continue reading →

Categories: 3-year memory lane, abandon statistical significance, stat activist watch 2023, stat wars and their casualties | Leave a comment

5-year review: “Les stats, c’est moi”: We take that step here! (Adopt our fav word or phil stat!)(iii)

Posted on May 21, 2024 by Mayo

 

les stats, c’est moi

This is the last of the selected posts I will reblog from 5 years ago on the 2019 statistical significance controversy. The original post, published on this blog on December 13, 2019, had 85 comments, so you might find them of interest.  I invite readers to share their thoughts as to where the field is now, in relation to that episode, and to alternatives being used as replacements for statistical significance tests. Use the comments and send me guest posts.  Continue reading →

Categories: 5-year memory lane, Error Statistics, statistical significance tests | Leave a comment

5-year Review: The ASA’s P-value Project: Why it’s Doing More Harm than Good (cont from 11/4/19)

Posted on May 19, 2024 by Mayo

.

I continue my selective 5-year review of some of the posts revolving around the statistical significance test controversy from 2019. This post was first published on the blog on November 14, 2019. I feared then that many of the howlers of statistical significance tests would be further etched in granite after the ASA’s P-value project, and in many quarters this is, unfortunately, true. One that I’ve noticed quite a lot is the (false) supposition that negative results are uninformative. Some fields, notably psychology, keep to a version of simple Fisherian tests, ignoring Neyman-Pearson (N-P) tests (never minding that Jacob Cohen was a psychologist who gave us “power analysis”).  (See note [1]) For N-P, “it is immaterial which of the two alternatives…is labelled the hypothesis tested” (Neyman 1950, 259). Failing to find evidence of a genuine effect, coupled with a test’s having high capability to detect meaningful effects, warrants inferring the absence of meaningful effects. Even with the simple Fisherian test, failing to reject H0 is informative. Null results figure importantly throughout science, such as when the ether was falsified by Michelson-Morley, and in directing attention away from unproductive theory development.

Please share your comments on this blogpost. Continue reading →

Categories: 5-year memory lane, statistical significance tests, straw person fallacy | 1 Comment

5-year Review: P-Value Statements and Their Unintended(?) Consequences: The June 2019 ASA President’s Corner (b)

Posted on May 10, 2024 by Mayo

I continue my 5-year review of some highlights from the “abandon significance” movement from 2019. This post was first published on this blog on November 30, 2019,  It was based on a call by then American Statistical Association President, Karen Kafadar, which sparked a counter-movement. I will soon begin sharing a few invited guest posts reflecting on current thinking either on the episode or on statistical methodology more generally. I may continue to post such reflections over the summer, as they come in, so let me know if you’d like to contribute something. Share your thoughts in the comments.

2208388671_0d8bc38714

Mayo writing to Kafadar

I never met Karen Kafadar, the 2019 President of the American Statistical Association (ASA), but the other day I wrote to her in response to a call in her extremely interesting June 2019 President’s Corner: “Statistics and Unintended Consequences“:

  • “I welcome your suggestions for how we can communicate the importance of statistical inference and the proper interpretation of p-values to our scientific partners and science journal editors in a way they will understand and appreciate and can use with confidence and comfort—before they change their policies and abandon statistics altogether.”

I only recently came across her call, and I will share my letter below. First, here are some excerpts from her June President’s Corner (her December report is due any day). Continue reading →

Categories: 5-year memory lane, stat wars and their casualties, statistical significance tests | Leave a comment

5-year review: Hardwicke and Ioannidis, Gelman, and Mayo: P-values: Petitions, Practice, and Perils

Posted on May 4, 2024 by Mayo

 

.

Soon after the Wasserstein et al (2019) “don’t say significance” editorial, John Ioannidis invited Andrew Gelman and I to write editorials from our different perspectives on an associated editorial that Nature invited. It was written by Amrhein, Greenland and McShane (AGM, 2019). Prior to the publication of AGM 2019, people were given the opportunity to add their names to the Nature article.

A campaign followed that aimed at the collection of signatures in what was called a ‘petition’ on the widely popular blogsite of Andrew Gelman. Ultimately, 854 scientists signed the petition and the list of their names was published along with commentary. (Hardwicke and Ioannidis, 2019, p. 2)

Tom Hardwicke and John Ioannidis (2019) took advantage of the opportunity “to perform a survey of the signatories to understand how and why they signed the endorsement” (ibid.). This post, reblogged from September 25 2019, includes all 3 articles: the survey by Hardwicke and Ioannidis, and the editorials by Gelman and I. They appeared in the European Journal of Clinical Investigations (2019). I’m still interested in reader responses (in the comments) to the question I pose. Continue reading →

Categories: 5-year memory lane, abandon statistical significance | Leave a comment

5-year Review: B. Haig: [TAS] 2019 update on P-values and significance (ASA II)(Guest Post)

Posted on April 23, 2024 by Mayo

This is the guest post by Bran Haig on July 12, 2019 in response to the “abandon statistical significance” editorial in The American Statistician (TAS) by Wasserstein, Schirm, and Lazar (WSL 2019). In the post it is referred to as ASAII with a note added once we learned that it is actually not a continuation of the 2016 ASA policy statement. (I decided to leave it that way, as otherwise the context seems lost. But in the title to this post, I refer to the journal TAS.) Brian lists some of the benefits that were to result from abandoning statistical significance. I welcome your constructive thoughts in the comments.

Brian Haig, Professor Emeritus
Department of Psychology
University of Canterbury
Christchurch, New Zealand Continue reading →

Categories: 5-year memory lane, abandon statistical significance, ASA Guide to P-values, Brian Haig | Tags: | Leave a comment

5-year review: The NEJM Issues New Guidelines on Statistical Reporting: Is the ASA P-Value Project Backfiring? (i)

Posted on April 19, 2024 by Mayo

In a July 19, 2019 post I discussed The New England Journal of Medicine’s response to Wasserstein’s (2019) call for journals to change their guidelines in reaction to the “abandon significance” drive. The NEJM said “no thanks” [A]. However confidence intervals CIs got hurt in the mix. In this reblog, I kept the reference to “ASA II” with a note, because that best conveys the context of the discussion at the time. Switching it to WSL (2019) just didn’t read right. I invite your comments. Continue reading →

Categories: 5-year memory lane, abandon statistical significance, ASA Guide to P-values | 6 Comments

5-year review: Don’t let the tail wag the dog by being overly influenced by flawed statistical inferences

Posted on April 17, 2024 by Mayo

.

On June 1, 2019, I posted portions of an article [i],“There is Still a Place for Significance Testing in Clinical Trials,” in Clinical Trials responding to the 2019 call to abandon significance. I reblog it here. While very short, it effectively responds to the 2019 movement (by some) to abandon the concept of statistical significance [ii]. I have recently been involved in researching drug trials for a condition of a family member, and I can say that I’m extremely grateful that they are still reporting error statistical assessments of new treatments, and using carefully designed statistical significance tests with thresholds. Without them, I think we’d be lost in a sea of potential treatments and clinical trials. Please share any of your own experiences in the comments. The emphasis in this excerpt is mine: 

Much hand-wringing has been stimulated by the reflection that reports of clinical studies often misinterpret and misrepresent the findings of the statistical analyses. Recent proposals to address these concerns have included abandoning p-values and much of the traditional classical approach to statistical inference, or dropping the concept of statistical significance while still allowing some place for p-values. How should we in the clinical trials community respond to these concerns? Responses may vary from bemusement, pity for our colleagues working in the wilderness outside the relatively protected environment of clinical trials, to unease about the implications for those of us engaged in clinical trials…. Continue reading →

Categories: 5-year memory lane, abandon statistical significance, statistical tests | 9 Comments

My 2019 friendly amendments to that “abandon significance” editorial

Posted on April 5, 2024 by Mayo

.

It was 3 months before I decided to write a blogpost in response to Wasserstein, Schirm and Lazar (2019)’s editorial in The American Statistician in which they recommend that the concept of “statistical significance” be abandoned, hereafter, WSL 2019. (I titled it “Don’t Say What You don’t Mean”.) In that June 17, 2019 blogpost, pasted below, I proposed 3 “friendly amendments” to the language of that document. (There are 97 comments on that post!) The problem is that WSL 2019 presents several of the 6 principles from ASA I (the 2016 ASA statement on Statistical Significance) in a far stronger fashion so as to be inconsistent or at least in tension with some of them. I didn’t think they really meant what they said. I discussed these amendments with Ron Wasserstein, Executive Director of the ASA at the time. Had these friendly amendments been carried out, the document would not have caused as much of a problem, and people might focus more on the positive recommendations it includes about scientific integrity. The proposed ban on a key concept of statistics would still be problematic, resulting in the 2019 ASA President’s Task Force, but it would have helped the document.  At the time, it was still not known whether WSL 2019 was intended as a continuation of the 2016 ASA policy document [ASA I]. That explains why I first referred to WSL 2019 in this blogpost as ASA II. Once it was revealed that it was not official policy at all (many months later), but only the recommendations of the 3 authors, I placed a “note” after each mention of ASA II. But given it caused sufficient confusion as to result in the then ASA president (Karen Kafadar) appointing an ASA Task Force on Statistical Significance and Replicability in 2019 (see here and here), and later, a disclaimer by the authors, in this reblog I refer to it as WSL 2019. You can search this blog for other posts on the 2019 Task Force: their report is here, and the disclaimer here. Continue reading →

Categories: 2016 ASA Statement on P-values, ASA Guide to P-values, ASA Task Force on Significance and Replicability | Leave a comment

5 years ago today, March 20, 2019: the Start of “Abandon Significance”

Posted on March 20, 2024 by Mayo

 

..

Five years ago on this day, a news correspondent at NPR, Richard Harris, published this article, the same day as Wasserstein et al., (2019). Moving to a world beyond “p < 0.05”. TAS, and Amrhein et al., (2019), Comment: Retire statistical significance. Nature. I was one of several people Harris interviewed for his article. He starts by talking of flip-flops regarding the healthfulness of eggs.

Statisticians say it may not be wise to put all your eggs in the significance basket.

A recent study that questioned the healthfulness of eggs raised a perpetual question: Why do studies, as has been the case with health research involving eggs, so often flip-flop from one answer to another? Continue reading →

Categories: stat wars and their casualties, statistical significance | 1 Comment

Preregistration, promises and pitfalls, continued v2

Posted on March 17, 2024 by Mayo

..

In my last post, I sketched some first remarks I would have made had I been able to travel to London to fulfill my invitation to speak at a Royal Society conference, March 4 and 5, 2024, on “the promises and pitfalls of preregistration.” This is a continuation. It’s a welcome consequence of today’s statistical crisis of replication that some social sciences are taking a page from medical trials and calling for preregistration of sampling protocols and full reporting. In 2018, Brian Nosek and others wrote of the “Preregistration Revolution”, as part of open science initiatives. Continue reading →

Categories: Bayesian/frequentist, Likelihood Principle, preregistration, Severity | 3 Comments

Promises and Pitfalls of Preregistration: A Royal Society conference I was to speak at

Posted on March 6, 2024 by Mayo

.

I had been invited to speak at a Royal Society meeting, held March 4 and 5, 2024, on “the promises and pitfalls of preregistration”—a topic in which I’m keenly interested. The meeting was organized by Dr Tom Hardwicke, Professor Marcus Munafò, Dr Sophia Crüwell, Professor Dorothy Bishop FRS FMedSci, and Professor Eric-Jan Wagenmakers. Unfortunately, I was unable to travel to London, so I had to decline attending a few months ago. But, I thought I might jot down some remarks here. Continue reading →

Categories: predesignation | 4 Comments

Happy Birthday R.A. Fisher: “Statistical methods and Scientific Induction” with replies by Neyman and E.S. Pearson

Posted on February 17, 2024 by Mayo

17 Feb 1890-29 July 1962

Today is R.A. Fisher’s birthday! I am reblogging what I call the “Triad”–an exchange between  Fisher, Neyman and Pearson (N-P) published 20 years after the Fisher-Neyman break-up. While my favorite is still the reply by E.S. Pearson, which alone should have shattered Fisher’s allegations that N-P “reinterpret” tests of significance as “some kind of acceptance procedure”, all three are chock full of gems for different reasons. They are short and worth rereading. Neyman’s article pulls back the cover on what is really behind Fisher’s over-the-top polemics, what with Russian 5-year plans and commercialism in the U.S. Not only is Fisher jealous that N-P tests came to overshadow “his” tests, he is furious at Neyman for driving home the fact that Fisher’s fiducial approach had been shown to be inconsistent (by others). The flaw is illustrated by Neyman in his portion of the triad. Details may be found in my book, SIST (2018) especially pp 388-392 linked to here. It speaks to a common fallacy seen every day in interpreting confidence intervals. As for Neyman’s “behaviorism”, Pearson’s last sentence is revealing.

HAPPY BIRTHDAY R.A. FISHER! Continue reading →

Categories: E.S. Pearson, Fisher, Neyman, phil/history of stat | 1 Comment

Conference: Is Philosophy Useful for Science, and/or Vice Versa? (Jan 30- Feb 2, 2024)

Posted on January 31, 2024 by Mayo

I will be giving an online talk on Friday, Feb 2, 4:30-5:45 NYC time, at a conference you can watch on zoom this week (Jan 30-Feb 2): Is Philosophy Useful for Science, and/or Vice Versa?  It’s taking place in-person and online at Chapman University. My talk is: “The importance of philosophy of science for Statistical Science and vice versa”. I’ll touch on a current paper I’m writing that (finally) gets back to “Bayesian conceptions of severity”, (in contrast to error statistical severity) as begun on the post on Van Dongen, Springer, and Wagenmaker (2022). Continue reading →

Categories: Announcement | Leave a comment

Blog at WordPress.com.