Author Archives: Mayo

5-year Review: The ASA’s P-value Project: Why it’s Doing More Harm than Good (cont from 11/4/19)

Posted on May 19, 2024 by Mayo

.

I continue my selective 5-year review of some of the posts revolving around the statistical significance test controversy from 2019. This post was first published on the blog on November 14, 2019. I feared then that many of the howlers of statistical significance tests would be further etched in granite after the ASA’s P-value project, and in many quarters this is, unfortunately, true. One that I’ve noticed quite a lot is the (false) supposition that negative results are uninformative. Some fields, notably psychology, keep to a version of simple Fisherian tests, ignoring Neyman-Pearson (N-P) tests (never minding that Jacob Cohen was a psychologist who gave us “power analysis”).  (See note [1]) For N-P, “it is immaterial which of the two alternatives…is labelled the hypothesis tested” (Neyman 1950, 259). Failing to find evidence of a genuine effect, coupled with a test’s having high capability to detect meaningful effects, warrants inferring the absence of meaningful effects. Even with the simple Fisherian test, failing to reject H0 is informative. Null results figure importantly throughout science, such as when the ether was falsified by Michelson-Morley, and in directing attention away from unproductive theory development.

Please share your comments on this blogpost. Continue reading →

Categories: 5-year memory lane, statistical significance tests, straw person fallacy | 1 Comment

5-year Review: P-Value Statements and Their Unintended(?) Consequences: The June 2019 ASA President’s Corner (b)

Posted on May 10, 2024 by Mayo

I continue my 5-year review of some highlights from the “abandon significance” movement from 2019. This post was first published on this blog on November 30, 2019,  It was based on a call by then American Statistical Association President, Karen Kafadar, which sparked a counter-movement. I will soon begin sharing a few invited guest posts reflecting on current thinking either on the episode or on statistical methodology more generally. I may continue to post such reflections over the summer, as they come in, so let me know if you’d like to contribute something. Share your thoughts in the comments.

2208388671_0d8bc38714

Mayo writing to Kafadar

I never met Karen Kafadar, the 2019 President of the American Statistical Association (ASA), but the other day I wrote to her in response to a call in her extremely interesting June 2019 President’s Corner: “Statistics and Unintended Consequences“:

  • “I welcome your suggestions for how we can communicate the importance of statistical inference and the proper interpretation of p-values to our scientific partners and science journal editors in a way they will understand and appreciate and can use with confidence and comfort—before they change their policies and abandon statistics altogether.”

I only recently came across her call, and I will share my letter below. First, here are some excerpts from her June President’s Corner (her December report is due any day). Continue reading →

Categories: 5-year memory lane, stat wars and their casualties, statistical significance tests | Leave a comment

5-year review: Hardwicke and Ioannidis, Gelman, and Mayo: P-values: Petitions, Practice, and Perils

Posted on May 4, 2024 by Mayo

 

.

Soon after the Wasserstein et al (2019) “don’t say significance” editorial, John Ioannidis invited Andrew Gelman and I to write editorials from our different perspectives on an associated editorial that Nature invited. It was written by Amrhein, Greenland and McShane (AGM, 2019). Prior to the publication of AGM 2019, people were given the opportunity to add their names to the Nature article.

A campaign followed that aimed at the collection of signatures in what was called a ‘petition’ on the widely popular blogsite of Andrew Gelman. Ultimately, 854 scientists signed the petition and the list of their names was published along with commentary. (Hardwicke and Ioannidis, 2019, p. 2)

Tom Hardwicke and John Ioannidis (2019) took advantage of the opportunity “to perform a survey of the signatories to understand how and why they signed the endorsement” (ibid.). This post, reblogged from September 25 2019, includes all 3 articles: the survey by Hardwicke and Ioannidis, and the editorials by Gelman and I. They appeared in the European Journal of Clinical Investigations (2019). I’m still interested in reader responses (in the comments) to the question I pose. Continue reading →

Categories: 5-year memory lane, abandon statistical significance | Leave a comment

5-year Review: B. Haig: [TAS] 2019 update on P-values and significance (ASA II)(Guest Post)

Posted on April 23, 2024 by Mayo

This is the guest post by Bran Haig on July 12, 2019 in response to the “abandon statistical significance” editorial in The American Statistician (TAS) by Wasserstein, Schirm, and Lazar (WSL 2019). In the post it is referred to as ASAII with a note added once we learned that it is actually not a continuation of the 2016 ASA policy statement. (I decided to leave it that way, as otherwise the context seems lost. But in the title to this post, I refer to the journal TAS.) Brian lists some of the benefits that were to result from abandoning statistical significance. I welcome your constructive thoughts in the comments.

Brian Haig, Professor Emeritus
Department of Psychology
University of Canterbury
Christchurch, New Zealand Continue reading →

Categories: 5-year memory lane, abandon statistical significance, ASA Guide to P-values, Brian Haig | Tags: | Leave a comment

5-year review: The NEJM Issues New Guidelines on Statistical Reporting: Is the ASA P-Value Project Backfiring? (i)

Posted on April 19, 2024 by Mayo

In a July 19, 2019 post I discussed The New England Journal of Medicine’s response to Wasserstein’s (2019) call for journals to change their guidelines in reaction to the “abandon significance” drive. The NEJM said “no thanks” [A]. However confidence intervals CIs got hurt in the mix. In this reblog, I kept the reference to “ASA II” with a note, because that best conveys the context of the discussion at the time. Switching it to WSL (2019) just didn’t read right. I invite your comments. Continue reading →

Categories: 5-year memory lane, abandon statistical significance, ASA Guide to P-values | 6 Comments

5-year review: Don’t let the tail wag the dog by being overly influenced by flawed statistical inferences

Posted on April 17, 2024 by Mayo

.

On June 1, 2019, I posted portions of an article [i],“There is Still a Place for Significance Testing in Clinical Trials,” in Clinical Trials responding to the 2019 call to abandon significance. I reblog it here. While very short, it effectively responds to the 2019 movement (by some) to abandon the concept of statistical significance [ii]. I have recently been involved in researching drug trials for a condition of a family member, and I can say that I’m extremely grateful that they are still reporting error statistical assessments of new treatments, and using carefully designed statistical significance tests with thresholds. Without them, I think we’d be lost in a sea of potential treatments and clinical trials. Please share any of your own experiences in the comments. The emphasis in this excerpt is mine: 

Much hand-wringing has been stimulated by the reflection that reports of clinical studies often misinterpret and misrepresent the findings of the statistical analyses. Recent proposals to address these concerns have included abandoning p-values and much of the traditional classical approach to statistical inference, or dropping the concept of statistical significance while still allowing some place for p-values. How should we in the clinical trials community respond to these concerns? Responses may vary from bemusement, pity for our colleagues working in the wilderness outside the relatively protected environment of clinical trials, to unease about the implications for those of us engaged in clinical trials…. Continue reading →

Categories: 5-year memory lane, abandon statistical significance, statistical tests | 9 Comments

My 2019 friendly amendments to that “abandon significance” editorial

Posted on April 5, 2024 by Mayo

.

It was 3 months before I decided to write a blogpost in response to Wasserstein, Schirm and Lazar (2019)’s editorial in The American Statistician in which they recommend that the concept of “statistical significance” be abandoned, hereafter, WSL 2019. (I titled it “Don’t Say What You don’t Mean”.) In that June 17, 2019 blogpost, pasted below, I proposed 3 “friendly amendments” to the language of that document. (There are 97 comments on that post!) The problem is that WSL 2019 presents several of the 6 principles from ASA I (the 2016 ASA statement on Statistical Significance) in a far stronger fashion so as to be inconsistent or at least in tension with some of them. I didn’t think they really meant what they said. I discussed these amendments with Ron Wasserstein, Executive Director of the ASA at the time. Had these friendly amendments been carried out, the document would not have caused as much of a problem, and people might focus more on the positive recommendations it includes about scientific integrity. The proposed ban on a key concept of statistics would still be problematic, resulting in the 2019 ASA President’s Task Force, but it would have helped the document.  At the time, it was still not known whether WSL 2019 was intended as a continuation of the 2016 ASA policy document [ASA I]. That explains why I first referred to WSL 2019 in this blogpost as ASA II. Once it was revealed that it was not official policy at all (many months later), but only the recommendations of the 3 authors, I placed a “note” after each mention of ASA II. But given it caused sufficient confusion as to result in the then ASA president (Karen Kafadar) appointing an ASA Task Force on Statistical Significance and Replicability in 2019 (see here and here), and later, a disclaimer by the authors, in this reblog I refer to it as WSL 2019. You can search this blog for other posts on the 2019 Task Force: their report is here, and the disclaimer here. Continue reading →

Categories: 2016 ASA Statement on P-values, ASA Guide to P-values, ASA Task Force on Significance and Replicability | Leave a comment

5 years ago today, March 20, 2019: the Start of “Abandon Significance”

Posted on March 20, 2024 by Mayo

 

..

Five years ago on this day, a news correspondent at NPR, Richard Harris, published this article, the same day as Wasserstein et al., (2019). Moving to a world beyond “p < 0.05”. TAS, and Amrhein et al., (2019), Comment: Retire statistical significance. Nature. I was one of several people Harris interviewed for his article. He starts by talking of flip-flops regarding the healthfulness of eggs.

Statisticians say it may not be wise to put all your eggs in the significance basket.

A recent study that questioned the healthfulness of eggs raised a perpetual question: Why do studies, as has been the case with health research involving eggs, so often flip-flop from one answer to another? Continue reading →

Categories: stat wars and their casualties, statistical significance | 1 Comment

Preregistration, promises and pitfalls, continued v2

Posted on March 17, 2024 by Mayo

..

In my last post, I sketched some first remarks I would have made had I been able to travel to London to fulfill my invitation to speak at a Royal Society conference, March 4 and 5, 2024, on “the promises and pitfalls of preregistration.” This is a continuation. It’s a welcome consequence of today’s statistical crisis of replication that some social sciences are taking a page from medical trials and calling for preregistration of sampling protocols and full reporting. In 2018, Brian Nosek and others wrote of the “Preregistration Revolution”, as part of open science initiatives. Continue reading →

Categories: Bayesian/frequentist, Likelihood Principle, preregistration, Severity | 3 Comments

Promises and Pitfalls of Preregistration: A Royal Society conference I was to speak at

Posted on March 6, 2024 by Mayo

.

I had been invited to speak at a Royal Society meeting, held March 4 and 5, 2024, on “the promises and pitfalls of preregistration”—a topic in which I’m keenly interested. The meeting was organized by Dr Tom Hardwicke, Professor Marcus Munafò, Dr Sophia Crüwell, Professor Dorothy Bishop FRS FMedSci, and Professor Eric-Jan Wagenmakers. Unfortunately, I was unable to travel to London, so I had to decline attending a few months ago. But, I thought I might jot down some remarks here. Continue reading →

Categories: predesignation | 4 Comments

Happy Birthday R.A. Fisher: “Statistical methods and Scientific Induction” with replies by Neyman and E.S. Pearson

Posted on February 17, 2024 by Mayo

17 Feb 1890-29 July 1962

Today is R.A. Fisher’s birthday! I am reblogging what I call the “Triad”–an exchange between  Fisher, Neyman and Pearson (N-P) published 20 years after the Fisher-Neyman break-up. While my favorite is still the reply by E.S. Pearson, which alone should have shattered Fisher’s allegations that N-P “reinterpret” tests of significance as “some kind of acceptance procedure”, all three are chock full of gems for different reasons. They are short and worth rereading. Neyman’s article pulls back the cover on what is really behind Fisher’s over-the-top polemics, what with Russian 5-year plans and commercialism in the U.S. Not only is Fisher jealous that N-P tests came to overshadow “his” tests, he is furious at Neyman for driving home the fact that Fisher’s fiducial approach had been shown to be inconsistent (by others). The flaw is illustrated by Neyman in his portion of the triad. Details may be found in my book, SIST (2018) especially pp 388-392 linked to here. It speaks to a common fallacy seen every day in interpreting confidence intervals. As for Neyman’s “behaviorism”, Pearson’s last sentence is revealing.

HAPPY BIRTHDAY R.A. FISHER! Continue reading →

Categories: E.S. Pearson, Fisher, Neyman, phil/history of stat | 1 Comment

Conference: Is Philosophy Useful for Science, and/or Vice Versa? (Jan 30- Feb 2, 2024)

Posted on January 31, 2024 by Mayo

I will be giving an online talk on Friday, Feb 2, 4:30-5:45 NYC time, at a conference you can watch on zoom this week (Jan 30-Feb 2): Is Philosophy Useful for Science, and/or Vice Versa?  It’s taking place in-person and online at Chapman University. My talk is: “The importance of philosophy of science for Statistical Science and vice versa”. I’ll touch on a current paper I’m writing that (finally) gets back to “Bayesian conceptions of severity”, (in contrast to error statistical severity) as begun on the post on Van Dongen, Springer, and Wagenmaker (2022). Continue reading →

Categories: Announcement | Leave a comment

Friends of David R. Cox (2022)

Posted on January 10, 2024 by Mayo

$8,765.53 X 2

I want to extend my warmest thanks to all who became Friends of David R. Cox in 2022. Your generous donations to the David R. Cox Foundations of Statistics Award are honoring the contributions of David R. Cox, and promoting the importance of statistical foundations in the American Statistical Association (ASA):

Karim Abadir, Heather Battey, Yoav Benjamini, Stuart Bevan, Alex Blocker, John Bibby, Lynne Billard, Sheila M. Bird, William Browning, John Byrd, Nancy Cartwright, Michael P. Cohen, Noel Cressie, Robert Crouchley, Gary R. Cutter, Anthony C. Davison, Bianca De Stavola, Edgar, Dobriban, Christl Donnelly, Vern Farewell, Samuel Fletcher, David Firth, David Freeborn, Andrew Gelman, David J. Hand, Sylvia Halasz, Frank E. Harrell, Maria-Eglee Perez Hernandez, Klaus Hinkelmann, Michelle Jackson, Patricia A. Jacobs, Harold Jaffe, Bimal Jain, Christiana Kartsonaki, Robert and Loretta Kass, Jesse Krijthe, Daniel Lakens, Ji-Hyun Lee, Lei Liu, Francisco Louzada, Donald Macnaughton, Giovanni M. Marchetti, Kanti V. Mardia, Peter McCullagh, Xiao-Li Meng, Jean Miller, Georges A. A Monette, Pavlos Msaouel, David Oakes, David Oliver, Yusuke Ono, John Park, Jose G. Ramirez, Nancy Reid, James L. Rosenberger, Richard J. Samworth, Stephen J. Senn, Dylan S. Small, David M. Smith, Aris Spanos, Alex Sutherland, John Tomenson, Tengyao Wang, Ronald L. Wasserstein, Gideon Weiss, Manyu Wong, Henry L. Wyneken, Henry Wynn

We exceeded our increased goal (from $5,000-$7,500) last year, raising $8,765.53, all of which is being matched! * Continue reading →

Categories: David R. Cox Foundations of Statistics Award | Leave a comment

Midnight With Birnbaum: Happy New Year 2024!

Posted on December 31, 2023 by Mayo

.

For three of the last four years, it was not feasible to actually revisit that spot in the road, looking to get into a strange-looking taxi, to head to “Midnight With Birnbaum”.  Even last year was iffy. But this year I will, and I’m about to leave at 9pm. (The pic on the left is the only blurry image I have of the club I’m taken to.) My book Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (CUP, 2018)  doesn’t include the argument from my article in Statistical Science (“On the Birnbaum Argument for the Strong Likelihood Principle”), but you can read it at that link along with commentaries by A. P. David, Michael Evans, Martin and Liu, D. A. S. Fraser, Jan Hannig, and Jan Bjornstad. David Cox, who very sadly did in January 2022, is the one who encouraged me to write and publish it. (The first David R. Cox Foundations of Statistics Prize will be awarded at the JSM 2023.) Not only does the (Strong) Likelihood Principle (LP or SLP) remain at the heart of many of the criticisms of Neyman-Pearson (N-P) statistics and of error statistics in general, but a decade after my 2014 paper, it is more central than ever–even if it is often unrecognized.  Continue reading →

Categories: Birnbaum | Leave a comment

A weekend to binge read the (Strong) Likelihood Principle

Posted on December 30, 2023 by Mayo

.

If you read my 2023 paper on Cox’s philosophy of statistics, you’ll have come across Cox’s famous “weighing machine” example, which is thought to have caused “a subtle earthquake” in foundations of statistics. If you’re curious as to why that is, you’ll be interested to know that each year, on New Year’s Eve, I return to the conundrum. This post gives some background, and collects the essential links. Continue reading →

Categories: Likelihood Principle | Leave a comment

Princeton talk: Statistical Inference as Severe Testing: Beyond Performance and Probabilism

Posted on December 27, 2023 by Mayo

On November 14, I gave a talk at the Seminar in Advanced Research Methods for the Department of Psychology, Princeton University.

“Statistical Inference as Severe Testing: Beyond Probabilism and Performance”

The video of my talk is below along with the slides. It reminds me to return to a paper, half-written, replying to a paper on “A Bayesian Perspective on Severity” (van Dongen, Sprenger, Wagenmakers (2022). These authors claim that Bayesians can satisfy severity “regardless of whether the test has been conducted in a severe or less severe fashion”, but what they mean is that data can be much more probable on hypothesis H1 than on H0 –the Bayes factor can be high. However, “severity” can be satisfied in their comparative (subjective) Bayesian sense even for claims that are poorly probed in the error statistical sense (slides 55-6). Share your comments. Continue reading →

Categories: Severity, Severity vs Posterior Probabilities | Leave a comment

1 Year Ago Today: “The Statistics Wars and Their Casualties” workshop

Posted on December 8, 2023 by Mayo
The Statistics Wars and Their Casualties Workshop

 

It’s been 1 year (December 8, 2022) since our workshop, The Statistics Wars and Their Casualties! There were four sessions, held over 4 days. Below are the videos and slides from all four sessions of the Workshop. The first two sessions were held on September 22 & 23, 2022. Session 1 speakers were: Deborah Mayo (Virginia Tech), Richard Morey (Cardiff University), Stephen Senn (Edinburgh, Scotland). Session 2 speakers were:  Daniël Lakens (Eindhoven University of Technology),  Christian Hennig (University of Bologna), Yoav Benjamini (Tel Aviv University).  The last two sessions were held on December 1 and 8. Session 3 speakers were: Daniele Fanelli (London School of Economics and Political Science), Stephan Guttinger (University of Exeter), and David Hand (Imperial College London).  Session 4 speakers were: Jon Williamson (University of Kent),  Margherita Harris  (London School of Economics and Political Science), Aris Spanos (Virginia Tech), and Uri Simonsohn (Esade Ramon Llull University).

Abstracts can be found here and the schedule here. Some participant related publications are on this page. Continue reading →

Categories: Philosophy of Statistics, statistical significance, The Statistics Wars and Their Casualties | Leave a comment

New Paper: “Sir David Cox’s Statistical Philosophy and its Relevance to Today’s Statistical Controversies” (JSM Proceedings)

Posted on October 31, 2023 by Mayo

.

After some wrestling with the Zenodo system of uploading, my paper “Sir David Cox’s Statistical Philosophy and its Relevance to Today’s Statistical Controversies” is now published (open access) in the JSM 2023 Proceedings (link).

Continue reading →

Categories: JSM 2023 proceedings | 3 Comments

Oct 26 Update: If you want to add your name to the list of Friends of Sir David Cox: Matching funds are extended to Dec 1, 2023

Posted on October 24, 2023 by Mayo

..

I’m not comfortable in the role of fundraiser, but I am comfortable in the role of promoting the importance of statistical foundations, and that’s how I see the David R. Cox Foundations of Statistics Award. Thus, I’m sharing the news the ASA sent out yesterday that we’re only around $300 away from our goal, and so the matching period has been extended until Dec 1. (We have $4,698.75 towards to $5,000).  Thus, I’m sharing the new news, updating what the ASA sent out yesterday: The fund is just at $6,000*, but all donations to the award until November 30, midnight (ESA) will still be matched unless $7,500. is reached before that. There is a “thermometer” on the donation page so that donors will know when we have reached that goal, however, on its first day on the job, the thermometer is malfunctioning slightly, failing to include around $500. It should be fixed tomorrow (this is outside my control). For gifts of $50 and above, you will be included in the following list of those recognized as “Friends of David R. Cox”: Continue reading →

Categories: David R. Cox Foundations of Statistics Award | Leave a comment

Philosophy of Scientific Experiment: 40+ years on

Posted on September 26, 2023 by Mayo

.

Some time, around the 1980s, philosophers of science turned their attention to scientific experiments in a way that contrasted with the reigning approaches to philosophy of science. My colleague, Wendy Parker, and I decided to embark on an experiment of our own, aimed at elucidating some central themes of this evolving movement, sometimes referred to as the ‘new experimentalism.’ It was to begin tomorrow, but due to unexpected weather conditions, I’ll be traveling back then, and find myself with an additional afternoon in New York City. So I’ll take this opportunity to begin my reflections, with the expectation of later incorporating Wendy’s insights, and refining my own. The philosophical concepts and ideas stemming from the philosophy of experiment provide powerful tools for making progress on fundamental problems of how we find things out in the face of limitations of data, models, and methods. The time is ripe for a comprehensive examination of this field, but our “experiment on experiment” here will be the bare beginnings of themes that come to mind. Please suggest corrections and additions in the comments. Continue reading →

Categories: experiment, new experimentalism | 7 Comments

Blog at WordPress.com.