Karen Kafadar, Yoav Benjamini, and Donald Macnaughton will be in a session:
Should Science Abandon Statistical Significance?
Friday, Feb 18 from 2-2:45 PM (EST) at the AAAS 2022 annual meeting.
The general program is here. To register*, go to this page.
Synopsis
The concept of statistical significance is central in scientific research. However, the concept is often poorly understood and thus is often unfairly criticized. This presentation includes three independent but overlapping arguments about the usefulness of the concept of statistical significance to reliably detect “effects” in frontline scientific research data. We illustrate the arguments with examples of scientific importance from genomics, physics, and medicine. We explain how the concept of statistical significance provides a cost-efficient objective way to empower scientific research with evidence.
Papers
It’s Not the p-Value’s Fault Yoav Benjamini, Statistics & Operations Research, Tel Aviv University, Tel Aviv, Israel, Israel
I shall review the way the reproducibility and replicability problems in science evolved into the attack on the p-value and statistical significance. I shall argue that the problem is not with the p-value but is that of selective inference which is not adjusted for. In current research practices many indications of the success of a study are possible and selective inference means selecting for highlighting those results that come out to be most promising after viewing the data. Ignoring such selection grossly distorts the properties of all statistical methods, including confidence intervals and Bayesian methods. The use of unadjusted confidence intervals, which is advocated as a replacement to the p-value offers no escape from the problem. I shall further argue that in a well-designed, well executed and properly adjusted p-value is the first defense line against being fooled by randomness, as it requires the minimal set of assumptions.
Significance Tests, p-Values, and Statistical Methods in Scientific Research Karen Kafadar, Statistics, University of Virginia, Charlottesville, VA
The value of hypothesis testing and the frequent misinterpretation of p-values continue to be debated, sometimes productively among statisticians, but often leading to confusion among scientific researchers and study designers who rely on them. In 2020, a Task Force of the American Statistical Association (ASA) prepared a succinct statement about the use of statistical methods in scientific studies, specifically significance tests and p-values, and their connection to replicability. I will discuss some of the benefits as well as the unintended consequences of this discussion, and focus on the Statement from the 2019 ASA President’s Task Force which informs researchers about the use, importance, need, and continued relevance of significance tests in research.
Scientific Journals and Statistical Significance Donald Macnaughton, MatStat Research Consulting, Inc., Toronto, ON, Canada
A scientific journal can use the concept of “statistical significance” as a gateway that a submitted paper must pass through to be considered for publication in the journal. The gateway enables the journal to efficiently and optimally balance the long-run rates of costly false-positive errors and costly false-negative errors in the papers accepted or rejected for consideration for publication. Optimally balancing the error rates helps the journal to maximize the long-run scientific and social benefit of the papers published in the journal.
*There are various fee categories for different groups; no discount for members (!), and I assume this isn’t part of the free public program.
Error Statistics Philosophy 