Guest Post (part 2 of 2): Daniël Lakens: “How were we supposed to move beyond  p < .05, and why didn’t we?”

13 thoughts on “Guest Post (part 2 of 2): Daniël Lakens: “How were we supposed to move beyond  p < .05, and why didn’t we?””

1. 

   July 4, 2024
   Mayo

   I am grateful to Daniel Lakens for his thoughtful and illuminating 2-part guest post. I have a lot to say about Part 2, which connects to a talk Lakens recently gave at a conference on Foundations of Applied Statistics, but that would call for a blogpost of its own.

   https://www.youtube.com/watch?v=pOaqDbm38yo

   Here are a few remarks: I totally agree that the authors proposing alternative methods should explain and defend their alternate aims for statistical science, and show how their methodology achieves those ends. It does not suffice to repeat again and again and again the same exact allegations against statistical significance tests and error statistical methods. I don’t know another field or problem where you can keep publishing repetitions of the identical points. (I gave a list in “Farewell Keepsake p. 436- 443) in my book SIST (2018).

   While I strongly endorse Lakens’ idea of putting together teams of statisticians, scientists and philosophers of science (if they are engaged in statistical practice) to weigh in on methodology, the idea of convening a group to reach consensus fills me with terror. Responding to criticisms, recognizing and including diverse perspectives are essential for scientific progress, but that’s not how the current controversies about statistical significance have played out as of late. One need only consider how even the 2016 ASA policy was created, the 2017 conference organized, and this special issue introduced. The worst thing that could happen would be to force agreement, although, unfortunately, to some degree, it is already happening (by dint of the reward structure). That’s why I wrote the editorial “The statistics wars and intellectual conflicts of interest” in Conservation Biology https://conbio.onlinelibrary.wiley.com/doi/10.1111/cobi.13861

   Dissent might well be treated as was the report of the 2019 ASA Task Force on statistical significance and replicability (which opposed abandoning statistical significance). The use of multiple methods, as in the paper Lakens co-authored, is preferable. I’m grateful to him for his post, which I hope will encourage disagreement with the new orthodoxy in (at least some of) statistics.

   Reply
• 

  July 6, 2024
  rkenett

  The slides of the conference mentioned by Mayo are available in https://neaman.org.il/en/On-the-foundations-of-applied-statistics

  Topics discussed there include the role of philosophical concepts in Statistics (Lakens), an historical perspective (Senn), selective inference (Benjamini), DOE for generalizability (Steinberg) and the need for a foundation of applied Statistics (Kenett).

  The argument by Lakens on why Popper was not a Bayesian, as it considered science not to depend on priors, was particularly interesting. It reminded me of Paul Erdos, the famous mathematician, who used to talk about The Book, in which God maintains the perfect proofs of mathematical theorems. See:

  Kenett, R. S., & Redman, T. C. (2019). The real work of data science: turning data into information, better decisions, and stronger organizations. John Wiley & Sons.

  Aigner, M. and Ziegler, G. (2000). Proofs from the Book. Springer.

  Is science about “the book”?

  Reply
2. 

   July 5, 2024
   vaccinelegit0q

   Many people like myself working in the field (e.g. quality control) have already moved beyond p-values. We use methods in the tradition of Shewhart, Deming, Fisher, Taguchi, and Moen. E.g. factorial experimentation, degree of belief, orthogonal arrays,… and it works quite well. In fact, it works tremendously well. Just look at how consistently we make high quality products.

   These methods do not depend upon p-values at all. Or if they do employ them, they are used as Fisher intended, as a guide of inference rather than a tool for selecting a hypothesis or making a decision.

   Why not include a discussion of these approaches?

   Perhaps you mentioned this and I missed it, but the solution seems clear: you get away from p-values by “doing science”, meaning through successful prediction in a variety of contexts, outside the lab, in the real world.

   Perhaps that brings us to the real problem: so much “science” that is done today is done by “proving” effects highly controlled environments (labs), not in the field. Sure, ergodic, closed systems can be tested in the lab (e.g. particle physics, chemistry), but nearly everything at human scale is open / non-ergodic.

   For instance, while we do test new polymers in the lab, that is only part of the experiment. We still have to test it in the field, in real world conditions. This is just as Fisher did in his agricultural experiments. Fisher (AFAIK) never used p-values achieved in the lab to justify a treatment. He relied upon tangible results in the real world: the fertilizer improved crop yields on real farms, or it didn’t.

   Am I missing anything?

   Reply
• 

  July 6, 2024
  rkenett

  this might interest you https://statmodeling.stat.columbia.edu/2017/10/29/quality-control-rather-hypothesis-testing-inference-discovery-better-metaphor-statistical-processes-science/

  Reply
• 

  July 6, 2024
  Mayo

  Let me respond to Kenett’s comment on quality control rather than inference and testing:
  “Quality control” from Deming, Neyman and Pearson and others never went away. It’s what goes on in full-bodied uses of error statistics which combines piece-meal studies, each keeping to error control. It’s quite ironic if today’s critics of statistical significance tests point to “quality control” as opposed to frequentist error statistical methodology, when that is at the heart of its foundations. One of the strongest criticisms of N-P tests and related methods is that they focus on ensuring one will not act or infer an erroneous claim too often in the long run.
  What is it we are controlling when we apply quality control to statistics? We’re controlling claims about data, data models, and links between statistics and what we’re trying to find out about. Inferences are our product! The accusations that we’re not interested in inference and hypothesis testing must be forgetting that the goal of using data to find things out is to INFER claims or solutions to problems that go beyond the data. If people understood that statistical significance tests are a small part of a rich methodology intended to combine local, stringent probes of error, they wouldn’t be charging, as they do, that frequentist error statistics does not use background information. What they mean is that it doesn’t use prior probabilities (unless parameters can be treated as random variables with frequentist distributions), but it systematically and brilliantly uses background information to design and control error and variability at multiple levels from data to substantive claims.

  As Mayo and Cox (2006) say:

  “A fundamental tenet of the conception of inductive learning most at home with the frequentist philosophy is that inductive inference requires building up incisive arguments and inferences by putting together several different piece-meal results; we have set out considerations to guide these pieces. Although the complexity of the issues makes it more difficult to set out neatly, as, for example, one could by imagining that a single algorithm encompasses the whole of inductive inference, the payoff is an account that approaches the kind of arguments that scientists build up in order to obtain reliable knowledge and understanding of a field.”

  Click to access 2006-mayocox-freq-stats-as-a-theory-of-inductive-inference.pdf

  I disagree with Gelman’s allegation about hypothesis testing: “Hypothesis testing, in which the purpose of data collection and analysis is to rule out a null hypothesis (typically, zero effect and zero systematic error) that nobody believes in the first place.” The purpose of rejecting an appropriate test hypothesis (Fisher dubbed them nulls) is to find out if there is a genuine phenomenon. That is NOT the same as inferring there’s some non-systematic error someplace e.g., in measurement procedures. What we absolutely do NOT know ahead of time is whether we have got hold of a genuine phenomenon or real effect. That’s what scientific theories explain and predict, and we only falsify scientific theories with error-prone predictions by dint of rejecting local statistical claims.

  One of the dangers in abandoning statistical significance and the error probing methodology, is that inference—finding things out so as to go beyond the data—is downplayed or replaced with mere description or perhaps claims about degrees of belief. Inference may be risky, but that’s how we discover new things, and we can control the risks in just the way envisioned by Deming and Neyman-Pearson.

  I’m traveling, so this is quick ; but I felt the need to respond right away.

  Reply
• 

  July 6, 2024
  rkenett

  Mayo

  Thanks for your response. Some quick thoughts are listed below.

  Shewhart reverted the statistical approach in the testing of hypothesis. In classical statistics you have data and you fit a model. The data is fixed.

  In Shewhart’s conceptual framework this is reversed. You have an abstract model for a process under control. If the data shows differently, you can stop the process and get it back under control.

  This brings up the difference between process control and surveillance (when the process cannot be reset). This gets us to look at subsequent false alarms (i.e. one after anther). I believe my paper with Moshe Pollak in 1983, listed below, was the first to make this distinction and proposed using the probability of false alarms (PFA). PFA works in surveillance and process control. See

  Kenett, R., & Pollak, M. (1983). On sequential detection of a shift in the probability of a rare event. Journal of the American Statistical Association, 78(382), 389-395.

  Kenett, R. S., & Pollak, M. (2012). On assessing the performance of sequential procedures for detecting a change. Quality and Reliability Engineering International, 28(5), 500-507.

  Reply
3. 

   July 6, 2024
   Peter Monnerjahn

   Lakens:

   “I might be the only person in the world who has read all 43 contributions to this special issue.”

   I’m pretty sure there are a couple of us. 🙂

   As far as I noticed, not a single contribution to the special issue engaged with philosophy of science.

   Bingo! And that’s the major failing of the whole special issue: nobody talks about either the aims of science, what a consensus on a methodology of science might be, or even how the statistical methods commonly used in science relate to any overarching ideas about what it actually is that we (should) do. And nobody seems to have as much as thought about even asking us philosophers of science whether we might have anything worthwhile to add to the debate. It very much looks like there is a (tacit) consensus that we don’t.

   And yet, both high-level concepts like “falsificationism” and relatively low-level logical concepts like “inference” are constantly used in those discussions. Unless one thinks that practitioners simply know better about all of this, it would be decidedly odd not to at least wonder what people think about these things whose actual specialty they are.

   So, as a way of extending an invitation to start a conversation, allow me to ask a question that most of you may find odd (or even perverse) but that I promise holds a key to unlocking the debate about p-values. Here goes: Have you ever heard anyone advance the idea (or thought about that idea yourself) that a single p-value, regardless of any other considerations such as the significance level, is meaningless? And before you answer, maybe consider this: If we somehow were to respond to the substantive question in the affirmative, it would immediately suggest that any dichotomous decisions based on p-values would be illegitimate (in science). (And I might teasingly add that all three Founding Fathers of statistical tests–Fisher, Neyman, and Pearson–explicitly said just that.)

   And one other thing that I think may enrich the debate. I found it rather curious that not a single contribution to the whole replication crisis back-and-forth, neither in the special issue nor thereafter (as far as I can tell), has even mentioned the place of theory in the social sciences. Denny Borsboom had something interesting to say on that 10+ years ago: “Theoretical Amnesia“.

   I’d be very happy to discuss this further with you. 🙂

   Reply
• 

  July 6, 2024
  Mayo

  Peter:
  Thanks for your comment.

  Philosophers of statistics have long been well aware of the foundational and philosophical controversies in statistical methodology. If they’re not engaging with philosophical problems in scientific practice, it is they who are remiss. Philosophy of statistics was ahead of its time in engaging with practitioners and scientists on foundational issues. When I started out, it was common to find philosophers of science and at statistics meetings. I identify 1998 as the turning point, when Colin Howson said he would turn his attention away from statistics to focus on “logic” or probability logic. “Formal epistemology” was born. Despite the fact that philosophers of science in general have been pledging to be relevant to practice, this is not so when it comes to statistical practice, where, ironically, pressing philosophical debates are occurring. Few Bayesian epistemologists consider themselves working on foundational problems in statistical practice: it’s the statisticians who have taken the lead. That’s why my book Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (2018, CUP) is mostly directed to the debates in statistical practice, with the exception of Excursion 2.

  Lack of philosophical skills hampers practitioners just as lack of formal statistical training hampers formal epistemologists (who mostly have different goals).

  Ron Wasserstein invited me to be a “philosophical observer” at the 2015 pow-wow on which the ASA 2016 Statement on Statistical Significance and Replication was based. No other philosophers were involved. The problem isn’t that they haven’t been asked, but rather that very few are trained in statistics. In 2019, I led (with Aris Spanos) a summer seminar for 15 faculty with the goal of providing philosophers a background in statistics and central statistical controversies.
  https://liberalarts.vt.edu/news/articles/2019/10/experts-convene-to-explore-new-philosophy-of-statistics-field.html

  During the pandemic I organized and ran a series of Phil Stat Forums and workshops:
  Phil-stat-wars.com, connected with the LSE. Although philosophers of science are engaged in projects that are relevant to statistical debates, very, very few are actually doing research in the area. I would be glad to hear of some people I’m missing. In my view, unless they wish to stay on the side lines occupying themselves with philosophical puzzles, however interesting, philosophers of science should be immersed in central problems revolving around the nature and role of probability in learning about the world in the face of error.

  The leading dogma, of course, has long been that the only statistical methodology with proper philosophical foundations is Bayesian—in some classical subjectivist sense. But most practicing Bayesians are non-subjective or ‘pragmatic’, which is to say their foundations are messy and not the coherent affair that underlies the supposition that, unlike frequentist error statisticians, they alone are properly philosophical. The search for a neat logic underlies many of the criticisms of statistical significance tests. However, outside of some fields, error probabilities still matter. Although error statistical tools are a hodge-podge, I argue that there is an overarching logic that connects and interrelates them. That was the main theme of my Error and the Growth of Experimental Knowledge (Chicago, 1996, Lakatos Prize), and my work since then.

  I’d be glad to hear more about your work.

  By the way, Benjamini has a paper in Harvard Review of Data Science that gives a chart of the papers in the 2019 special issue.

  Reply
• 

  July 7, 2024
  Christian Hennig

  @Peter Monnerjahn: “Here goes: Have you ever heard anyone advance the idea (or thought about that idea yourself) that a single p-value, regardless of any other considerations such as the significance level, is meaningless?”

  Wouldn’t the answer to that question just be: It may not mean much, but for sure it means more than nothing?

  Reply
4. 

   July 7, 2024
   Roger Jones

   Thank you, Daniel for your posts. Part 2 said what I hoped it would when I read Part 1. The offering of alternatives, lack of an underlying philosophical basis to aspects of the discussion and the different aims and methods of science are three intersecting issues that have been lacking from the debate.

   I’m a working scientist who has had to learn to use different statistical tools over the course of my journey. My undergrad training in stats was totally inadequate. When I got into applied research understanding past climates, quickly learned that there were few established statistical tools for exploring past uncertainties except for Monte Carlo techniques. This became even more evident when projecting future climate and assessing potential risks across a wide range of sectors (e.g., ecology, ag, economics, health). I pioneered a lot of the early risk work in these areas.

   I’m not a mathy type, being more comfortable with sandpits, so when I need to learn and use a new method, I play with it a lot to understand how it works. The contexts I aim to use it in have their own sensitivities, so I want to understand the interplay between those and sensitivities/robustness within the test. This means avoiding black boxes and the latest fashions.

   This approach highlights the importance of model selection, understanding which methods are suitable for which tasks, rather than taking a p-hammer to everything. But quantification of uncertainty is important and should be pursued wherever possible, so no (potential) likelihood should go unexamined.

   Natural and human systems are complex and we analyse them for a wide variety of interactions. This includes detection and attribution, interventions, projections, predictions and causality (if we want to understand how and why a particular phenomenon has emerged). Some of these involve mechanistic representations and others statistical (e.g., excess deaths and extreme weather). The output of mathematical models, such as earth system models, are also analysed statistically.

   In practice, if there’s time series data, someone will unquestioningly put a line through it. Trend is a synonym for change in much of the literature.

   The presence of complex systems highlights the diversity in real-world phenomena and the approaches used to explore them. The idea that there can be a single approach to how we test them, relying on a specific kind of metric, is hard to sustain.

   It is not surprising that many of the efforts to explore replicability expose potential flaws if it’s hard to pin down the actual nature of the studies. This point has been highlighted in previous posts in this series.

   Would it be possible to develop a typology of different families of tests, used for different purposes and from there develop a more systematic understanding of strengths and weaknesses? Individual practitioners with wide experience do this for themselves but efforts to impose a blanket approach have failed.

   Methods in practice range from being highly developed to totally exploratory. Pre-registration and strict guidelines (e.g., recipes and standards) are suitable for the former but not the latter.

   For exploratory work, or situations with competing processes and explanations is where I would argue methods such as severe testing is most needed. But would need bespoke approaches that are not well understood and, furthermore, will be rejected by those who favour some kind of orthodox approach.

   Tests that combine scientific and statistical hypotheses will depend on the subject matter. To this end, there are many text books that address these needs, but they are largely aimed at undergraduates and practitioners of established methods.

   Personally, I’m not overly concerned about replicability. Cutting edge research is hard to replicate, so overburdening it with rules that emphasise orthodoxy at the cost of innovation will be counter-productive.

   What I am concerned about is gate-keeping. We are in a situation where massive amounts of literature are being produced each year and merit is assessed on the capacity to produce more. The low-quality end of research will continue to increase. Paper mills will keep springing up, leading to whack-a-mole tactics as they do.

   Research governing bodies have to decide which matters more. Headlines featuring crises always get attention and research replicability has been right up there. This raises concerns and nervousness about science’s reputation. But if we look around, science is being attacked constantly – the problem is not topical, it’s structural. If it wasn’t replicability, it would be something else.

   Although the failure of replicability gets headlines, the role of science publishers who generate those headlines, while joining in the general wailing and gnashing of teeth, also needs to be examined. For an industry that gets similar rates of return to arms and drug dealers, while drawing on free labour, this is far too cute. They need to invest in better quality research. This doesn’t mean doubling down the current inequalities that disenfranchise the global south, but the opposite.

   These issues will not be addressed from within individual silos. The idea of putting together interdisciplinary teams to work through some of these issues in detail could be really beneficial. For example, building a library of exemplars is specific areas could be useful. Leaning on publishers to invest in this for their own best interests (long term) could also be tried.

   Science and research generally, needs to emphasise discovery and excellence, and encourage as many as possible to do so. It can’t be gatekeeper and (p)-police. That’s a recipe for mediocrity and things are bad enough as it is. (apologies for the soapbox)

   Reply
5. 

   July 7, 2024
   vaccinelegit0q

   In 1990 Deming wrote the following:

   Unfortunately, the statistical methods in textbooks and in the classroom do not tell the student that the problem in the use of data is prediction. What the student learns is how to calculate a variety of tests (t- test, F-test, chi-square, goodness of fit, etc.) in order to announce that the difference between the two methods or treatments is either significant or not significant. Unfortunately, such calculations are a mere formality. Significance or the lack of it provides no degree of belief—high, moderate, or low—about prediction of performance in the future, which is the only reason to carry out the comparison, test, or experiment in the first place.

   The theory behind this statement primary is because:

   1. A test of significance is only useful / relevant when studying “dead” things or processes in equilibrium / maximum entropy.
   2. When dealing with non-equilibrium / fat-tailed (non-ergodic) systems, a test of significance does not preserve the structure or order of the data because it is a symmetric function of the data. Thus, changes in the structure of the data observed have no effect upon the first, second, or fourth moments.

   For instance, it is only appropriate to apply Neyman and Pearson procedure on populations with fixed boundary conditions (e.g. to determine if you should send the 1000 radios you just made to market, or do a recall). However, it is inappropriate to use this procedure when studying whether a change to a manufacturing process is desirable or not.

   Deming also made this point in his papers On Probability as a Basis For Action and On the Distinction Between Enumerative and Analytic Surveys. And is why Shewhart said that such methods could only be used when a process is “under statistical control”

   In fact the recent Covid-19 vaccine debacle illustrates how statistics is misused / abused. When a vaccine is introduced to the body, its complete effects cannot be known in advance (i.e. are computationally irreducible). You have to run the whole process (let someone live with the vaccine for decades, have children, etc) to determine whether it’s “safe” or not. A good example is Thalidomide. When given to a pregnant woman, its effects can only be known after the fetus-maturity process executes. It was impossible to know the outcomes of taking the medication in advance because the boundary conditions of the sample space have not yet emerged.

   So, back to test of significance, there seems to me too little education on the basics of statistical theory and whether not data from an experiment meet the assumptions required – as Deming pointed out.

   Perhaps this is one reason why this debate seems to go nowhere.

   Reply
6. 

   July 9, 2024
   rkenett

   “Tests of variables that affect a process are useful only if they predict what will happen if this or that variable is increased or decreased. Statistical theory, as taught in the books, is valid and leads to operationally verifiable tests and criteria for an enumerative study. Not so with an analytic problem, as the conditions of the experiment will not be duplicated in the next  trial. Unfortunately, most problems in industry are analytic.”

   W. E. Deming, from preface to The Economic Control of  Quality of  manufactured product by W. Shewhart, 1931

   The enumerative/analytic dichotomy is key. Handling analytic problems implies generalizability of findings. In AI/ML/DL this is achieved with a split between training and validation data. A first step in addressing generalizability is developing methods representation of findings. A step in this direction is proposed in: Kenett, R. S., & Rubinstein, A. (2021). Generalizing research findings for enhanced reproducibility: an approach based on verbal alternative representations. Scientometrics, 126(5), 4137-4151. https://papers.ssrn.com/sol3/Papers.cfm?abstract_id=3035070

   Deming indicates the limited generalizability of enumerative studied to a current population frame. He was an expert in sampling methods aiming at this. He also indicates that analytic studies are most important. Generalizability in this context is an open challenge. For COVID related aspects of this, see Serio, C. D., Malgaroli, A., Ferrari, P., & Kenett, R. S. (2022). The reproducibility of COVID-19 data analysis: paradoxes, pitfalls, and future challenges. PNAS nexus, 1(3), pgac125 and Kenett, R. S., Manzi, G., Rapaport, C., & Salini, S. (2022). Integrated analysis of behavioural and health COVID-19 data combining Bayesian networks and structural equation models. International Journal of Environmental Research and Public Health, 19(8), 4859.

   Reply
7. 

   August 2, 2024
   Christopher Tong

   I was made aware of this series of discussions via an unsolicited email from an associate of Prof. Mayo that I received earlier this week. I would like to clarify Prof. Lakens’ comment above, regarding my paper (Tong, 2019) in the TAS special issue. He correctly notes that my paper belongs in the category of not offering alternatives to p-values. He should have added that I argued that most of the time there should be no such alternatives, because most of science is exploratory, not confirmatory, where model building and model criticism are actively taking place. At this stage statistical inference is premature, due to model uncertainty (which isn’t quantified by inferential statistics, whether they be p-values, confidence intervals, Bayes factors, likelihood ratios, or whatever). In exploratory research, the data must be allowed to influence the form of the model, and the boundary between fitting and overfitting cannot be defined too sharply. The examination of many sets of data, as part of the model/experiment iteration, can play a key role in recognizing and managing overfitting, though other precautions should also be taken (details in my paper). Only at the later stages of research, when model specification before data collection can finally be contemplated, could statistical inference be done under the conditions where various putative properties (e.g., unbiasedness, coverage probabilities, Type 1 and Type 2 error rates, etc.) are reasonably approximate. Meanwhile Statistics as a discipline should prioritize teaching good study design and execution, disciplined data exploration, and statistical thinking, which apply throughout all stages of research, unlike statistical inference, which has a very narrow scope.

   Pedantically one might still ask: for a confirmatory study, what are alternatives to p-values? My paper did not address this, as I correctly assumed that many other authors in the special issue would focus solely on this point. I don’t pretend to be smarter than they are. I only recommended that “Methods with alleged generality, such as the p-value or Bayes factor, should be avoided in favor of discipline- and problem-specific solutions that can be designed to be fit for purpose.” To provide a concrete example, here I offer the case study of vaccine efficacy trials. Consider the FDA’s original 2020 Guidance for Industry on “Development and Licensure of Vaccines to Prevent COVID-19”. The efficacy portion of the guidance specified the agency’s expectations for vaccine efficacy: “a point estimate for a placebo-controlled efficacy trial of at least 50%, with a lower bound of the appropriately alpha-adjusted confidence interval around the primary efficacy endpoint estimate of > 30%.” No p-values were mentioned anywhere in this document. I know of five major pivotal efficacy studies for COVID-19 vaccines published in the New England Journal of Medicine in 2020-2021, subsequent to the issuance of the FDA guidance: Pfizer/BioNTech, Moderna/NIAID, Janssen (Johnson & Johnson), AstraZeneca/Oxford, and Novavax. All five included estimates of vaccine efficacy (based on 1 – the incidence rate ratio, estimated using different statistical models, such as Poisson regression or the Cox proportional hazards model) with either 95% confidence or 95% credible intervals. Only two of them (Moderna and AstraZeneca) also included p-values in the main NEJM papers (I did not search the supplementary files). To emphasize my earlier point, pivotal studies occur at the end of vaccine development, after an extensive (and in this case accelerated) series of preclinical and clinical studies, and when comprehensive prespecification of the trial protocol (including statistical analysis plan) is expected.

   This is not an isolated example in vaccine clinical trials. See for example the FDA guidance on seasonal flu vaccines, or the book of Halloran et al., Design and Analysis of Vaccine Studies (Springer, 2010). In Sec. 6.2, Halloran et al. discuss seven real-world vaccine trials. In each case, interval estimates of vaccine efficacy are provided; p-values are also reported in only two of these seven. I conjecture that many investigators in the vaccine clinical trial arena have been comfortable with interval estimation, with p-values as optional at best, for a very long time. The ASA statements and special issue had absolutely no influence on such long-standing attitudes, because (in this particular arena of medical research) we were already living in “a world beyond p < 0.05″. An exercise for the reader (I have not done this myself) would be to examine the clinical efficacy sections (Sec. 14) of the package inserts for all licensed vaccines in the U.S., and tabulate how many provide one or more of the following: interval estimates of vaccine efficacy, p-values, or other inferential statistics. Here is where you can find them: https://www.fda.gov/vaccines-blood-biologics/vaccines/vaccines-licensed-use-united-states

   The views expressed are my own. (I am aware of several errors in my 2019 paper, but I stand by the overall spirit and major themes of the paper.)

   References

   The original 2020 FDA guidance for COVID-19 vaccines may no longer be extant; however the updated version does quote the original, the relevant part of which I quoted above. It may be found here: https://www.fda.gov/media/142749/download

   FDA guidance on seasonal flu vaccines: https://www.fda.gov/files/vaccines%2C%20blood%20%26%20biologics/published/Guidance-for-Industry–Clinical-Data-Needed-to-Support-the-Licensure-of-Seasonal-Inactivated-Influenza-Vaccines.pdf

   The NEJM papers are as follows:

   Pfizer/BioNTech: DOI: 10.1056/NEJMoa2034577
   Moderna/NIAID: DOI: 10.1056/NEJMoa2035389
   Janssen: DOI: 10.1056/NEJMoa2101544
   AstraZeneca/Oxford: DOI: 10.1056/NEJMoa2105290
   Novavax: DOI: 10.1056/NEJMoa2107659

   Reply