Part 1: Imaginary scientist at an imaginary company, Prionvac, and an imaginary Reformer

Prionvac: Our experiments yield a statistically significant increase in survival among scrapie-infected mice who are given our new vaccine (p = .01) compared to infected mice who are treated with a placebo. The data indicate H: an increased survival time of 9 months, compared to untreated mice.*

Reformer: You are exaggerating what your data show. In fact, there is a fairly high probability, more than .5, that your study would produce a p = .01 difference, even if the actual increased survival were only 1 month! (That is, the power to reject the null and infer H: increase of 9 months, is more than .5.)

Prionvac: Well, then, the data indicate an increased survival of 2 years.

Reformer: Now that is a more reasonable interpretation of your result!

We have:
J: the increased survival time in vaccinated mice is at least 2 years.
H: the increased survival in vaccinated mice is at least 6 months.

The “reformer” (rightly) denies that the statistically significant data are evidence for H. But then goes on to accept that the same data are good evidence for J! This makes no sense. (Note that J entails H!)

This is what would be condoned, nay, recommended, if a rejection, at a fixed level (e.g., 01) were taken as better evidence for an increased survival, the greater the power the test has for detecting that increase, were it to exist.

Is it possible that some of the leaders of the New Reform movement are advocating that tests be construed this way? Stay tuned….

[*The null hypothesis might be Ho: no increased survival, or it might be no difference in survival among treated and non-treated mice. I leave this open since the reformers do; but nothing turns on this.]