Prionvac: Our experiments yield a statistically significant increase in survival among scrapie-infected mice who are given our new vaccine (p = .01) compared to infected mice who are treated with a placebo. The data indicate H: an increased survival time of 9 months, compared to untreated mice.*
Reformer: You are exaggerating what your data show. In fact, there is a fairly high probability, more than .5, that your study would produce a p = .01 difference, even if the actual increased survival were only 1 month! (That is, the power to reject the null and infer H: increase of 9 months, is more than .5.)
Prionvac: Well, then, the data indicate an increased survival of 2 years.
Reformer: Now that is a more reasonable interpretation of your result!
J: the increased survival time in vaccinated mice is at least 2 years.
H: the increased survival in vaccinated mice is at least 6 months.
The “reformer” (rightly) denies that the statistically significant data are evidence for H. But then goes on to accept that the same data are good evidence for J! This makes no sense. (Note that J entails H!)
This is what would be condoned, nay, recommended, if a rejection, at a fixed level (e.g., 01) were taken as better evidence for an increased survival, the greater the power the test has for detecting that increase, were it to exist.
Is it possible that some of the leaders of the New Reform movement are advocating that tests be construed this way? Stay tuned….
[*The null hypothesis might be Ho: no increased survival, or it might be no difference in survival among treated and non-treated mice. I leave this open since the reformers do; but nothing turns on this.]
I don’t think you’re going to reach many statisticians writing like this.
Do you mean; rates of 6-month survival differed significantly between vaccine and placebo groups? All fine, but there’s no reason to believe the 2-year survival rates would differ in anything like the same way.
If you mean that the difference in mean lifetimes of the vaccine and placebo group was 6 months (p=.01, whatever) then Prionvac’s subsequent statement about a difference of 2 years is nonsense.
Or are there some missing “at least”s? What’s with the 6 month/9 month difference? And I do hope “stay tuned” is going to discuss Schervish (1996, Am Stat 50:3 203-206)