Spot the fallacy!
- The power of a test is the probability of correctly rejecting the null hypothesis. Write it as 1 – β.
- So, the probability of incorrectly rejecting the null hypothesis is β.
- But the probability of incorrectly rejecting the null is α (the type 1 error probability).
So α = β.
I’ve actually seen this, and variants on it [i].
 Although they didn’t go so far as to reach the final, shocking, deduction.
If I were to guess where this came from, I would say Ziliak. Beta is the probability of failing to reject the null when the null is false (type II error), not the probability of incorrectly rejecting the null (type I), or else I am my own grandpa….
The mischief that Ziliak has done in the legal system through the Matrixx Initiatives case is substantial. Some parties and their expert witnesses (including Sander Greenland in a recent report) now misrepresent the statements from the case as though they were holdings of the Supreme Court. As I have explained before, the statements were necessarily dicta (and thus non-binding) because once the Court held that causation was not necessary for materiality of the undisclosed information, then anything that might, or might not, be necessary for causation was no longer relevant to the Court’s consideration. And yet the Court plowed on, and stepped in the mule poop. The Court cited three cases for its dictum, two which were so-called differential etiology cases (ruling in by ruling out specific causes by process of elimination), which had nothing to do with statistical significance. The third case was Wells v. Ortho Pharmaceuticals, in which plaintiffs’ expert witnesses did have some statistically significant studies, but the problem was that the studies were plagued by bias and confounding. Wells is one of the most discredited legal decisions in the federal system, even though it was a case tried to the judge (not a jury).
Nate: Thanks for your comment. It’s really too bad the Supreme Court has “stepped into the mule poop”. You should send me updates on this issue for posting.
But I think we must distinguish between what’s all wrong about the logic in that case, as you showed in your guest post:
Interstitial Doubts about the Matrixx
and as I argued in:
Distortions in the Court (Mayo)
as opposed to common abuses of power and related error probabilities. (As you also note, Ziliac and McCloskey go further into mule territory in interpreting error probabilities as posterior probabilities in their brief to the Court).
So back to my specific power howler: Of course you are correct that beta (in a test against H’) is “the probability of failing to reject the null when the null is false (type II error)” and alternative H’ is true. (Note the addition I made to your definition.) But if you start with the ambiguous premise #1, you can see how it could happen that you land in erroneous line #2.
The power of a test is the probability of rejecting the null hypothesis when the alternative H’ is true (in the sense of adequately describing the data generation).
It is NOT the probability of correctly rejecting the null—in and of itself–unless this is qualified to give the correct definition.
Premise #1 should be: POW(H’) = Pr(test T rejects Ho; H’). Then it’s clear that the correct complement is
1 – POW(H’) = Pr(test T does not reject Ho; H’)–as you observe.
I have numerous posts on power (look up Neyman for some, Dierdre for others) on this blog.
Someone just twittered me this disaster of misinterpretations of P-values: http://effectivehealthcare.ahrq.gov/index.cfm/glossary-of-terms/?pageaction=showterm&termid=67