I’m really glad to see that the Normal Deviate has posted about the error in taking the p-value as any kind of conditional probability. I consider the “second” misunderstanding to be the (indirect) culprit behind the “first”.
March 14, 2013 – 7:57 pm
It’s been said a million times and in a million places that a p-value is not the probability of H0 given the data.
But there is a different type of confusion about p-values. This issue arose in a discussion on Andrew’s blog.
Andrew criticizes the New York times for giving a poor description of the meaning of p-values. Of course, I agree with him that being precise about these things is important. But, in reading the comments on Andrew’s blog, it occurred to me that there is often a double misunderstanding.
First, let me say that I am neither defending nor criticizing p-values in this post. I am just going to point out that there are really two misunderstandings floating around.
(1) The p-value is not the probability of H0 given the data.
(2) But neither is the p-value the probability of something conditional on H0 .
Deborah Mayo pointed this fact out in the discussion on Andrew’s blog (as did a few other people).
When we use p-values we are in frequentist-land. H0 (the null hypothesis) is not a random variable. It makes no sense to talk about the posterior probability of H0 . But it also makes no sense to talk about conditioning on H0 . You can only condition on things that were random in the first place.
You can read the rest of his post (with symbols intact) here.