Excerpt from Excursion 4 Tour II*
4.4 Do P-Values Exaggerate the Evidence?
“Significance levels overstate the evidence against the null hypothesis,” is a line you may often hear. Your first question is:
What do you mean by overstating the evidence against a hypothesis?
Several (honest) answers are possible. Here is one possibility:
What I mean is that when I put a lump of prior weight π0 of 1/2 on a point null H0 (or a very small interval around it), the P-value is smaller than my Bayesian posterior probability on H0.
More generally, the “P-values exaggerate” criticism typically boils down to showing that if inference is appraised via one of the probabilisms – Bayesian posteriors, Bayes factors, or likelihood ratios – the evidence against the null (or against the null and in favor of some alternative) isn’t as big as 1 − P. Continue reading


















Excursion 3 Statistical Tests and Scientific Inference





