Evidential Meaning and Methods of Inference
PhD student, History and Philosophy of Science
Master’s student, Statistics
University of Pittsburgh
Bayesian methods conform to the Likelihood Principle, while frequentist methods do not. Thus, proofs of the Likelihood Principle* such as Birnbaum’s (1962) appear to be threats to frequentist positions. Deborah Mayo has recently argued that Birnbaum’s proof is no threat to frequentist positions because it is invalid (Ch. 7(III) in Mayo and Spanos 2010). In my view, Birnbaum’s proof is valid and his premises are intuitively compelling. Nevertheless, I agree with Professor Mayo that the proof, properly understood, does not imply that frequentist methods should not be used.
There are actually at least two different Likelihood Principles: one, which I call the Evidential Likelihood Principle, says that the evidential meaning of an experimental outcome with respect to a set of hypotheses depends only on its likelihood function for those hypothesis (i.e., the function that maps each of those hypotheses to the probability it assigns to that outcome, defined up to a constant of proportionality); the other, which I call the Methodological Likelihood Principle, says that a statistical method should not be used if it can generate different conclusions from outcomes that have the same likelihood function, without a relevant difference in utilities or prior probabilities.
Birnbaum’s proof is a proof of the Evidential Likelihood Principle. It is often taken to show that frequentist methods should not be used, but the Evidential Likelihood Principle does not imply that claim. The Methodological Likelihood Principle does imply that frequentist methods should not be used, but Birnbaum’s proof—at least as originally presented—is not a proof of the Methodological Likelihood Principle.
There are two ways one might respond to this point on behalf of the claim that Birnbaum’s proof does show that frequentist methods should not be used. One is to argue for an additional premise that would allow one to derive the Methodological Likelihood Principle from the Evidential Likelihood Principle. Another is to argue that the point is pedantic because Birnbaum’s proof might as well be recast as a proof of the Methodological Likelihood Principle. I will consider each of these responses in turn.
One could derive the Methodological Likelihood Principle from the Evidential Likelihood Principle by invoking what I call the Evidential Equivalence Norm, which says that a statistical method should not be used if it can generate different conclusions from outcomes that are evidentially equivalent, without a relevant difference in utilities or prior probabilities.
The Evidential Equivalence Norm has intuitive appeal. It seems to say less than Hume’s dictum, “A wise man proportions his belief to the evidence,” which is often regarded as a truism. But is this truism actually true? It seems to me that Humen’s dictum and the Evidential Equivalence Norm are justified only insofar as they help us achieve our epistemic and practical ends. If Birnbaum’s proof is sound, then proportioning one’s belief to the evidence seems to require using Bayesian methods. There are non-Bayesian methods that abide by the Likelihood Principle, but they claim only to characterize data as evidence, not to tell one how to proportion one’s beliefs (Royall 1997). But there are situations in which it is not at all clear that Bayesian methods are the best approach for achieving our epistemic and practical ends. Take, for instance, the search for the Higgs boson. One could try to use the data generated at CERN to update one’s subjective prior probability that the Higgs exists in the reported energy range, but it’s hard to see why we should think that doing so will serve the goal of arriving at an approximately truthlike theory given that we seem to have no reasonable basis at all for choosing our priors in this case. I am inclined to agree with Larry Wasserman that frequentist hypothesis testing is exactly the right tool for this case, despite its shortcomings.
I am of course running roughshod over a number of subtle and important issues. My point is only that the Evidential Equivalence Norm cannot be taken for granted. If norms are justified only insofar as they help us achieve our ends, then pointing out that the Evidential Equivalence Norm appeals to our intuitions is not enough to show that it is justified.
A second way to respond to my claim that Birnbaum’s proof is no threat to frequentist methods because it only establishes the Evidential Likelihood Principle is to claim that it would be unproblematic to recast Birnbaum’s proof as a proof of the Methodological Likelihood Principle. One would simply have to reformulate Birnbaum’s premises (what he calls the Sufficiency and Conditionality Principles) in a methodological vein. Thus, the Sufficiency Principle would become the following:
A statistical method should not be used if it can generate different conclusions from outcomes that give the same value for a sufficient statistic, without a relevant difference in utilities or prior probabilities.
And the Conditionality Principle would become the following:
A statistical method should not be used if it can generate different conclusions from the outcome of a mixture experiment and the corresponding outcome of the component of that mixture experiment that was actually performed.
It is easy to show that these premises do imply the Likelihood Principle. But the same argument I just gave for resisting the Evidential Relevance Norm can be given for resisting these claims: if a norm can be justified only by showing that it helps us achieve desired ends, then the fact that these principles gratify our intuitions is not enough to justify them.
Accepting Birnbaum’s proof of the Likelihood Principle does require frequentists to give up the claim that their methods respect evidential equivalence. It does not require them to give up the claim that their methods are, at least in some cases, the best inferential tools we have.
*Strong Likelihood Principle (From Dec. 6):
If two data sets y’ and y” from experiments E’ and E” respectively, have likelihood functions which are functions of the same parameter(s) µ and are proportional to each other, then y’ and y” should lead to identical inferential conclusions about µ.
Birnbaum, A. (1962). On the foundations of statistical inference. In S. Kotz and N. Johnson (eds), Breakthroughs in statistics, (Vol.1, pp. 478-518). Springer Series in Statistics, New York: Springer-Verlag. Reprinted from Journal of the American Statistical Association, 57, 269–306.
Mayo, D. G. (2010). An error in the argument from conditionality and sufficiency to the likelihood principle. In D. Mayo and A. Spanos (Eds.), Error and inference: Recent exchanges on experimental reasoning, reliability, and the objectivity and rationality of science (pp. 305-314). Cambridge: Cambridge University Press.
Royall, R. (1997), Statistical Evidence: a Likelihood Paradigm, London: Chapman and Hall.