A famous chestnut given by Cox (1958) recently came up in conversation. The example “is now usually called the ‘weighing machine example,’ which draws attention to the need for conditioning, at least in certain types of problems” (Reid 1992, p. 582). When I describe it, you’ll find it hard to believe many regard it as causing an earthquake in statistical foundations, unless you’re already steeped in these matters. If half the time I reported my weight from a scale that’s always right, and half the time use a scale that gets it right with probability .5, would you say I’m right with probability ¾? Well, maybe. But suppose you knew that this measurement was made with the scale that’s right with probability .5? The overall error probability is scarcely relevant for giving the warrant of the particular measurement,knowing which scale was used.
The following is an excerpt from Cox and Mayo (2010,295-8):
It had long been thought that the (WCP) entails the (strong) Likelihood Principle (LP) which renders error probabilities irrelevant to parametric inference once the data are known. I give a disproof in Mayo (2010), but later recognized the need for a deeper argument which I gave in Mayo (2014). .If you’re interested, the ink to Statistical Science includes comments by Bjornstad, Dawid, Evans, Fraser, Hannig, and Martin and Liu. You can find quite a lot on the LP searching this blog; it was a main topic for the first few years of this blog.
Cox D. R. and Mayo. D. G. (2010). “Objectivity and Conditionality in Frequentist Inference” in Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability and the Objectivity and Rationality of Science (D Mayo and A. Spanos eds.), Cambridge: Cambridge University Press: 276-304.
Mayo, D. G. (2010). “An Error in the Argument from Conditionality and Sufficiency to the Likelihood Principle” in Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability and the Objectivity and Rationality of Science (D Mayo and A. Spanos eds.), Cambridge: Cambridge University Press: 305-14.
Mayo, D. G. (2014). “On the Birnbaum Argument for the Strong Likelihood Principle,” Paper with discussion and Mayo rejoinder: Statistical Science 29(2) pp. 227-239, 261-266.
Reid, N. (1992). Introduction to Fraser (1966) structural probability and a generalization. In Breakthroughs in Statistics (S. Kotz and N. L. Johnson, eds.) 579–586. Springer Series in Statistics. Springer, New York.
It’s highly relevant to consider what Erich Lehmann says in
Lehmann, E. L. 1993. “The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two?” Journal of the American Statistical Association 88 (424): 1242–1249.
A relevant post with links is here:
“…To summarize, p values, fixed-level significance statements,conditioning, and power considerations can be combined into a unified approach. When long-term power and conditioning are in conflict, specification of the appropriate frame of reference takes priority, because it determines the meaning of the probability statements. A fundamental gap in the theory is the lack of clear principles for selecting the appropriate framework. Additional work in this area will have to come to terms with the fact that the decision in any particular situation must be based not only on abstract principles but also on contextual aspects.”
He opts for “conditioning” over power in scientific contexts, but i think some will take issue as to why the problem is even put this way. The relevant framework should have been considered at the start, some may say, rather than “conditioning” afterward.
But the most notable upshot is, far from this great catastrophic foundational problem, the main spokesperson for N-P seems to be saying, “no big deal”. It depends on your question, problem, and goal.
Click to access lehmann_1993.pdf