Equivocations between informal and formal uses of “probability” (as well as “likelihood” and “confidence”) are responsible for much confusion in statistical foundations, as is remarked in a famous paper I was rereading today by Allan Birnbaum:
“It is of course common nontechnical usage to call any proposition probable or likely if it is supported by strong evidence of some kind. .. However such usage is to be avoided as misleading in this problem-area, because each of the terms probability, likelihood and confidence coefficient is given a distinct mathematical and extramathematical usage.” (1969, 139 Note 4).
For my part, I find that I never use probabilities to express degrees of evidence (either in mathematical or extramathematical uses), but I realize others might. Even so, I agree with Birnbaum “that such usage is to be avoided as misleading in” foundational discussions of evidence. We know, infer, accept, and detach from evidence, all kinds of claims without any inclination to add an additional quantity such as a degree of probability or belief arrived at via, and obeying, the formal probability calculus.
It is interesting, as a little exercise, to examine scientific descriptions of the state of knowledge in a field. A few days ago, I posted something from Weinberg on the Higgs particle. Here are some statements, with some terms emphasized:
The general features of the electroweak theory have been well tested; their validity is not what has been at stake in the recent experiments at CERN and Fermilab, and would not be seriously in doubt even if no Higgs particle had been discovered.
I see no suggestion of a formal application of Bayesian probability notions.
But one of the consequences of the electroweak symmetry is that, if nothing new is added to the theory, all elementary particles, including electrons and quarks, would be massless, which of course they are not.
So, something has to be added to the electroweak theory, some new kind of matter or field, not yet observed in nature or in our laboratories. The search for the Higgs particle has been a search for the answer to the question: What is this new stuff we need?
Without pretending anything like a full understanding of the theoretical claims here, they are claims about empirical and theoretical knowledge, and about questions we want to find out about. No degree of probability assignments. Philosophers like to “rationally reconstruct” scientific talk in terms of their philosophies, and “degrees of probability” and “degrees of confirmation” have in the past often been invoked. I think it makes for an inadequate reconstruction, but that is beside the point just now. Statisticians are not trying to rationally reconstruct inference and learning in extramathematical terms. They must be talking mathematically, not extramathematically. And one sees no such mathematical probability in these theoretical and observational claims.
It has been known since the work of Yoichiro Nambu and Jeffrey Goldstone in 1960–1961 that symmetry-breaking of this sort is possible in various theories, but it had seemed that it would, as a matter of theory, necessarily entail new massless particles, which we knew did not exist in fact.
Again, we know things, based on a combination of theoretical and experimental evidence and constraints.
…. But this still left open the question: What sort of new matter or field is actually breaking the electroweak symmetry?
There were two possibilities. One possibility was that there are hitherto unobserved fields …. These are called “scalar” fields, meaning that unlike magnetic fields they do not have directions in ordinary space. …
These particles were discovered at CERN in 1983–1984, and found to have the masses predicted by our electroweak theory.
No claim about posterior probabilities.
One of the scalar fields was left over to be manifested as a physical particle, a bundle of the energy and momentum of this field. This is the “Higgs particle” for which physicists have been searching for nearly thirty years.
….The discovery of the new particle certainly casts a very strong vote in favor of the electroweak symmetry being broken by scalar fields, rather than by technicolor forces. This is why the discovery is important.
Perhaps likelihoodists will see this as a claim about comparative likelihoods. But this was not the statistical analysis actually used in obtaining the discovery. And to view “the strong vote” as a comparative probability assignment, would be an extramathematical use of probability.
But there is one explicit use of probability:
The electroweak theory of 1967–1968 predicted all of the properties of the Higgs particle, except its mass. With the mass now known experimentally, we can calculate the probabilities for all the various ways that Higgs particles can decay, and see if these predictions are borne out by further experiment. This will take a while.
These are probabilities of events, and checking if these “probabilistic predictions are borne out by further experiment” will involve some kind of statistical test. Some statisticians might claim to hold a philosophy where science only involves observable events, and deny out of hand any knowledge of theoretical properties, forces or the like. I assume they have a philosophical argument for this position;it will not suffice that it squares with a particular monist Bayesian philosophy.
The discovery of a new particle that appears to be the Higgs also leaves theorists with a difficult task: to understand its mass.
The goal of understanding is quite different from being able to assign a high probability to anything, or for that matter, from being able to predict anything.
But it may be objected that when Weinberg says “we know” this or that claim, he does not claim infallible knowledge. True, but it does not follow that what he means is that we assign these claims a high degree of mathematical probability. On the other hand, there are all kinds of qualifications about the limits of knowledge at any stage, about aspects not well-checked-so-far, and bounds on precision with which various claims are known. This is the kind of knowledge we actually obtain. My view is that a philosophy of science should be interested in explaining, and showing how to get more of, the kind of knowledge we actually obtain.
A. Birnbaum (1969), “Concepts of Statistical Evidence,” in Philosophy, Science, and Method (eds., Morgenbesser, Suppes, and White), NY: St. Martins, 112-143.
I guess like O’Hagan you are being deliberately provocative to stimulate discussion….
“Some statisticians might claim to hold a philosophy where science only involves observable events, and deny out of hand any knowledge of theoretical properties, forces or the like. I assume they have a philosophical argument for this position, it will not suffice that it squares with a particular monist Bayesian philosophy.”
I had to look up “monist” and can’t see why you would use that word (except for provocation) if you want some isms you could try positivism, instumentalism or in particular operationalism. This idea is close to the Copenhagen intepretation the most common interpretation in quantum mechanics.
Physics may be the area in which talking about an external objective truth might pose the least problems. In problems where I have worked in astronomy, geography and signal processing talking about an external truth that you will discover doesn’t make sense except in contrived simulation experiments. In my opinion “physics envy” at times drives fields in the wrong direction by attempting find an objective truth in situations where this is a meaningless idea.
Also FWIW in cosmology in particular determining the cosmological parameters Bayesian methods of analysis are used almost exclusively.
… also …
“The electroweak theory of 1967–1968 predicted all of the properties of the Higgs particle, except its mass. With the mass now known experimentally, we can calculate the probabilities for all the various ways that Higgs particles can decay, and see if these predictions are borne out by further experiment. This will take a while.”
this seems perfectly consistent with
p(observations related to Higgs decays | observations related to mass)
… also …
I do agree with your last paragraph, traditional Bayesianism can be so tied up with optimality and require heroic levels of probability specifications that its connections with real problems and real science sometimes seems distant. Also if the scientists in the field do not want to talk about probabilities of hypothesis (or if they do) I would not want to argue with them. On the other hand, if you are interested in prediction or decision making under uncertainty the theory drives you very strongly in a subjective Bayesian direction (again a fully specified analysis is most likely unworkable – due to issues of scale). I appreciate your main area of interest is finding true theories, but what is the goal of finding the truth if not for reducing uncertainty in decision making problems?