Mayo mirror
Equivocations between informal and formal uses of “probability” (as well as “likelihood” and “confidence”) are responsible for much confusion in statistical foundations, as is remarked in a famous paper I was rereading today by Allan Birnbaum:
“It is of course common nontechnical usage to call any proposition probable or likely if it is supported by strong evidence of some kind. .. However such usage is to be avoided as misleading in this problem-area, because each of the terms probability, likelihood and confidence coefficient is given a distinct mathematical and extramathematical usage.” (1969, 139 Note 4).
For my part, I find that I never use probabilities to express degrees of evidence (either in mathematical or extramathematical uses), but I realize others might. Even so, I agree with Birnbaum “that such usage is to be avoided as misleading in” foundational discussions of evidence. We know, infer, accept, and detach from evidence, all kinds of claims without any inclination to add an additional quantity such as a degree of probability or belief arrived at via, and obeying, the formal probability calculus.
It is interesting, as a little exercise, to examine scientific descriptions of the state of knowledge in a field. A few days ago, I posted something from Weinberg on the Higgs particle. Here are some statements, with some terms emphasized:
The general features of the electroweak theory have been well tested; their validity is not what has been at stake in the recent experiments at CERN and Fermilab, and would not be seriously in doubt even if no Higgs particle had been discovered.
I see no suggestion of a formal application of Bayesian probability notions.
But one of the consequences of the electroweak symmetry is that, if nothing new is added to the theory, all elementary particles, including electrons and quarks, would be massless, which of course they are not.
So, something has to be added to the electroweak theory, some new kind of matter or field, not yet observed in nature or in our laboratories. The search for the Higgs particle has been a search for the answer to the question: What is this new stuff we need?
Without pretending anything like a full understanding of the theoretical claims here, they are claims about empirical and theoretical knowledge, and about questions we want to find out about. No degree of probability assignments. Philosophers like to “rationally reconstruct” scientific talk in terms of their philosophies, and “degrees of probability” and “degrees of confirmation” have in the past often been invoked. I think it makes for an inadequate reconstruction, but that is beside the point just now. Statisticians are not trying to rationally reconstruct inference and learning in extramathematical terms. They must be talking mathematically, not extramathematically. And one sees no such mathematical probability in these theoretical and observational claims.
It has been known since the work of Yoichiro Nambu and Jeffrey Goldstone in 1960–1961 that symmetry-breaking of this sort is possible in various theories, but it had seemed that it would, as a matter of theory, necessarily entail new massless particles, which we knew did not exist in fact.
Again, we know things, based on a combination of theoretical and experimental evidence and constraints.
…. But this still left open the question: What sort of new matter or field is actually breaking the electroweak symmetry?
There were two possibilities. One possibility was that there are hitherto unobserved fields …. These are called “scalar” fields, meaning that unlike magnetic fields they do not have directions in ordinary space. …
These particles were discovered at CERN in 1983–1984, and found to have the masses predicted by our electroweak theory.
No claim about posterior probabilities.
One of the scalar fields was left over to be manifested as a physical particle, a bundle of the energy and momentum of this field. This is the “Higgs particle” for which physicists have been searching for nearly thirty years.
….The discovery of the new particle certainly casts a very strong vote in favor of the electroweak symmetry being broken by scalar fields, rather than by technicolor forces. This is why the discovery is important.
Perhaps likelihoodists will see this as a claim about comparative likelihoods. But this was not the statistical analysis actually used in obtaining the discovery. And to view “the strong vote” as a comparative probability assignment, would be an extramathematical use of probability.
But there is one explicit use of probability:
The electroweak theory of 1967–1968 predicted all of the properties of the Higgs particle, except its mass. With the mass now known experimentally, we can calculate the probabilities for all the various ways that Higgs particles can decay, and see if these predictions are borne out by further experiment. This will take a while.
These are probabilities of events, and checking if these “probabilistic predictions are borne out by further experiment” will involve some kind of statistical test. Some statisticians might claim to hold a philosophy where science only involves observable events, and deny out of hand any knowledge of theoretical properties, forces or the like. I assume they have a philosophical argument for this position;it will not suffice that it squares with a particular monist Bayesian philosophy.
The discovery of a new particle that appears to be the Higgs also leaves theorists with a difficult task: to understand its mass.
The goal of understanding is quite different from being able to assign a high probability to anything, or for that matter, from being able to predict anything.
But it may be objected that when Weinberg says “we know” this or that claim, he does not claim infallible knowledge. True, but it does not follow that what he means is that we assign these claims a high degree of mathematical probability. On the other hand, there are all kinds of qualifications about the limits of knowledge at any stage, about aspects not well-checked-so-far, and bounds on precision with which various claims are known. This is the kind of knowledge we actually obtain. My view is that a philosophy of science should be interested in explaining, and showing how to get more of, the kind of knowledge we actually obtain.
A. Birnbaum (1969), “Concepts of Statistical Evidence,” in Philosophy, Science, and Method (eds., Morgenbesser, Suppes, and White), NY: St. Martins, 112-143.
Error Statistics Philosophy 
I guess like O’Hagan you are being deliberately provocative to stimulate discussion….
“Some statisticians might claim to hold a philosophy where science only involves observable events, and deny out of hand any knowledge of theoretical properties, forces or the like. I assume they have a philosophical argument for this position, it will not suffice that it squares with a particular monist Bayesian philosophy.”
I had to look up “monist” and can’t see why you would use that word (except for provocation) if you want some isms you could try positivism, instumentalism or in particular operationalism. This idea is close to the Copenhagen intepretation the most common interpretation in quantum mechanics.
Physics may be the area in which talking about an external objective truth might pose the least problems. In problems where I have worked in astronomy, geography and signal processing talking about an external truth that you will discover doesn’t make sense except in contrived simulation experiments. In my opinion “physics envy” at times drives fields in the wrong direction by attempting find an objective truth in situations where this is a meaningless idea.
Also FWIW in cosmology in particular determining the cosmological parameters Bayesian methods of analysis are used almost exclusively.
… also …
“The electroweak theory of 1967–1968 predicted all of the properties of the Higgs particle, except its mass. With the mass now known experimentally, we can calculate the probabilities for all the various ways that Higgs particles can decay, and see if these predictions are borne out by further experiment. This will take a while.”
this seems perfectly consistent with
p(observations related to Higgs decays | observations related to mass)
… also …
I do agree with your last paragraph, traditional Bayesianism can be so tied up with optimality and require heroic levels of probability specifications that its connections with real problems and real science sometimes seems distant. Also if the scientists in the field do not want to talk about probabilities of hypothesis (or if they do) I would not want to argue with them. On the other hand, if you are interested in prediction or decision making under uncertainty the theory drives you very strongly in a subjective Bayesian direction (again a fully specified analysis is most likely unworkable – due to issues of scale). I appreciate your main area of interest is finding true theories, but what is the goal of finding the truth if not for reducing uncertainty in decision making problems?
David: I don’t think I’m being like O’Hagan. The reason for mentioning “monism” in relation to Bayesians is that most Bayesians are dualists and do not deny we can learn about aspects of a real world including theoretical entities and processes (as opposed to countenancing only degrees of belief). But aren’t you glad you were led to look up a new word?
You say “if you are interested in prediction or decision making under uncertainty the theory drives you very strongly in a subjective Bayesian” and also that ” the goal of finding the truth” is for reducing uncertainty in decision making problems. But if one wants decisions based on truth, then how can one desire subjective probability all the way down?
I seem to think we had much of this discussion before. –Just to note, I’m glad to have it again, even if we did…
If you’re searching for discussions of probabilities for hypothesis in expositions of settled science, you’re digging in the wrong place. Try looking at various physics bloggers’ posts over the past two years as to what they thought the settled science *would be* as a result of the search for the Higgs boson.
Hi Corey, haven’t heard from you in awhile. I’m not searching for discussions of probabilities of hypotheses, but noting that I find it odd to overlook the forms that knowledge, inference, learning, evidence etc. actually take, both in inquiry at the frontiers and in appraising “settled science”, and also in day-to-day life. (Some probabilists do seek to add “rules of acceptance” to their repertoire, but it’s not clear any are adequate.) But you know my views here, I’m not sure I understand what you’re saying in your comments. Are you saying that physics blog posts are about assigning mathematical probabilities to the successes of various searches, or to conjectures about what we will or were expected to learn? Are they finding things haven’t turned out as they conjectured? Things are less settled, more settled? I’m curious.
Howdy, Mayo. I was prompted to respond to the post because I vaguely recalled reading at least one physics blogger discuss some then-recent findings and conclude the post by stating their personal odds for various hypotheses that could explain the data.
Let me put it this way. In your examination of Weinberg’s scientific description of the state of knowledge in particle physics, you find no degree-of-probability assignments. But even under the hypothesis that scientists reason in an essentially Bayesian way, we don’t expect to see probabilistic language in expositions of claims whose probabilities scientists judge to be negligibly smaller than one. In other words, this is not a severe test of your preferred explanation for scientific knowledge/evidence. In particular, the post title gets no warrant from the contents.
Under the hypothesis that scientists reason in an essentially Bayesian way, we expect to find probabilistic language and reasoning akin to Bayesian epistemology in expositions of ongoing investigations. Hence, my suggestion that you look at physics bloggers is intended to point to a more severe test of your explanation for scientific knowledge/evidence.
Are you saying then that there is something like a “ rule of acceptance” (as some philosophers have called it) wherein hypotheses, having attained a sufficiently high posterior probability are regarded as “known”? Is that your view? In any event, the bulk of this is about discovery, pursuit, and what seems possible and why—no degrees of mathematical probability are assigned to the broad possibilities. To say something like, “I think it’s probable we’ll find such and such,” and even to give one’s personal odds, is an informal, extramathematical use of probability of an event. It might mean something like, given the rate at which they are analyzing the data, it’s probable that by 2013 we’ll know such and such properties. Those speculations aren’t the numbers they’d use in the next day’s experiment are they?
I love that you put “known” in quotes. I do that every time I write about epidemiologists controlling for “known” confounders.
All I’m saying is: let’s suppose, contra Mayo, that scientists reason, update, and make decisions what to do in an essentially Bayesian way. Where can we find evidence of that? It is necessary to look *there* to put your account to a severe test.
George Pólya wrote a book called “Mathematics and Plausible Reasoning Volume II: Patterns of Plausible Reasoning” in which he laid out ways in which mathematicians can use mathematical “evidence” to reason about the plausibility of conjectures. E.T. Jaynes showed, in his book “Probability Theory: The Logic of Science”, how Bayesian probability captures and formalizes Pólya’s patterns of plausible reasoning. It would be enough for me to see that scientists also use these patterns of plausible reasoning, albeit these are “extramathematical”.
I don’t think there are explicit “rules of acceptance”.
Corey: It is necessary to look *where*? and what do the *’s mean? Are we back t the blog probability estimates (as the place to look?)
I’ve been studying science for many years in many fields and time periods. I don’t really see the relevance of the Polya mathematical reasoning here…anyway, as you say, they are extramathematical in nature. I think the philosopher/scientist who had the patterns of scientific reasoning down best was C.S. Peirce. On rules of acceptance, are you saying that even if the rules aren’t explicit, at some point high enough probability yields knowledge, or an acceptance of something as known?
Mayo: It is necessary to look wherever it is most likely to find discrepant evidence, supposing your account is in error. (Straight up severity criterion (ii).) I suggest expositions of ongoing investigations. Blogs are one good source for such.
Asterisks are for emphasis, like italics. I can’t figure out if/how italics can be put in comments here.
Polya’s reasoning wasn’t mathematical per se. It was reasoning about the plausibility of claims/conjectures. I think that’s clearly relevant to science.
I haven’t read Peirce on science. Where would you recommend I start?
The existence of the catchall hypothesis means that all high Bayesian probabilities (both explicitly mathematically calculated and informally assessed) and hence all apparent acceptances are conditional on the assumption that everything causal/relevant has been considered. I think in your terms, this is equivalent to “there’s always ceteris paribus assumptions”.
Sorry for terseness. Life intervenes. Hope this is not to opaque.