On August 20 I posted the start of “Discussion and Digest” by Bayesian statistician Tony O’Hagan– an oveview of responses to his letter (ISBA website) on the use of p-values in analyzing the Higgs data, prompted, in turn, by a query of subjective Bayesian Dennis Lindley. I now post the final section in which he discusses his own view. I think it raises many questions of interest both as regards this case, and more generally about statistics and science. My initial July 11 post is here.
“Higgs Boson – Digest and Discussion” By Tony O’Hagan
Discussion
So here are some of my own views on this.
There are good reasons for being cautious and demanding a very high standard of evidence before announcing something as momentous as H. It is acknowledged by those who use it that the 5-sigma standard is a fudge, though. They would surely be willing to make such an announcement if they were, for instance, 99.99% certain of H’s existence, as long as that 99.99% were rigorously justified. 5-sigma is used because they don’t feel able to quantify the probability of H rigorously. So they use the best statistical analysis that they know how to do, but because they also know there are numerous factors not taken into account by this analysis – the multiple testing, the likelihood of unrecognised or unquantified deficiencies in the data, experiment or statistics, and the possibility of other explanations – they ask for what on the face of it is an absurdly high level of significance from that analysis.
To do better would be a very demanding task. First, to be able to give a probability for H’s existence requires a Bayesian analysis, which means embracing a prior probability, and also prior distributions on the parameters of the background and the signal. A range of choices for those would have to be considered in order to ensure that the certainty of H’s existence at the end is uncontroversial. A benefit of a Bayesian analysis is that it would avoid the well-documented problems with p-values as measures of the strength of evidence, but a Bayesian analysis would not be easy to do.
The hardest part is to quantify all the things that could go wrong, which we can consider generically as model errors. It is so often said that ‘all models are wrong’ that it is something of a cliché, but very little consideration has been given to the implications. I have recently been working on this issue and, I am pleased to say, so have others. In particular I have contributed a discussion for a paper written by Stephen Walker to be published in the Journal of Statistical Planning and Inference on this subject.
Having said that to do better than the 5-sigma fudge would be a very demanding task, I see no reason why it should not be attempted. The discovery of H would have been a great opportunity. Of the hundreds or thousands of people working on that project, including no doubt in some way all of the top particle physicists and many other leading figures in physics, engineering, etc., how many were statisticians? And how many of those were top-flight statistics researchers?
I wrote a follow-up to my original message when it was clear how many people had taken the trouble already to respond, including physicists. In it, I made it clear that I was delighted that those people had risen to my rather crude insinuations. I wrote:
Incidentally, I am delighted to have received a number of responses from physicists. Some of them take me to task for the provocative style of my original message. I know how important it is to respect the skills and expertise of people in other professions, but I also did want to generate discussion – in which objective I seem to have been successful. I am genuinely grateful to all those who have taken the time to give me the benefit of their knowledge and wisdom.
Respect for the skills of others cuts both ways. I may be maligning the physicists again, but my experience is that in most fields where statistics is needed it is generally done by non-statisticians. By that I mean people who are not primarily trained as statisticians and whose employment is not mainly as statisticians. I do not wish to suggest that a physicist, or an ecologist or whatever, who has taken an interest in statistics and studied statistical methods in order to apply them in their discipline is necessarily incompetent as a statistician. What bothers me is the commonly prevailing notion that a fully trained, specialist, professional statistician who has spent a lifetime honing their skills in their chosen field has nothing to teach them.
The discovery of H is big. The people involved would not dream of failing to use the very best possible physicists, not to mention the very best possible engineers to build their equipment, the very best possible computers, etc. But I’m willing to bet they didn’t consider it necessary to seek the advice, throughout every stage of working with their data, of the very top professional statisticians. Yet I would be very surprised if those top statisticians would not have welcomed the opportunity to work on such a high-profile, exciting and challenging project, in the same way that top physicists would jump at the chance to be involved.
It is simply not possible for us to do this now from outside the project. First, we do not have access to the data, and I’m sure will not be allowed access to it. Second, it’s a big job and should have had man-years of high-level statistical expertise devoted to it, like the very many man-years of effort of other top scientists.
In short, from the perspective of statistics, H looks to me like an opportunity missed. And so particle physicists will continue to use their 5-sigma rule without any real understanding of whether it does protect them (or over-protect them) against the feared slings and arrows of outrageous fortune.
To read the full article, see “Discussion and Digest”
Error Statistics Philosophy 
It seems to me that an STS grad student could write an entire Ph.D dissertation just deconstructing the philosophical, statistical, political, and sociological issues of this single episode within the huge Higgs boson discovery arena. I will just make two quick, remarks (for others to puzzle):
(1) O’Hagan regrets that the particle physicists did not consult with professional statisticians, and yet some responders, e.g., Larry Wasserman, notes: “I was on the statistics committee for one of the detectors at Fermi-Lab for a year. These physicists think about these things pretty carefully.”
(2) If O’Hagan really believes that an in-depth (subjective?) Bayesian analysis would provide physicists with a “real understanding of whether [their 5-sigma rule] does protect them (or over-protect them)” (end of “digest”), then why does he claim that while physicists “generally regard science as uncompromisingly objective. I believe that this is misguided”?
I see what you mean, I hadn’t even noticed that. It does seem a little odd, since I think lots and lots people are on the ISBA list who aren’t Bayesians (not even sure when I started getting their stuff). So conceivably he intended to direct his queries to Bayesians to get them to seek answers from p-value users, …or something..
Why write to “dear bayesians” to ask why 5-sigma is used?
“First, to be able to give a probability for H’s existence requires a Bayesian analysis, which means embracing a prior probability, and also prior distributions on the parameters of the background and the signal.” There’s the rub. Frankly, I see O’Hagan’s criticism as incredibly condescending and misguided, not only of physicists but also of statisticians who would disagree entirely with his proposed approach. As one who is “primarily trained as [a] statistician and whose employment is mainly as [a] statistician,” I would never even think of giving, estimating, or approximating anything like a probability that a hypothesis is true. To me, that is completely meaningless. That would require an interpretation of probability that is, well, little more than someone’s imagination… down the rabbit hole.
For the Bayesians out there, a question has always haunted me. What is the difference between “prior probability” and “expert opinion”?
Fisher said: “We are quite in danger of sending highly-trained and highly intelligent young men [it was 1958] out into the world with tables of erroneous number under their arms, and with a dense fog in the place where their brains outght to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort.” Quite right.
Mark: I completely agree with you. Yet O’Hagan seems to suggest that the Bayesian posterior, given enough “man-years” would serve to correct the error probability assessment. That suggests the two are aiming at the same measurement, which, if true, would be interesting, but would speak against any suggestion that they are guilty of “bad science”. Where is the Fisher quote from?
Hi Deborah,
The Fisher quote is from his 1958 paper “The Nature of Probability” from the Centenial Review: http://www.york.ac.uk/depts/maths/histstat/fisher272.pdf
… last paragraph. I love this particular paper, although I’m still trying to understand a couple of parts of it. I really think Fisher always understood exactly what he was saying, but he didn’t always give enough clues for the rest of us to catch up.
Dear prof. Mayo,
With pleasure I have read the discussion on the Higgs particle discovery. You definetely managed to get an interesting discussion flowing. I believe it is a good point that you make about the fact that there were surprisingly little statisticians involved (Relatively) in the discovery of such a probabalistic phenomenon. And you argue in the post that it would be strange to think physicists could not learn something from trained statisticians about these matters. However, as a student of philosophy of the social sciences I have followed the discovery with admiration and was in fact inspired by the sophistication of the statistics. In fact, controlling for the look-else-where effect is not something that is done in the social sciences. My question to you then is, can users of statistics in the social sciences not learn something from the H discovery?
In a illuminating article by Louis Lyons (http://arxiv.org/pdf/0811.1663.pdf) he explains that it has happened often that particle physicists have found a value of 2 or 3 sigma’s, which disappeared when more data was found. This makes me wonder, how many rejected null hypotheses in the social sciences meet the same fate, while we will never find out because we stop looking at 2 sigma’s?
I believe you are right that the LEE is similar to the data mining problem. However, due to the type of question ATLAS and CMS were posing, it is easier to correct for this problem than it is for data-miners in the social sciences. Of course, the fact that the statisticians can control for it is merely due to the fact that ATLAS and CMS have the opportunity to worked collectively to find H. However, is the dissability of social scientists to do the same not a major threat to the reliability of their work?
I would love to hear your thoughts on this.
Kind regards
Willem: Thanks much for your comment. Just as regards to your point that I argue “that it that it would be strange to think physicists could not learn something from trained statisticians.” It was not my intent to claim this at all, it was O’Hagan. My own view is that they’re quite sophisticated about the statistics they use, that they evidently rely on a slew of panels with statisticians, and I do not think they deserved the insinuation that they might be guilty of “bad science” because they employed significance levels rather than doing a subjective Bayesian analysis. I have several questions on the Lyons paper which I might raise at another time.