On August 20 I posted the start of “Discussion and Digest” by Bayesian statistician Tony O’Hagan– an oveview of responses to his letter (ISBA website) on the use of p-values in analyzing the Higgs data, prompted, in turn, by a query of subjective Bayesian Dennis Lindley. I now post the final section in which he discusses his own view. I think it raises many questions of interest both as regards this case, and more generally about statistics and science. My initial July 11 post is here.
“Higgs Boson – Digest and Discussion” By Tony O’Hagan
So here are some of my own views on this.
There are good reasons for being cautious and demanding a very high standard of evidence before announcing something as momentous as H. It is acknowledged by those who use it that the 5-sigma standard is a fudge, though. They would surely be willing to make such an announcement if they were, for instance, 99.99% certain of H’s existence, as long as that 99.99% were rigorously justified. 5-sigma is used because they don’t feel able to quantify the probability of H rigorously. So they use the best statistical analysis that they know how to do, but because they also know there are numerous factors not taken into account by this analysis – the multiple testing, the likelihood of unrecognised or unquantified deficiencies in the data, experiment or statistics, and the possibility of other explanations – they ask for what on the face of it is an absurdly high level of significance from that analysis.
To do better would be a very demanding task. First, to be able to give a probability for H’s existence requires a Bayesian analysis, which means embracing a prior probability, and also prior distributions on the parameters of the background and the signal. A range of choices for those would have to be considered in order to ensure that the certainty of H’s existence at the end is uncontroversial. A benefit of a Bayesian analysis is that it would avoid the well-documented problems with p-values as measures of the strength of evidence, but a Bayesian analysis would not be easy to do.
The hardest part is to quantify all the things that could go wrong, which we can consider generically as model errors. It is so often said that ‘all models are wrong’ that it is something of a cliché, but very little consideration has been given to the implications. I have recently been working on this issue and, I am pleased to say, so have others. In particular I have contributed a discussion for a paper written by Stephen Walker to be published in the Journal of Statistical Planning and Inference on this subject.
Having said that to do better than the 5-sigma fudge would be a very demanding task, I see no reason why it should not be attempted. The discovery of H would have been a great opportunity. Of the hundreds or thousands of people working on that project, including no doubt in some way all of the top particle physicists and many other leading figures in physics, engineering, etc., how many were statisticians? And how many of those were top-flight statistics researchers?
I wrote a follow-up to my original message when it was clear how many people had taken the trouble already to respond, including physicists. In it, I made it clear that I was delighted that those people had risen to my rather crude insinuations. I wrote:
Incidentally, I am delighted to have received a number of responses from physicists. Some of them take me to task for the provocative style of my original message. I know how important it is to respect the skills and expertise of people in other professions, but I also did want to generate discussion – in which objective I seem to have been successful. I am genuinely grateful to all those who have taken the time to give me the benefit of their knowledge and wisdom.
Respect for the skills of others cuts both ways. I may be maligning the physicists again, but my experience is that in most fields where statistics is needed it is generally done by non-statisticians. By that I mean people who are not primarily trained as statisticians and whose employment is not mainly as statisticians. I do not wish to suggest that a physicist, or an ecologist or whatever, who has taken an interest in statistics and studied statistical methods in order to apply them in their discipline is necessarily incompetent as a statistician. What bothers me is the commonly prevailing notion that a fully trained, specialist, professional statistician who has spent a lifetime honing their skills in their chosen field has nothing to teach them.
The discovery of H is big. The people involved would not dream of failing to use the very best possible physicists, not to mention the very best possible engineers to build their equipment, the very best possible computers, etc. But I’m willing to bet they didn’t consider it necessary to seek the advice, throughout every stage of working with their data, of the very top professional statisticians. Yet I would be very surprised if those top statisticians would not have welcomed the opportunity to work on such a high-profile, exciting and challenging project, in the same way that top physicists would jump at the chance to be involved.
It is simply not possible for us to do this now from outside the project. First, we do not have access to the data, and I’m sure will not be allowed access to it. Second, it’s a big job and should have had man-years of high-level statistical expertise devoted to it, like the very many man-years of effort of other top scientists.
In short, from the perspective of statistics, H looks to me like an opportunity missed. And so particle physicists will continue to use their 5-sigma rule without any real understanding of whether it does protect them (or over-protect them) against the feared slings and arrows of outrageous fortune.
To read the full article, see “Discussion and Digest”