2024-2025 Cruise
Our first stop in 2025 on the leisurely tour of SIST is Excursion 4 Tour I which you can read here. I hope that this will give you the chutzpah to push back in 2025, if you hear that objectivity in science is just a myth. This leisurely tour may be a bit more leisurely than I intended, but this is philosophy, so slow blogging is best. (Plus, we’ve had some poor sailing weather). Please use the comments to share thoughts.
.
Tour I The Myth of “The Myth of Objectivity”*
Objectivity in statistics, as in science more generally, is a matter of both aims and methods. Objective science, in our view, aims to find out what is the case as regards aspects of the world [that hold] independently of our beliefs, biases and interests; thus objective methods aim for the critical control of inferences and hypotheses, constraining them by evidence and checks of error. (Cox and Mayo 2010, p. 276) [i]
Whenever you come up against blanket slogans such as “no methods are objective” or “all methods are equally objective and subjective” it is a good guess that the problem is being trivialized into oblivion. Yes, there are judgments, disagreements, and values in any human activity, which alone makes it too trivial an observation to distinguish among very different ways that threats of bias and unwarranted inferences may be controlled. Is the objectivity–subjectivity distinction really toothless, as many will have you believe? I say no. I know it’s a meme promulgated by statistical high priests, but you agreed, did you not, to use a bit of chutzpah on this excursion? Besides, cavalier attitudes toward objectivity are at odds with even more widely endorsed grass roots movements to promote replication, reproducibility, and to come clean on a number of sources behind illicit results: multiple testing, cherry picking, failed assumptions, researcher latitude, publication bias and so on. The moves to take back science are rooted in the supposition that we can more objectively scrutinize results – even if it’s only to point out those that are BENT. The fact that these terms are used equivocally should not be taken as grounds to oust them but rather to engage in the difficult work of identifying what there is in “objectivity” that we won’t give up, and shouldn’t.
The Key Is Getting Pushback! While knowledge gaps leave plenty of room for biases, arbitrariness, and wishful thinking, we regularly come up against data that thwart our expectations and disagree with the predictions we try to foist upon the world. We get pushback! This supplies objective constraints on which our critical capacity is built. Our ability to recognize when data fail to match anticipations affords the opportunity to systematically improve our orientation. Explicit attention needs to be paid to communicating results to set the stage for others to check, debate, and extend the inferences reached. Which conclusions are likely to stand up? Where do the weakest parts remain? Don’t let anyone say you can’t hold them to an objective account.
Excursion 2, Tour II led us from a Popperian tribe to a workable demarcation for scientific inquiry. That will serve as our guide now for scrutinizing the myth of the myth of objectivity. First, good sciences put claims to the test of refutation, and must be able to embark on an inquiry to pin down the sources of any apparent effects. Second, refuted claims aren’t held on to in the face of anomalies and failed replications; they are treated as refuted in further work (at least provisionally); well-corroborated claims are used to build on theory or method: science is not just stamp collecting. The good scientist deliberately arranges inquiries so as to capitalize on pushback, on effects that will not go away, on strategies to get errors to ramify quickly and force us to pay attention to them. The ability to register how hunting, optional stopping, and cherry picking alter their error-probing capacities is a crucial part of a method’s objectivity. In statistical design, day-to-day tricks of the trade to combat bias are consciously amplified and made systematic. It is not because of a “disinterested stance” that we invent such methods; it is that we, quite competitively and self-interestedly, want our theories to succeed in the market place of ideas.
Admittedly, that desire won’t suffice to incentivize objective scrutiny if you can do just as well producing junk. Successful scrutiny is very different from success at grants, getting publications and honors. That is why the reward structure of science is so often blamed nowadays. New incentives, gold stars and badges for sharing data and for resisting the urge to cut corners are being adopted in some fields. Fortunately, for me, our travels will bypass lands of policy recommendations, where I have no special expertise. I will stop at the perimeters of scrutiny of methods which at least provide us citizen scientists armor against being misled. Still, if the allure of carrots has grown stronger than the sticks, we need stronger sticks.
Problems of objectivity in statistical inference are deeply intertwined with a jungle of philosophical problems, in particular with questions about what objectivity demands, and disagreements about “objective versus subjective” probability. On to the jungle!
[i] Mayo and Cox (2010), “Objectivity and Conditionality in Frequentist Inference”, is the paper that led me to the critical analysis of Birnbaum on the Likelihood Principle. How could I write on “conditionality” if it leads to renouncing error probabilities? I asked David Cox. We agreed that it did not.
*From Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (Mayo 2018, CUP)
Error Statistics Philosophy 
A problem with the term “objectivity” is that it is used in various different ways by different people (of which you are obviously aware), often in a way that seems to be intended to add authority to certain claims that they otherwise wouldn’t have. I have no issue with scrutiny, pushback, and capitalizing from it as mentioned here; indeed I think they are essential and important parts of science. However I think that the term “objectivity” for this is rather imprecise and prone to be misunderstood, and I prefer our more detailed listing of ideals connected to the concepts of objectivity and subjectivity given in the paper “Beyond Subjective and Objective in Statistics” with Andrew Gelman, https://academic.oup.com/jrsssa/article/180/4/967/7068392. Our list includes “correspondence to observable reality”, which is where pushback comes in.
Christian:
Thanks for your comment and the link to your paper with Gelman. Many crucial terms are used in various ways. We should clarify them, not replace them. That is a central job of the philosopher. That is what I do here. My focus, of course, is objectivity in science. I would not want to curtail use of the term, nor replace it with various criteria that arise in promoting objectivity. Likewise, I would not be in favor of using “’multiple perspectives’ and ‘context dependence’, instead of ‘subjectivity’”, as you propose in your paper. These notions can be quite important in objectivity and they do not capture subjectivity in the sense of failures to achieve scientific objectivity. If you give me reasons why you think my notion comes up short, I can consider them. I will reread your paper with Gelman since it has been a while.
You write that “there are judgments, disagreements, and values in any human activity”. I believe many people who talk about “the myth of objectivity” mean just that, “too trivial” or not. The term “objectivity” is often used in sales pitches that are meant to make us forget that humans trivially won’t leave their skins and that whatever we come up with is affected by such judgments and values. Not buying such claims makes sense, regardless of how you use the term.
Around the time we wrote the “Beyond Subjective and Objective” paper, I went to a philosophical conference on objectivity. I saw many presentations in which a philosopher W said something to the effect that “the concept of objectivity advocated by philosopher X doesn’t work for reason Y, but we would still like to use it, so I propose amending it by making the change Z” Needless to say that in the following discussions, Z usually turned out to be controversial as well. As you know, I am a constructivist, so I wasn’t a fan of the concept of objectivity even before that, but this experience certainly felt like a confirmation.
What we do in the paper is that we try to be more specific about what good scientific work requires, in terms of aspects than can be more or less directly checked. I certainly don’t support the “blanket slogans” in the beginning. I could probably be fine with the term “objectivity” as referring to an attitude, acknowledging that for the “trivial” reasons mentioned above it can never be fully achieved, but we try to get there as well as we can anyway. But then the term wouldn’t apply to methods in themselves as they are separate from attitudes, and neither to claims or results. We use “multiple perspectives” and “context dependence” to capture what can be valuable for science from what is usually called “subjective”, but you are right that “subjective” in science more often refers to personal bias and the like. However, in the “trivial” sense above, subjectivity is a basic condition of humans trying to understand the world they live in, so I’d prefer to be more specific also when referring to negative implications of it.
It is important to me, in any case, to acknowledge that skepticism against the objectivity concept does not mean that everything is equally valid or invalid, and nothing can be done. As said before I largely agree with what you write about pushback, putting claims to the test etc.
Christian:
Your claim that “subjectivity is a basic condition of humans” is redolent of the “all flesh is grass” variety: “everything humans do is done by a subject, so they are all subjective”.
I’m providing an account of objectivity in science and inquiry. The goal isn’t to capture everything humans do, or capture various other things humans do outside of the context of scientific inquiry.
I don’t know whose views you heard at the philosophy conference. I don’t see how the fact that you heard disagreement among philosophers (who disagree about everything) diminishes my articulation of scientific objectivity.
There are lots of terms that are “used in various different ways by different people … often in a way that seems to be intended to add authority” and I am radically opposed to banning such words for that reason. The job of a philosopher, or anyone prepared to take on the task of providing useful clarity to important, often misunderstood notions, is to do precisely that. That is why I devote a “tour” in SIST to debunking the myth that objectivity is a myth. I recommended putting “objectivity” in the title of my paper with David Cox (“Objectivity and Conditionality in Frequentist Inference”) for the same reason. Lists of (vague)”family resemblance” terms (including “correspondence to observable relity”) will not suffice to charge that a method, inference, criticism is not objective. And we need that. I’m prepared to take on challenges to my notion of objectivity in science. I take on some other doozies in my book (SIST) like demarcation of science/pseudoscience, evidence, severe test, biasing selection effect and many others. Why would I choose to become a philosopher of science if I wasn’t prepared to analyze and illuminate important and difficult concepts whose unclarity was holding us back in tackling threats to good science. Of course, my analyses are part of a full philosophy of science I put forward.
I happen to notice Ben Recht wrote an article on “A Bureaucratic Theory of Statistics”. The comment I just posted on his blog seems relevant to our discussion of objectivity.
https://open.substack.com/pub/argmin/p/bureaucratic-statistics?r=1qyhw0&utm_campaign=comment-list-share-cta&utm_medium=web&comments=true&commentId=85154748
I wrote: You “introduce the concept of ‘ex ante policy,’ describing statistical rules and procedures designed before data collection to govern future actions” and claim “The ex ante frame obviates heated debates about inferential interpretations of probability and statistical tests, p-values, and rituals”. I don’t see how it sidesteps the debates about how methods serve the function of learning from data, and about which methods ought to constrain statistical inferences and subsequent policy.
My point is illustrated in the (2021)editorial of mine that you cite:
“While it is well known that stopping when the data look good inflates the type I error probability, a strict Bayesian is not required to adjust for interim checking because the posterior probability is unaltered. Advocates of Bayesian clinical trials are in a quandary because “The [regulatory] requirement of Type I error control for Bayesian [trials] causes them to lose many of their philosophical advantages, such as compliance with the likelihood principle” (Ryan et al., 2020: 7).” https://conbio.onlinelibrary.wiley.com/doi/full/10.1111/cobi.13861
Thus, the foundational questions reappear in the form of which regulations are warranted for the goals at hand, and why. We still need to ask what warranted scientific inference requires.
The authors, researchers in radiation oncology, go on to say: If we “simply report the posterior probability of benefit, then we could potentially avoid having to specify the type I error of a Bayesian design”. If they strongly believe the effect, they claim, the need to control type I errors is of no interest to Bayesians. That is what led to the remark of mine you cite.
“It may be retorted that implausible inferences will indirectly be blocked by appropriate prior degrees of belief (informative priors), but this misses the crucial point. The key function of statistical tests is to constrain the human tendency to selectively favor views they believe in.” (Mayo 2021)
My remark was intended in this context, where the constraint had to do with Type I errors control. (It would have been too vague on its own.) A discussion of Ryan et al., is in my blogpost, “Should Bayesian clinical trialists wear error statistical hats?” https://errorstatistics.com/2021/08/21/should-bayesian-clinical-trialists-wear-error-statistical-hats/
Ben Recht responded to my comment, and I replied with the following comment on his blog:
https://open.substack.com/pub/argmin/p/bureaucratic-statistics?r=1qyhw0&utm_campaign=comment-list-share-cta&utm_medium=web&comments=true&commentId=85880829
I need to write a blogpost on his view of “Bureaucratic statistics”:
This is a much needed reaffirmation in an era in which science is under attack by postmodernists in the universities; by newly empowered religious fundamentalists who reject any scientific claims that contradict their religious, nonscientific beliefs; and by political tribes who reject any scientific claims that contradict their leaders’ assertions.
Bill:
Yet it will not convince those who remove themselves from the remit of science.I hope that it helps everyone else.
The Mayo-Hennig exchange is very interesting. The so called pushback assumes an ability to pose questions which is not as evident as it may seem. In fact, models allow you to sharpen your questions. Statisticians know that implicitly but do not address this with a methodology. See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4683252 and https://onlinelibrary.wiley.com/doi/abs/10.1002/qre.3449
For a cognitive science perspective on posing questions see https://osf.io/preprints/psyarxiv/qzsgf
ron
Ken:
I think statistical methodology, including experimental design, do address how to pose questions in order to promote pushback via tests with good error probability characteristics. That it doesn’t provide recipes, but rather requires thinking about how variability can create obstacles (as well as clever tools) for finding things out, does not mean the riches aren’t already embedded in the methods. I will check out your links.