“Using PhilStat to Make Progress in the Replication Crisis in Psych” at Society for PhilSci in Practice (SPSP)

I’m giving a joint presentation with Caitlin Parker[1] on Friday (June 17) at the meeting of the Society for Philosophy of Science in Practice (SPSP): “Using Philosophy of Statistics to Make Progress in the Replication Crisis in Psychology” (Rowan University, Glassboro, N.J.)[2] The Society grew out of a felt need to break out of the sterile straightjacket wherein philosophy of science occurs divorced from practice. The topic of the relevance of PhilSci and PhilStat to Sci has often come up on this blog, so people might be interested in the SPSP mission statement below our abstract.

Using Philosophy of Statistics to Make Progress in the Replication Crisis in Psychology

Deborah Mayo Virginia Tech, Department of Philosophy United States
Caitlin Parker Virginia Tech, Department of Philosophy United States

The discussion surrounding the replication crisis in psychology has raised philosophical issues that remain to be seriously addressed. These touch on foundational questions in the philosophy of statistics about the role of probability in scientific inference and the proper interpretation of statistical tests. Such matters are key to understanding a paradox related to replicability criticisms in social science. This is that, although critics argue that it is too easy to obtain statistically significant results, the comparably low rate of positive results in replication studies shows that it is quite difficult to obtain low p-values. The resolution of the paradox is that small p-values aren’t easy to come by when experimental protocols are preregistered and researcher flexibility is minimized. They are easy to generate thanks to biasing selection effects: cherry-picking, multiple testing, and the type of questionable research practices that are widely lampooned. The consequence of these influences is that the reported, ‘nominal’ p-value for the original study differs greatly from the ‘actual’ p-value. As Gelman and Loken (2014) have argued, the same problem occurs due to the flexibility of choices in the “forking paths” leading from data to inferences, even if the critique remains informal. It follows that to avoid problematic inferences, researchers need statistical tools with the capacity to pick up on the effects of biasing selections. Significance tests have a limited but important goal, especially in testing model assumptions. To trade them in for methods that do not pick up on alterations to error probabilities (Bayes ratios, posterior probabilities, likelihood ratios) is not progress, but would enable their effects to remain hidden . The sensitivity of p-values to selection effects is actually the key to understanding their relevance to appraising particular inferences, not just to long-run error control. The problems of hunting and cherry picking are not a matter of getting it wrong in the long run, but failing to provide good grounds for the intended inference in the immediate inquiry. There’s a second way in which reforms are in danger of enabling fallacies. It is fallacious to take the falsification of a null hypothesis as evidence for a substantive theory (confusing statistical and substantive hypotheses). Neither Fisherian nor NP tests permit moving directly from statistical significance to research hypotheses, let alone from a single, just significant result. Yet in order to block an inference to a research hypothesis, a popular reform is to assign a lump of prior probability on the “no effect” null hypothesis. But this countenances rather than prohibiting blurring statistical and substantive hypotheses! This is not only a statistical fallacy, it draws attention away from what is most needed in psychology experiments with poor replication: a scrutiny of the relevance of the measurements and experiments to the research hypotheses of interest [3]. Slides are here.

[1] Parker had been my Masters’ student at Virginia Tech, and is beginning Ph.D work at Carnegie Mellon University in the fall.
[2] We’re on at 11:30, Enterprise Center, room 409. (Third paper in a session that starts at 10:30). The conference goes from June 17-19.
[3] Links to some relevant posts are at the end.

SPSP Mission Statement

Philosophy of science has traditionally focused on the relation between scientific theories and the world, at the risk of disregarding scientific practice. In social studies of science and technology, the predominant tendency has been to pay attention to scientific practice and its relation to theories, sometimes willfully disregarding the world except as a product of social construction. Both approaches have their merits, but they each offer only a limited view, neglecting some essential aspects of science. We advocate a philosophy of scientific practice, based on an analytic framework that takes into consideration theory, practice and the world simultaneously.

The direction of philosophy of science we advocate is not entirely new: naturalistic philosophy of science, in concert with philosophical history of science, has often emphasized the need to study scientific practices; doctrines such as Hacking’s “experimental realism” have viewed active intervention as the surest path to the knowledge of the world; pragmatists, operationalists and late-Wittgensteinians have attempted to ground truth and meaning in practices. Nonetheless, the concern with practice has always been somewhat outside the mainstream of English-language philosophy of science. We aim to change this situation, through a conscious and organized programme of detailed and systematic study of scientific practice that does not dispense with concerns about truth and rationality.

Practice consists of organized or regulated activities aimed at the achievement of certain goals. Therefore, the epistemology of practice must elucidate what kinds of activities are required in generating knowledge. Traditional debates in epistemology (concerning truth, fact, belief, certainty, observation, explanation, justification, evidence, etc.) may be re-framed with benefit in terms of activities. In a similar vein, practice-based treatments will also shed further light on questions about models, measurement, experimentation, etc., which have arisen with prominence in recent decades from considerations of actual scientific work.

There are some salient aspects of our general approach that are worth highlighting here.

1. We are concerned with not only the acquisition and validation of knowledge, but its use. Our concern is not only about how pre-existing knowledge gets applied to practical ends, but also about how knowledge itself is fundamentally shaped by its intended use. We aim to build meaningful bridges between the philosophy of science and the newer fields of philosophy of technology and philosophy of medicine; we also hope to provide fresh perspectives for the latter fields.
2. We emphasize how human artifacts, such as conceptual models and laboratory instruments, mediate between theories and the world. We seek to elucidate the role that these artifacts play in the shaping of scientific practice.
3. Our view of scientific practice must not be distorted by lopsided attention to certain areas of science. The traditional focus on fundamental physics, as well as the more recent focus on certain areas of biology, will be supplemented by attention to other fields such as economics and other social/human sciences, the engineering sciences, and the medical sciences, as well as relatively neglected areas within biology, physics, and other physical sciences.
4. In our methodology, it is crucial to have a productive interaction between philosophical reasoning and a study of actual scientific practices, past and present. This provides a strong rationale for history-and-philosophy of science as an integrated discipline, and also for inviting the participation of practicing scientists, engineers and policymakers.

I. Some relevant recent posts on p-values (search this blog for many others):

“Don’t Throw Out the Error Control Baby with the Bad Statistics Bathwater”

“A Small P-value Indicates that the Results are Due to Chance Alone: Fallacious or not: More on the ASA P-value Doc”

“P-Value Madness: A Puzzle About the Latest Test Ban, or ‘Don’t Ask, Don’t Tell’”

II. Posts on replication research in psychology:

Repligate Returns (or, the Non Significance of Nonsignificant Results Are the New Significant Results)

This includes links to:

“Some Ironies in the Replication Crisis in Social Psychology”

“The Paradox of Replication and the Vindication of the P-value, but She Can Go Deeper”

“Out Damned Pseudoscience: Nonsignificant Results Are the New Significant Results” 

Save