Is it true that all epistemic principles can only be defended circularly? A Popperian puzzle

Posted on February 10, 2014 by Mayo

Current day Popperians, the “critical rationalists”, espouse the following epistemic principle CR:[i]

(CR) it is reasonable to adopt or believe a claim or theory P which best survives serious criticism.

What justifies CR? To merely declare it is a reasonable epistemic principle without giving evidence that following it advances any epistemic goals is entirely unsatisfactory, and decidedly un-Popperian in spirit.

Alan Musgrave (1999), leading critical rationalist, mounts a defence of CR that he openly concedes is circular, admitting, as he does, that such circular defences could likewise be used to argue for principles he himself regards as ‘crazy’.
However, he also gives a subtle and clever argument that it’s impossible to do better, that such a circular defense is the only kind possible. So since we’re reading Popper this week (some of us), and since an analogous argument arises in defending principles of statistical inference, try your hand at this conundrum. Continue reading →

Categories: philosophy of science, Popper, Statistics | 2 Comments

Phil 6334: Day #3: Feb 6, 2014

Posted on February 9, 2014 by Mayo

Day #3: Spanos lecture notes 2, and reading/resources from Feb 6 seminar

6334 Day 3 slides: Spanos-lecture-2

___

Crupi & Tentori (2010). Irrelevant Conjunction: Statement and Solution of a New Paradox, Phil Sci, 77, 1–13.

Hawthorne & Fitelson (2004). Re-Solving Irrelevant Conjunction with Probabilistic Independence, Phil Sci 71: 505–514.

Skryms (1975) Choice and Chance 2nd ed. Chapter V and Carnap (pp. 206-211), Dickerson Pub. Co.

Mayo posts on the tacking paradox: Oct. 25, 2013: “Bayesian Confirmation Philosophy and the Tacking Paradox (iv)*” & Oct 25.

An update on this issue will appear shortly in a separate blogpost.

READING FOR NEXT WEEK
Selection (pp. 35-59) from: Popper (1962). Conjectures and Refutations: The Growth of Scientific Knowledge. Basic Books.

Categories: Bayes' Theorem, Phil 6334 class material, Statistics | Leave a comment

“Probabilism as an Obstacle to Statistical Fraud-Busting” (draft iii)

Posted on February 5, 2014 by Mayo

Update: Feb. 21, 2014 (slides at end): Ever find when you begin to “type” a paper to which you gave an off-the-cuff title months and months ago that you scarcely know just what you meant or feel up to writing a paper with that (provocative) title? But then, pecking away at the outline of a possible paper crafted to fit the title, you discover it’s just the paper you’re up to writing right now? That’s how I feel about “Is the Philosophy of Probabilism an Obstacle to Statistical Fraud Busting?” (the impromptu title I gave for my paper for the Boston Colloquium for the Philosophy of Science):

The conference is called: “Revisiting the Foundations of Statistics in the Era of Big Data: Scaling Up to Meet the Challenge.”

Here are some initial chicken-scratchings (draft (i)). Share comments, queries. (I still have 2 weeks to come up with something*.) Continue reading →

Categories: P-values, significance tests, Statistical fraudbusting, Statistics | Leave a comment

PhilStock: Bad news is bad news on Wall St. (rejected post)

Posted on February 3, 2014 by Mayo

I’ve been asked for a PhilStock tip. Well, remember when it could be said that “bad news is good news on wall street“?

No longer. Now bad is bad. I call these “blood days” on the stock market, and the only statistical advice that has held up over the past turbulent years is: Never try to catch a falling knife*.

*For more, you’ll have to seek my stock blog.

Categories: PhilStock, Rejected Posts | 8 Comments

Comedy hour at the Bayesian (epistemology) retreat: highly probable vs highly probed (vs B-boosts)

Posted on February 1, 2014 by Mayo

Since we’ll be discussing Bayesian confirmation measures in next week’s seminar—the relevant blogpost being here--let’s listen in to one of the comedy hours at the Bayesian retreat as reblogged from May 5, 2012.

Did you hear the one about the frequentist error statistical tester who inferred a hypothesis H passed a stringent test (with data x)?

The problem was, the epistemic probability in H was so low that H couldn’t be believed! Instead we believe its denial H’! So, she will infer hypotheses that are simply unbelievable!

So it appears the error statistical testing account fails to serve as an account of knowledge or evidence (i.e., an epistemic account). However severely I might wish to say that a hypothesis H has passed a test, this Bayesian critic assigns a sufficiently low prior probability to H so as to yield a low posterior probability in H[i]. But this is no argument about why this counts in favor of, rather than against, their particular Bayesian computation as an appropriate assessment of the warrant to be accorded to hypothesis H.

To begin with, in order to use techniques for assigning frequentist probabilities to events, their examples invariably involve “hypotheses” that consist of asserting that a sample possesses a characteristic, such as “having a disease” or “being college ready” or, for that matter, “being true.” This would not necessarily be problematic if it were not for the fact that their criticism requires shifting the probability to the particular sample selected—for example, a student Isaac is college-ready, or this null hypothesis (selected from a pool of nulls) is true. This was, recall, the fallacious probability assignment that we saw in Berger’s attempt, later (perhaps) disavowed. Also there are just two outcomes, say s and ~s, and no degrees of discrepancy from H. Continue reading →

Categories: Comedy, confirmation theory | Tags: Comedy club, criticism of frequentist methods, epistemic probabilsm, Frequentist inference, Peter Achinstein | 28 Comments

Phil 6334: Day #2 Slides

Posted on January 31, 2014 by Mayo

Day #2, Part 1: D. Mayo:

Class, Part 2: A. Spanos:
Probability/Statistics Lecture Notes 1: Introduction to Probability and Statistical Inference

Day #1 slides are here.

Categories: Phil 6334 class material, Philosophy of Statistics, Statistics | 8 Comments

BOSTON COLLOQUIUM FOR PHILOSOPHY OF SCIENCE: Revisiting the Foundations of Statistics

Posted on January 29, 2014 by Mayo

BOSTON COLLOQUIUM FOR PHILOSOPHY OF SCIENCE

2013–2014
54th Annual Program

Download the 54th Annual Program

REVISITING THE FOUNDATIONS OF STATISTICS IN THE ERA OF BIG DATA: SCALING UP TO MEET THE CHALLENGE

Cosponsored by the Department of Mathematics & Statistics at Boston University.
Friday, February 21, 2014
10 a.m. – 5:30 p.m.
Photonics Center, 9th Floor Colloquium Room (Rm 906)
8 St. Mary’s Street

10 a.m.–noon

Computational Challenges in Genomic Medicine
Jill Mesirov Computational Biology and Bioinformatics, Broad Institute
Selection, Significance, and Signification: Issues in High Energy Physics
Kent Staley Philosophy, Saint Louis University

1:30–5:30 p.m.

Multi-Resolution Inference: An Engineering (Engineered?) Foundation of Statistical Inference
Xiao-Li Meng Statistics, Harvard University
Is the Philosophy of Probabilism an Obstacle to Statistical Fraud Busting?
Deborah Mayo Philosophy, Virginia Tech
Targeted Learning from Big Data
Mark van der Laan Biostatistics and Statistics, UC Berkeley

Panel Discussion

Categories: Announcement, philosophy of science, Philosophy of Statistics, Statistical fraudbusting, Statistics | Leave a comment

Winner of the January 2014 palindrome contest (rejected post)

Posted on January 27, 2014 by Mayo

Winner of the January 2014 Palindrome Context

Karthik Durvasula
Visiting Assistant Professor in Phonology & Phonetics at Michigan State University

Palindrome: Test’s optimal? Agreed! Able to honor? O no! Hot Elba deer gala. MIT-post set.

The requirement was: A palindrome with “optimal” and “Elba”.

Bio: I’m a Visiting Assistant Professor in Phonology & Phonetics at Michigan State University. My work primarily deals with probing people’s subconscious knowledge of (abstract) sound patterns. Recently, I have been working on auditory illusions that stem from the bias that such subconscious knowledge introduces.

Statement: “Trying to get a palindrome that was at least partially meaningful was fun and challenging. Plus I get an awesome book for my efforts. What more could a guy ask for! I also want to thank Mayo for being excellent about email correspondence, and answering my (sometimes silly) questions tirelessly.”

Book choice: EGEK 1996! 🙂
[i.e.,Mayo (1996): “Error and the Growth of Experimental Knowledge”]

CONGRATULATIONS! And thanks so much for your interest!

February contest: Elba plus deviate (deviation)*

New Rule: Using both deviate and deviant tops an acceptable palindrome that only uses deviate (but can earn 1/2 prize voucher for doubling on another month).

Categories: Announcement, Palindrome, Rejected Posts | Leave a comment

U-Phil (Phil 6334) How should “prior information” enter in statistical inference?

Posted on January 25, 2014 by Mayo

On weekends this spring (in connection with Phil 6334, but not limited to seminar participants) I will post relevant “comedy hours”, invites to analyze short papers or blogs (“U-Phils”, as in “U-philosophize”), and some of my “deconstructions” of articles. To begin with a “U-Phil”, consider a note by Andrew Gelman: “Ethics and the statistical use of prior information,”[i].

I invite you to send (to error@vt.edu) informal analyses (“U-Phil”, ~500-750 words) by February 10) [iv]. Indicate if you want your remarks considered for possible posting on this blog.

Writing philosophy differs from other types of writing: Some links to earlier U-Phils are here. Also relevant is this note: “So you want to do a philosophical analysis?”

U-Phil (2/10/14): In section 3 Gelman comments on some of David Cox’s remarks in a (highly informal and non-scripted) conversation we recorded:

“A Statistical Scientist Meets a Philosopher of Science: A Conversation between Sir David Cox and Deborah Mayo,” published in Rationality, Markets and Morals [iii] (Section 2 has some remarks on Larry Wasserman, by the way.)

Here’s the relevant portion of the conversation:

COX: Deborah, in some fields foundations do not seem very important, but we both think foundations of statistical inference are important; why do you think that is?

MAYO: I think because they ask about fundamental questions of evidence, inference, and probability. I don’t think that foundations of different fields are all alike; because in statistics we’re so intimately connected to the scientific interest in learning about the world, we invariably cross into philosophical questions about empirical knowledge and inductive inference.

COX: One aspect of it is that it forces us to say what it is that we really want to know when we analyze a situation statistically. Do we want to put in a lot of information external to the data, or as little as possible. It forces us to think about questions of that sort.

MAYO: But key questions, I think, are not so much a matter of putting in a lot or a little information. …What matters is the kind of information, and how to use it to learn. This gets to the question of how we manage to be so successful in learning about the world, despite knowledge gaps, uncertainties and errors. To me that’s one of the deepest questions and it’s the main one I care about. I don’t think a (deductive) Bayesian computation can adequately answer it.…..

COX: There’s a lot of talk about what used to be called inverse probability and is now called Bayesian theory. That represents at least two extremely different approaches. How do you see the two? Do you see them as part of a single whole? Or as very different? Continue reading →

Categories: Background knowledge, Philosophy of Statistics, U-Phil | Tags: D. G. Mayo, Sir David Cox | 2 Comments

Phil 6334: Slides from Day #1: Four Waves in Philosophy of Statistics

Posted on January 24, 2014 by Mayo

First installment 6334 syllabus (Mayo and Spanos)
D. Mayo slides from Day #1: Jan 23, 2014

I will post seminar slides here (they will generally be ragtag affairs), links to the papers are in the syllabus.

Categories: Phil 6334 class material, Philosophy of Statistics, Statistics | 14 Comments

Phil6334: “Philosophy of Statistical Inference and Modeling” New Course: Spring 2014: Mayo and Spanos: (Virginia Tech) UPDATE: JAN 21

Posted on January 22, 2014 by Mayo

FURTHER UPDATED: New course for Spring 2014: Thurs 3:30-6:15 (Randolph 209)

first installment 6334 syllabus_SYLLABUS (first) Phil 6334: Philosophy of Statistical Inference and Modeling

picture-072-1-1

D. Mayo and A. Spanos

Contact: error@vt.edu

This new course, to be jointly taught by Professors D. Mayo (Philosophy) and A. Spanos (Economics) will provide an introductory, in-depth introduction to graduate level research in philosophy of inductive-statistical inference and probabilistic methods of evidence (a branch of formal epistemology). We explore philosophical problems of confirmation and induction, the philosophy and history of frequentist and Bayesian approaches, and key foundational controversies surrounding tools of statistical data analytics, modeling and hypothesis testing in the natural and social sciences, and in evidence-based policy.

We now have some tentative topics and dates:

1. 1/23	Introduction to the Course: 4 waves of controversy in the philosophy of statistics
2. 1/30	How to tell what’s true about statistical inference: Probabilism, performance and probativeness
3. 2/6	Induction and Confirmation: Formal Epistemology
4. 2/13	Induction, falsification, severe tests: Popper and Beyond
5. 2/20	Statistical models and estimation: the Basics
6. 2/27	Fundamentals of significance tests and severe testing
7. 3/6	Five sigma and the Higgs Boson discovery Is it “bad science”?
	SPRING BREAK Statistical Exercises While Sunning
8. 3/20	Fraudbusting and Scapegoating: Replicability and big data: are most scientific results false?
9. 3/27	How can we test the assumptions of statistical models? All models are false; no methods are objective: Philosophical problems of misspecification testing: Spanos method
10. 4/3	Fundamentals of Statistical Testing: Family Feuds and 70 years of controversy
11. 4/10	Error Statistical Philosophy: Highly Probable vs Highly Probed Some howlers of testing
12. 4/17	What ever happened to Bayesian Philosophical Foundations? Dutch books etc. Fundamental of Bayesian statistics
13. 4/24	Bayesian-frequentist reconciliations, unifications, and O-Bayesians
14. 5/1	Overview: Answering the critics: Should statistical philosophy be divorced from methodology?
(15. TBA)	Topic to be chosen (Resampling statistics and new journal policies? Likelihood principle)

Interested in attending? E.R.R.O.R.S.* can fund travel (presumably driving) and provide accommodation for Thurs. night in a conference lodge in Blacksburg for a few people through (or part of) the semester. If interested, write ASAP for details (with a brief description of your interest and background) to error@vt.edu. (Several people asked about long-distance hook-ups: We will try to provide some sessions by Skype, and will put each of the seminar items here (also check the Phil6334 page on this blog).

A sample of questions we consider*:

What makes an inquiry scientific? objective? When are we warranted in generalizing from data?
What is the “traditional problem of induction”? Is it really insoluble? Does it matter in practice?
What is the role of probability in uncertain inference? (to assign degrees of confirmation or belief? to characterize the reliability of test procedures?) 3P’s: Probabilism, performance and probativeness
What is probability? Random variables? Estimates? What is the relevance of long-run error probabilities for inductive inference in science?
What did Popper really say about severe testing, induction, falsification? Is it time for a new definition of pseudoscience?
Confirmation and falsification: Carnap and Popper, paradoxes of confirmation; contemporary formal epistemology
What is the current state of play in the “statistical wars” e.g., between frequentists, likelihoodists, and (subjective vs. “non-subjective”) Bayesians?
How should one specify and interpret p-values, type I and II errors, confidence levels? Can one tell the truth (and avoid fallacies) with statistics? Do the “reformers” themselves need reform?
Is it unscientific (ad hoc, degenerating) to use the same data both in constructing and testing hypotheses? When and why?
Is it possible to test assumptions of statistical models without circularity?
Is the new research on “replicability” well-founded, or an erroneous use of screening statistics for long-run performance?
Should randomized studies be the “gold standard” for “evidence-based” science and policy?
What’s the problem with big data: cherry-picking, data mining, multiple testing
The many faces of Bayesian statistics: Can there be uninformative prior probabilities? (No) Principles of indifference over the years
Statistical fraudbusting: psychology, economics, evidence-based policy
Applied controversies (selected): Higgs experiments, climate modeling, social psychology, econometric modeling, development economic

D. Mayo (books):

How to Tell What’s True About Statistical Inference, (Cambridge, in progress).

Error and the Growth of Experimental Knowledge, Chicago: Chicago University Press, 1996. (Winner of 1998 Lakatos Prize).

Acceptable Evidence: Science and Values in Risk Management, co-edited with Rachelle Hollander, New York: Oxford University Press, 1994.

Aris Spanos (books):

Probability Theory and Statistical Inference, Cambridge, 1999.

Statistical Foundations of Econometric Modeling, Cambridge, 1986.

Joint (books): Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability and the Objectivity and Rationality of Science, D. Mayo & A. Spanos (eds.), Cambridge: Cambridge University Press, 2010. [Intro, Background & Chapter 1. (The book includes both papers and exchanges between Mayo and A. Chalmers, A. Musgrave, P. Achinstein, J. Worrall, C. Glymour, A. Spanos, and joint papers with Mayo and Sir David Cox)].

Categories: Announcement, Error Statistics, Statistics | 5 Comments

Sir Harold Jeffreys’ (tail area) one-liner: Sat night comedy [draft ii]

Posted on January 18, 2014 by Mayo

You might not have thought there could be new material for 2014, but there is, and if you look a bit more closely, you’ll see that it’s actually not Jay Leno who is standing up there at the mike ….

It’s Sir Harold Jeffreys himself! And his (very famous) joke, I admit, is funny. So, since it’s Saturday night, let’s listen in on Sir Harold’s howler* in criticizing the use of p-values.

“Did you hear the one about significance testers rejecting H₀ because of outcomes H₀ didn’t predict?

‘What’s unusual about that?’ you ask?

Well, what’s unusual, is that they do it when these unpredicted outcomes haven’t even occurred!”

Much laughter.

[The actual quote from Jeffreys: Using p-values implies that “An hypothesis that may be true is rejected because it has failed to predict observable results that have not occurred. This seems a remarkable procedure.” (Jeffreys 1939, 316)]

I say it’s funny, so to see why I’ll strive to give it a generous interpretation.

We can view p-values in terms of rejecting H₀, as in the joke: There’s a test statistic D such that H₀ is rejected if its observed value d₀ reaches or exceeds a cut-off d* where Pr(D > d*; H₀) is small, say .025.
Reject H₀ if Pr(D > d₀; H₀) < .025.
The report might be “reject H₀at level .025″.
Example: H₀: The mean light deflection effect is 0. So if we observe a 1.96 standard deviation difference (in one-sided Normal testing) we’d reject H₀ .

Now it’s true that if the observation were further into the rejection region, say 2, 3 or 4 standard deviations, it too would result in rejecting the null, and with an even smaller p-value. It’s also true that H₀ “has not predicted” a 2, 3, 4, 5 etc. standard deviation difference in the sense that differences so large are “far from” or improbable under the null. But wait a minute. What if we’ve only observed a 1 standard deviation difference (p-value = .16)? It is unfair to count it against the null that 1.96, 2, 3, 4 etc. standard deviation differences would have diverged seriously from the null, when we’ve only observed the 1 standard deviation difference. Yet the p-value tells you to compute Pr(D > 1; H₀), which includes these more extreme outcomes! This is “a remarkable procedure” indeed! [i]

So much for making out the howler. The only problem is that significance tests do not do this, that is, they do not reject with, say, D = 1 because larger D values might have occurred (but did not). D = 1 does not reach the cut-off, and does not lead to rejecting H_0.Moreover, looking at the tail area makes it harder, not easier, to reject the null (although this isn’t the only function of the tail area): since it requires not merely that Pr(D = d₀ ; H₀ ) be small, but that Pr(D > d₀ ; H₀ ) be small. And this is well justified because when this probability is not small, you should not regard it as evidence of discrepancy from the null. Before getting to this …. Continue reading →

Categories: Comedy, Fisher, Jeffreys, P-values, Statistics, Stephen Senn | 12 Comments

Objective/subjective, dirty hands and all that: Gelman/ Wasserman blogolog (ii)

Posted on January 16, 2014 by Mayo

Andrew Gelman says that as a philosopher, I should appreciate his blog today in which he records his frustration: “Against aggressive definitions: No, I don’t think it helps to describe Bayes as ‘the analysis of subjective beliefs’…” Gelman writes:

I get frustrated with what might be called “aggressive definitions,” where people use a restrictive definition of something they don’t like. For example, Larry Wasserman writes (as reported by Deborah Mayo):

“I wish people were clearer about what Bayes is/is not and what  frequentist inference is/is not. Bayes is the analysis of subjective  beliefs but provides no frequency guarantees. Frequentist inference  is about making procedures that have frequency guarantees but makes no  pretense of representing anyone’s beliefs.”

I’ll accept Larry’s definition of frequentist inference. But as for his definition of Bayesian inference: No no no no no. The probabilities we use in our Bayesian inference are not subjective, or, they’re no more subjective than the logistic regressions and normal distributions and Poisson distributions and so forth that fill up all the textbooks on frequentist inference.

To quickly record some of my own frustrations:*: First, I would disagree with Wasserman’s characterization of frequentist inference, but as is clear from Larry’s comments to (my reaction to him), I think he concurs that he was just giving a broad contrast. Please see Note [1] for a remark from my post: Comments on Wasserman’s “what is Bayesian/frequentist inference?” Also relevant is a Gelman post on the Bayesian name: [2].

Second, Gelman’s “no more subjective than…” evokes remarks I’ve made before. For example, in “What should philosophers of science do…” I wrote:

Arguments given for some very popular slogans (mostly by non-philosophers), are too readily taken on faith as canon by others, and are repeated as gospel. Examples are easily found: all models are false, no models are falsifiable, everything is subjective, or equally subjective and objective, and the only properly epistemological use of probability is to supply posterior probabilities for quantifying actual or rational degrees of belief. Then there is the cluster of “howlers” allegedly committed by frequentist error statistical methods repeated verbatim (discussed on this blog).

I’ve written a lot about objectivity on this blog, e.g., here, here and here (and in real life), but what’s the point if people just rehearse the “everything is a mixture…” line, without making deeply important distinctions? I really think that, next to the “all models are false” slogan, the most confusion has been engendered by the “no methods are objective” slogan. However much we may aim at objective constraints, it is often urged, we can never have “clean hands” free of the influence of beliefs and interests, and we invariably sully methods of inquiry by the entry of background beliefs and personal judgments in their specification and interpretation. Continue reading →

Categories: Bayesian/frequentist, Error Statistics, Gelman, Objectivity, Statistics | 41 Comments

Statistical Science meets Philosophy of Science: blog beginnings

Posted on January 14, 2014 by Mayo

“StatSci meets PhilSci”. (1/14/14)
As “Wasserman on Wasserman” (and links within) continues to rack up record hits (N.D.: you see you shouldn’t quit blogging)*, I’ve been asked about the origins of his and related discussions on this blog. For a quick answer:** many grew out of attempts to tackle the general question: “Statistical Science and Philosophy of Science: Where Do (Should) They meet?”–the title of a conference I organized (with A. Spanos***) at the London School of Economics, Center for the Philosophy of Natural and Social Science, CPNSS, in June 2010. In tackling this question, errorstatistics.com regularly returns to a set of contributions stemming from the conference, and conversations initiated soon after (with Andrew Gelman and Larry Wasserman)****. The conference site is here. My reflections in this general arena (Sept. 26, 2012) are here.

Opening with an informal (recorded) exchange: “A statistical scientist meets a philosopher of science: a conversation between Sir David Cox and Deborah Mayo”, this special topic of the on-line journal, Rationality, Markets and Morals (RMM), edited by Max Albert[i],—also a conference participant —has been an excellent home for continual updates (to which we may return at some point!)

Authors are: David Cox, Andrew Gelman, David F. Hendry, Deborah G. Mayo, Stephen Senn, Aris Spanos, Jan Sprenger, Larry Wasserman

To those who ask me what to read as background to some of the issues, have a look at those contributions. Many of them are discussed in specific blogposts (with “deconstructions” [by me], responses by authors, and insightful “U-Phil” analyses by readers) and comments.[ii]. (Search under U-Phil.) I have gathered a list of issues that we either haven’t taken up, or need to return to.

Here is the RMM blub:

Rationality, Markets and Morals: Studies at the Intersection of Philosophy and Economics
Guest Editors: Deborah G. Mayo, Aris Spanos and Kent W. Staley

Statistical Science Meets Philosophy of Science: The Two-Way Street

At one level of analysis, statisticians and philosophers of science ask many of the same questions: What should be observed and what may justifiably be inferred from the resulting data? How well-tested or confirmed are hypotheses with data? How can statistical models and methods bridge the gaps between data and scientific claims of interest? These general questions are entwined with long standing philosophical debates, so it is no wonder that the statistics crosses over so often into philosophical territory.

The “meeting grounds” of statistical science and philosophy of science are or should be connected by a two-way street: while general philosophical questions about evidence and inference bear on statistical questions (about methods to use, and how to interpret them), statistical methods bear on philosophical problems about inference and knowledge. As interesting as this two-way street has been over many years, we seem to be in need of some entirely new traffic patterns! That is the basis for this forum.

[i] Along with Hartmut Kliemt and Bernd Lahno.

[ii] The “deconstruction” activity on this blog began with my reaction to a paper by Jim Berger, in a recently reblogged post. Berger had replied in ‘Jim Berger on Jim Berger’.

*From the WordPress 2013 “annual report”: The busiest day of the year was February 18th. The most popular post that day was R. A. Fisher: how an outsider revolutionized statistics.

Also attracting huge hits was the guest post by Larry Laudan: Why Presuming Innocence is Not a Bayesian Prior: https://errorstatistics.com/2013/07/20/guest-post-larry-laudan-why-presuming-innocence-is-not-a-bayesian-prior/ Many other biggies (especially from guest posters) have attracted a large number of comments and views.

** This post adapts an earlier one here. This blog is on philosophy, after all: only careful and frequent rereading brings illumination.

***For a full list of collaborators, sponsors, logisticians, and related collaborations, see the conference page. The full list of speakers is found there as well. Should we do a 2015 update? or wait for ERROR 2016?

****Conference participants who never got around to sending papers: I think there’s still time.

Categories: StatSci meets PhilSci | 1 Comment

Two Severities? (PhilSci and PhilStat)

Posted on January 11, 2014 by Mayo

The blog “It’s Chancy” (Corey Yanofsky) has a post today about “two severities” which warrants clarification. Two distinctions are being blurred: between formal and informal severity assessments, and between a statistical philosophy (something Corey says he’s interested in) and its relevance to philosophy of science (which he isn’t). I call the latter an error statistical philosophy of science. The former requires both formal, semi-formal and informal severity assessments. Here’s his post:

In the comments to my first post on severity, Professor Mayo noted some apparent and some actual misstatements of her views.To avert misunderstandings, she directed readers to two of her articles, one of which opens by making this distinction:

“Error statistics refers to a standpoint regarding both (1) a general philosophy of science and the roles probability plays in inductive inference, and (2) a cluster of statistical tools, their interpretation, and their justiﬁcation.”

In Mayo’s writings I see two interrelated notions of severity corresponding to the two items listed in the quote: (1) an informal severity notion that Mayo uses when discussing philosophy of science and specific scientific investigations, and (2) Mayo’s formalization of severity at the data analysis level.

One of my besetting flaws is a tendency to take a narrow conceptual focus to the detriment of the wider context. In the case of Severity, part one, I think I ended up making claims about severity that were wrong. I was narrowly focused on severity in sense (2) — in fact, on one specific equation within (2) — but used a mish-mash of ideas and terminology drawn from all of my readings of Mayo’s work. When read through a philosophy-of-science lens, the result is a distorted and misstated version of severity in sense (1) .

As a philosopher of science, I’m a rank amateur; I’m not equipped to add anything to the conversation about severity as a philosophy of science. My topic is statistics, not philosophy, and so I want to warn readers against interpreting Severity, part one as a description of Mayo’s philosophy of science; it’s more of a wordy introduction to the formal definition of severity in sense (2).[It’s Chancy, Jan 11, 2014)

A needed clarification may be found in a post of mine which begins:

Error statistics: (1) There is a “statistical philosophy” and a philosophy of science. (a) An error-statistical philosophy alludes to the methodological principles and foundations associated with frequentist error-statistical methods. (b) An error-statistical philosophy of science, on the other hand, involves using the error-statistical methods, formally or informally, to deal with problems of philosophy of science: to model scientific inference (actual or rational), to scrutinize principles of inference, and to address philosophical problems about evidence and inference (the problem of induction, underdetermination, warranting evidence, theory testing, etc.).

I assume the interest here* is on the former, (a). I have stated it in numerous ways, but the basic position is that inductive inference—i.e., data-transcending inference—calls for methods of controlling and evaluating error probabilities (even if only approximate). An inductive inference, in this conception, takes the form of inferring hypotheses or claims to the extent that they have been well tested. It also requires reporting claims that have not passed severely, or have passed with low severity. In the “severe testing” philosophy of induction, the quantitative assessment offered by error probabilities tells us not “how probable” but, rather, “how well probed” hypotheses are. The local canonical hypotheses of formal tests and estimation methods need not be the ones we entertain post data; but they give us a place to start without having to go “the designer-clothes” route.

The post-data interpretations might be formal, semi-formal, or informal.

See also: Staley’s review of Error and Inference (Mayo and Spanos eds.)

Categories: Review of Error and Inference, Severity, StatSci meets PhilSci | 52 Comments

“Philosophy of Statistical Inference and Modeling” New Course: Spring 2014: Mayo and Spanos: (Virginia Tech)

Posted on January 7, 2014 by Mayo

New course for Spring 2014: Thursday 3:30-6:15

Phil 6334: Philosophy of Statistical Inference and Modeling

D. Mayo and A. Spanos

Contact: error@vt.edu

A sample of questions we consider*:

What makes an inquiry scientific? objective? When are we warranted in generalizing from data?
What is the “traditional problem of induction”? Is it really insoluble? Does it matter in practice?
What is the role of probability in uncertain inference? (to assign degrees of confirmation or belief? to characterize the reliability of test procedures?) 3P’s: Probabilism, performance and probativeness
What is probability? Random variables? Estimates? What is the relevance of long-run error probabilities for inductive inference in science?
What did Popper really say about severe testing, induction, falsification? Is it time for a new definition of pseudoscience?
Confirmation and falsification: Carnap and Popper, paradoxes of confirmation; contemporary formal epistemology
What is the current state of play in the “statistical wars” e.g., between frequentists, likelihoodists, and (subjective vs. “non-subjective”) Bayesians?
How should one specify and interpret p-values, type I and II errors, confidence levels? Can one tell the truth (and avoid fallacies) with statistics? Do the “reformers” themselves need reform?
Is it unscientific (ad hoc, degenerating) to use the same data both in constructing and testing hypotheses? When and why?
Is it possible to test assumptions of statistical models without circularity?
Is the new research on “replicability” well-founded, or an erroneous use of screening statistics for long-run performance?
Should randomized studies be the “gold standard” for “evidence-based” science and policy?
What’s the problem with big data: cherry-picking, data mining, multiple testing
The many faces of Bayesian statistics: Can there be uninformative prior probabilities? (No) Principles of indifference over the years
Statistical fraudbusting: psychology, economics, evidence-based policy
Applied controversies (selected): Higgs experiments, climate modeling, social psychology, econometric modeling, development economic

Interested in attending? E.R.R.O.R.S.* can fund travel (presumably driving) and provide lodging for Thurs. night in a conference lodge in Blacksburg for a few people through (or part of) the semester. Topics will be posted over the next week, but if you might be interested, write ASAP for details (with a brief description of your interest and background) to error@vt.edu.

*This course will be a brand new version of related seminar we’ve led in the past, so we don’t have the syllabus set yet. We’re going to try something different this time. I’ll be updating in subsequent installments to the blog.

Dates: January 23, 30; February 6, 13, 20, 27; March 6, [March 8-16 break], 20, 27; April 3,10, 17, 24; May 1

D. Mayo (books):

How to Tell What’s True About Statistical Inference, (Cambridge, in progress).

Error and the Growth of Experimental Knowledge, Chicago: Chicago University Press, 1996. (Winner of 1998 Lakatos Prize).

Acceptable Evidence: Science and Values in Risk Management, co-edited with Rachelle Hollander, New York: Oxford University Press, 1994.

Aris Spanos (books):

Probability Theory and Statistical Inference, Cambridge, 1999.

Statistical Foundations of Econometric Modeling, Cambridge, 1986.

Categories: Announcement, Error Statistics, Statistics | 9 Comments

Your 2014 wishing well….

Posted on January 4, 2014 by Mayo

A reader asks how I would complete the following sentence:
I wish that new articles* written in 2014 would refrain from_______.

Here are my quick answers, in no special order:
(a) rehearsing the howlers of significance tests and other frequentist statistical methods;

(b) misinterpreting p-values, ignoring discrepancy assessments (and thus committing fallacies of rejection and non-rejection);

(c) confusing an assessment of boosts in belief (or support) in claim H ,with assessing what (if anything) has been done to ensure/increase the severity of the tests H passes;

(d) declaring that “what we really want” are posterior probability assignments in statistical hypotheses without explaining what they would mean, and why we should want them;

(e) promoting the myth that frequentist tests (and estimates) form an inconsistent hybrid of incompatible philosophies (from Fisher and Neyman-Pearson);

(f) presupposing that a relevant assessment of the scientific credentials of research would be an estimate of the percentage of null hypothesis that are “true” (selected from an “urn of nulls”) given they are rejectable with a low p-value in an “up-down” use of tests;

(g) sidestepping the main sources of pseudoscience: insevere tests through interpretational and inferential latitude, and violations of statistical model assumptions.

The “2014 wishing well” stands ready for your sentence completions.

*The question alluded to articles linked with philosophy & methodology of statistical science.

Categories: Error Statistics, science communication, Statistics | Leave a comment

Error Statistics Philosophy: 2013

Posted on January 3, 2014 by Mayo

I blog ergo I blog

Error Statistics Philosophy: 2013
Organized by Nicole Jinn & Jean Anne Miller*

January 2013

(1/2) Severity as a ‘Metastatistical’ Assessment
(1/4) Severity Calculator
(1/6) Guest post: Bad Pharma? (S. Senn)
(1/9) RCTs, skeptics, and evidence-based policy
(1/10) James M. Buchanan
(1/11) Aris Spanos: James M. Buchanan: a scholar, teacher and friend
(1/12) Error Statistics Blog: Table of Contents
(1/15) Ontology & Methodology: Second call for Abstracts, Papers
(1/18) New Kvetch/PhilStock
(1/19) Saturday Night Brainstorming and Task Forces: (2013) TFSI on NHST
(1/22) New PhilStock
(1/23) P-values as posterior odds?
(1/26) Coming up: December U-Phil Contributions….
(1/27) U-Phil: S. Fletcher & N.Jinn
(1/30) U-Phil: J. A. Miller: Blogging the SLP

February 2013
(2/2) U-Phil: Ton o’ Bricks
(2/4) January Palindrome Winner
(2/6) Mark Chang (now) gets it right about circularity
(2/8) From Gelman’s blog: philosophy and the practice of Bayesian statistics
(2/9) New kvetch: Filly Fury
(2/10) U-PHIL: Gandenberger & Hennig: Blogging Birnbaum’s Proof
(2/11) U-Phil: Mayo’s response to Hennig and Gandenberger
(2/13) Statistics as a Counter to Heavyweights…who wrote this?
(2/16) Fisher and Neyman after anger management?
(2/17) R. A. Fisher: how an outsider revolutionized statistics
(2/20) Fisher: from ‘Two New Properties of Mathematical Likelihood’
(2/23) Stephen Senn: Also Smith and Jones
(2/26) PhilStock: DO < $70
(2/26) Statistically speaking… Continue reading →

Categories: blog contents, Error Statistics, Statistics | Leave a comment

Winner of the December 2013 Palindrome Book Contest (Rejected Post)

Posted on January 2, 2014 by Mayo

WINNER: Zachary David

See rejected posts.

Categories: Palindrome, Rejected Posts | Leave a comment

Midnight With Birnbaum (Happy New Year)

Posted on December 31, 2013 by Mayo

Just as in the past 2 years since I’ve been blogging, I revisit that spot in the road, get into a strange-looking taxi, and head to “Midnight With Birnbaum”. There are a couple of brief (12/31/13) updates at the end.

You know how in that (not-so) recent movie, “Midnight in Paris,” the main character (I forget who plays it, I saw it on a plane) is a writer finishing a novel, and he steps into a cab that mysteriously picks him up at midnight and transports him back in time where he gets to run his work by such famous authors as Hemingway and Virginia Wolf? He is impressed when his work earns their approval and he comes back each night in the same mysterious cab…Well, imagine an error statistical philosopher is picked up in a mysterious taxi at midnight (New Year’s Eve ~~2011~~ ~~2012~~, 2013) and is taken back fifty years and, lo and behold, finds herself in the company of Allan Birnbaum.[i]

ERROR STATISTICIAN: It’s wonderful to meet you Professor Birnbaum; I’ve always been extremely impressed with the important impact your work has had on philosophical foundations of statistics. I happen to be writing on your famous argument about the likelihood principle (LP). (whispers: I can’t believe this!)

BIRNBAUM: Ultimately you know I rejected the LP as failing to control the error probabilities needed for my Confidence concept.

ERROR STATISTICIAN: Yes, but I actually don’t think your argument shows that the LP follows from such frequentist concepts as sufficiency S and the weak conditionality principle WLP.[ii] Sorry,…I know it’s famous…

BIRNBAUM: Well, I shall happily invite you to take any case that violates the LP and allow me to demonstrate that the frequentist is led to inconsistency, provided she also wishes to adhere to the WLP and sufficiency (although less than S is needed).

ERROR STATISTICIAN: Well I happen to be a frequentist (error statistical) philosopher; I have recently (2006) found a hole in your proof,..er…well I hope we can discuss it.

BIRNBAUM: Well, well, well: I’ll bet you a bottle of Elba Grease champagne that I can demonstrate it!

ERROR STATISTICAL PHILOSOPHER: It is a great drink, I must admit that: I love lemons.

BIRNBAUM: OK. (A waiter brings a bottle, they each pour a glass and resume talking). Whoever wins this little argument pays for this whole bottle of vintage Ebar or Elbow or whatever it is Grease.

ERROR STATISTICAL PHILOSOPHER: I really don’t mind paying for the bottle.

BIRNBAUM: Good, you will have to. Take any LP violation. Let x’ be 2-standard deviation difference from the null (asserting m = 0) in testing a normal mean from the fixed sample size experiment E’, say n = 100; and let x” be a 2-standard deviation difference from an optional stopping experiment E”, which happens to stop at 100. Do you agree that:

(0) For a frequentist, outcome x’ from E’ (fixed sample size) is NOT evidentially equivalent to x” from E” (optional stopping that stops at n)

ERROR STATISTICAL PHILOSOPHER: Yes, that’s a clear case where we reject the strong LP, and it makes perfect sense to distinguish their corresponding p-values (which we can write as p’ and p”, respectively). The searching in the optional stopping experiment makes the p-value quite a bit higher than with the fixed sample size. For n = 100, data x’ yields p’= ~.05; while p” is ~.3. Clearly, p’ is not equal to p”, I don’t see how you can make them equal. Continue reading →

Categories: Birnbaum Brakes, strong likelihood principle | Tags: Birnbaum, fallacious argument, Likelihood Principle, proofs | 2 Comments

BOSTON COLLOQUIUM FOR PHILOSOPHY OF SCIENCE

2013–2014 54th Annual Program

The Statistics Wars & Their Casualties

Blog links (references)

Reviews of Statistical Inference as Severe Testing (SIST)

Interviews & Debates on PhilStat (2020)

Interviews on PhilStat (2019)

LSE PH500 Research Seminar (May 21-June 25, 2020): Controversies in Phil Stat

Summer Seminar 2019 (article)

Top Posts & Pages

Conferences & Workshops

RMM Special Topic

Mayo & Spanos, Error Statistics

Follow Blog via Email

My Websites

Recent Posts: PhilStatWars

LOG IN/OUT

Archives

© Deborah G. Mayo, Error Statistics Philosophy, 2011-2018 All Rights Reserved.

2013–2014
54th Annual Program