There is no reason regularisation needs to be done in parameter space – it can also be done in ‘data space.

In fact I would argue this is what the point is of including yrep. Since the ‘likelihood’ is not ‘god given’ as all here acknowledge you could think of this as modifying the likelihood, or as a hierarchical Bayes model or something similar.

RE complicated: one thing that taking an approach more akin to Laurie’s approach is the possibility of including eg ‘checklists’, and other processes without simple expressions, into the model.

]]>I don’t mind the idea of seeing a prior as part of the model–presumably, I guess, as a way to do what a likelihood would do, if there were no nuisance parameters. I’d really like to understand how it works, as you know, which is really why I put together the PSA symposium, and ideally compare your way of modeling using priors to a non-Bayesian approach to modeling. I’m not trying to be critical. On the contrary, you’ve said you’re doing something akin to error statistics, so I’d like to illuminate the philosophical underpinnings. We always come back to to the meaning of the prior. In a comment (to this post) Christian says they are frequentist, which I don’t think I’ve heard you say before, except for the fact that you do say, from time to time, that the prior for a parameter in a model might be construed as the relative frequency of values found in an actual or hypothetical universe of all the times you or others have used the model, regardless of field. You said something like this at the railroad station in New Jersey ]]>

Regarding, “if the goal is to determine those parameter values if any which are consistent with the data”: All this is conditional on a model for the data. And, as always, I don’t buy the argument that some model for the data is considered as God-given, but a prior distribution is not allowed. In my practice, the prior distribution is part of the model.

]]>Methods like what you can describe can work for simple problems. But for more complicated problems I prefer to write a full probability model. I don’t think it is onerous to supply prior distributions for real problems–typically this is a much easier task than setting up the “likelihood” or data model. And simulations are no problem at all–we can fit these models in Stan.

You can forget about the Kolmogorov-Smirnov thing; that was just a throwaway idea of mine that maybe didn’t make so much sense.

]]>The authors seem surprised that Popper denied you could have probabilistic guarantees about error rates*. They shouldnāt be. This was his key position, and itās why he never succeeded in giving us an adequate account of severity. I take it as a point of pride that some well known Popperians (Chalmers, Musgrave)āwho don’t do phil stat at allā say that my philosophy is like Popperās except that I have a better notion of a severe test. Popper regards H as highly corroborated if it has been subjected to a stringent probe of error and yet it survived. The trouble is, heās unable to allow that there are stringent error probes because they would requires endorsing claims about future error control. What these authors want (for a special learning theory context) is exactly what Popper says we cannot have.* That is why his ability to solve the problem of induction fails. I claim Popperās problem is in not taking the error statistical turn. He once wrote to me that he regretted not having learned modern statistics (this is when I sent him my approach to severity). He was locked in the kind of search for a ālogicā for science that logical empiricists craved (hence the disastrous theory of verisimilitude).

Iāve written a fair amount on Popper (e.g., in EGEK (1996), and in posts on this blog). But my new book āStatistical Inference as Severe Testingā (CUP)ānearing completionāhas a lot of new reflections on Popper. They seemed necessary to make out the contrasts with my approach. I thought maybe that made parts of the book too philosophical, so Iām very glad if whatever message is behind your link is of interest to statisticians. Yet Iām clueless as to what it has to do with your recent comment. That is, what’s the connection between working w/ densities vs distributions (or however you put your point about Gelman) on the one hand, and Popperian falsification vs error statistical control on the other.Can you explain?

*They write: āthere appears to be no way for Popper to speak about the reliability of well-tested theories, and yet one surely needs to be able to speak of oneās confidence in the future predictions of such theoriesā. Popper totally denies one needs to speak of any such thing. For him, an assessment of a theoryās degree of corroboration is merely a report of how well it has passed previous tests.

]]>inverting, and having got yourself into this mess you have to be very careful with your prior to extract yourself. This is not a good example of Bayesian EDA, but it is a good counter example.

Mayo: I am not sure myself what to make of it. I am not sure what the goal is. As Christian Hennig pointed out, if the goal is to determine those parameter values if any which are consistent with the data you do not need a prior to do this. If your goal is to check your posterior then you do it the other way round: determine the parameter values consistent with the data and then evaluate the posterior at these points. Another possibility is that you don’t know where to look for consistent parameter values and calculate the posterior in the hope that it will move you in that direction. This is pure speculation. Even if that works sampling from the posterior will tend to concentrate around “very” consistent values and be sparse on the edge, the very opposite of what one wants. I had hoped that my simulations would be enlightening but they were not except for the latter point.

On the two topologies see Chapter 1.2.8 of my book. The “dogma” is to work in one weak topology as exemplified by the Kolmogorov metric or metric defined by Vapnik-Cervonenkis classes. You may already know this but just in case here is a link.

http://is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/TR_145_%5B0%5D.pdf

]]>A more correct summary of my comments would be that evidence is reinterpreted or discounted or assumed to be extinguished by a later revelation of misleadingness, but where the evidence has an unknown state of misleadingness (as is most often the case) it is evidence. Even the results of a severe test can occasionally be misleading, and thus Fisher’s advice applies to such a result.

The origin of this set of comments was your final bullet point on the slide which implies that an isolated P-value is not evidence because it might be misleading. That is not correct, and in my opinion it does not follow from Fisher’s advice.

]]>See Sec. 5.5 here: https://arxiv.org/abs/1508.05453

Note that the main thing that is interpreted in a frequentist manner is the sampling model. In de Finetti’s terminology, the “prior” is the whole model as chosen before data, which includes the sampling model. What you probably mean by “prior” is what I call “parameter prior”. The arxiv paper lists three ways to interpret this, only the first of which is frequentist. But even if only the sampling model has a frequentist interpretation, one could still test it with misspecification tests that are independent of the parameters and therefore don’t depend on the parameter prior.

The main point of what Davies is saying regarding Gelman’s approach is that there’s some trouble with Gelman’s mixture example as analysed by Gelman, and it can be analysed in a better way not using a Bayesian approach and likelihoods. The major issue seems to be the potential degeneration of likelihoods in the mixture model, which Gelman addresses using “Bayesian regularisation”, i.e., clever choice of priors, and Davies states that no regularisation is needed if one doesn’t insist on analysing this using likelihoods.

]]>And since you’ve dropped by, which I’m glad,maybe you can explain in plain Jane language what Davies is saying in his comment. ]]>