I think I can answer this point:

>I fail to see how anyone can evaluate an inference from data x to >a claim C without learning about the capabilities of the method, >through the relevant sampling distribution.

I think you are essentially right here. On the other hand, a fully Bayesian approach would not attempt to ‘evaluate an inference about claim C’, rather it would predict x_{n+1} using p(x_{n+1}|x). I think if you go over the optimal stopping example you cite and compute the predictive distributions you will find that they are entirely satisfactory regardless of the stopping rule used. If the experiment stops early the predictive distribution will be broad, if not it is a bit narrower. If a lot of data is collected the predictive distribution will have significant uncertainty – but almost all of this is due to the structure of the model rather than posterior uncertainty. Its easy to see that trying to distort the result by using a strange stopping rule just doesn’t work.

For this example p(x_{n+1}|x) is probably sufficient, but you could also use the richer p(x_{n+1},…,x_{n+k}|x) for some k. If you want I can draw up some plots to illustrate the point.

Essentially there is a difference in aims between the Bayesian and error statistical approach.

I think the ‘howlers’ do have some problems and agree with much with what you have written. The ‘howlers’ also tend to mangle the fact that there is a different goal in the two approaches.

I would also add, that while the Bayesian approach tends to win the philosophical arguments, it has a much tougher time in practice. A good example Senn raised was medical trial problems where a decision must be made to either determine if the drug is safe or do further testing. While a Bayesian decision theoretic appraoch could be formulated it would require computing expectations over very complex objects (outcomes of contemplated experiments) and would probably be overly academic rather than practically useful.

]]>My intended follow-up is all on Royall, but I didt get around to putting it up.

I don’t know what you mean by Royall computing “pre-data probabilities of error …for hypothesis-space design purposes.” Royall was sufficiently perturbed at arguments about his lack of error control–many of which I brought up in EGEK and when we were in a session together–that he highlighted the point against point example (as Savage does). But this only holds for predesignated points. ]]>

Totally agreed here.

“and secondly, I can’t imagine on what grounds a Bayesian would criticize a sampling theorist for doing the same”

Really? their grounds are generally (a) that the appeal to the sampling distribution could only be relevant if your concern was to control long-run error rates, or (b) it violates the likelihood principle by considering outcomes other than the one observed.

]]>You wrote: “The sampling distribution is irrelevant to the likelihoodist (or holder of the likelihood principle)–once the data are in hand.”

And this is perfectly true, but it’s not on point, because what we’re really talking about is the *hypothesis space*, which is needed before anyone brings up sampling distributions or likelihood functions. Barnard, replying to Hacking, took at face value Hacking’s statement of the Law of Likelihood, which makes no mention of restricting the hypothesis space. But as you discuss, by the time Barnard was writing, even Hacking wasn’t defending that claim — by that time he was criticizing Edwards, who explicitly held that likelihoods are model-bound — that is, the Law of Likelihood applies in the context of a specific hypothesis space, which is chosen on non-data/non-likelihood grounds. This is what gives likelihoodists room to avoid absurd applications of the unrestricted Law of Likelihood.

You write: “The key difference is that data-dependent hypotheses alter the error probing capacities of tests and these are picked up by the sampling distribution.”

But it does not follow that a likelihoodist cannot avail herself of these arguments when choosing a hypothesis space or model. If we look at a modern likelihoodist like Royall, we can see that he’s perfectly comfortable calculating pre-data probabilities of error for experimental design purposes, so why not for hypothesis-space design purposes?

]]>If you say this has no bearing on your arguments, then I’m afraid I have no idea what you’re arguing.

]]>My continuation goes: suppose an error statistician wants to argue, in some particular case, that the Sure Thing hypothesis is, for whatever reason, not worth considering. (She may not want to make that argument, and that’s fine; but I’m considering the case in which she does.) Whatever argument she offers cannot be based on the observed data — as we’ve established, tests based on the data simply cannot rule out post-hoc Sure Things with severity. Therefore her argument, whatever it may be, will be available to a likelihoodist too, notwithstanding the fact that the data give the Sure Thing hypothesis the maximum possible likelihood.

This is why I say that the existence of Sure Things poses exactly as much, or as little, problem for a likelihoodist as for an error statistician.

I’m done.

]]>and by the way, I don’t want to rule out what you’re calling the post-hoc Sure Thing hypothesis. Hypotheses that pass poorly are not thereby “ruled out”. I may even know they are true.

If my answer is a hodge-podge, it’s because yours is. What are you talking about

]]>As for the null being well-warranted, we’d generally merely rule out discrepancies with severity. ]]>