Tour II It’s the Methods, Stupid
There is perhaps in current literature a tendency to speak of the Neyman–Pearson contributions as some static system, rather than as part of the historical process of development of thought on statistical theory which is and will always go on. (Pearson 1962, 276)
This goes for Fisherian contributions as well. Unlike museums, we won’ t remain static. The lesson from Tour I of this Excursion is that Fisherian and Neyman– Pearsonian tests may be seen as offering clusters of methods appropriate for different contexts within the large taxonomy of statistical inquiries. There is an overarching pattern:
Just as with the use of measuring instruments, applied to the specific case, we employ the performance features to make inferences about aspects of the particular thing that is measured, aspects that the measuring tool is appropriately capable of revealing. (Mayo and Cox 2006, p. 84)
This information is used to ascertain what claims have, and have not, passed severely, post-data. Any such proposed inferential use of error probabilities gives considerable fodder for criticism from various tribes of Fisherians,Neyman– Pearsonians, and Bayesians. We can hear them now:
…
How can we reply? To begin, we need to uncover how the charges originate in traditional philosophies long associated with error statistical tools. That’ s the focus of Tour II.
Only then do we have a shot at decoupling traditional philosophies from those tools in order to use them appropriately today. This is especially so when the traditional foundations stand on such wobbly grounds, grounds largely rejected by founders of the tools. There is a philosophical disagreement between Fisher and Neyman, but it differs importantly from the ones that you’re presented with and which are widely accepted and repeated in scholarly and popular treatises on significance tests. Neo-Fisherians and N-P theorists, keeping to their tribes, forfeit notions that would improve their methods (e.g., for Fisherians: explicit alternatives, with corresponding notions of sensitivity, and distinguishing statistical and substantive hypotheses; for N-P theorists, making error probabilities relevant for inference in the case at hand).
The spadework on this tour will be almost entirely conceptual: we won’t be arguing for or against any one view. We begin in Section 3.4 by unearthing the basis for some classic counterintuitive inferences thought to be licensed by either Fisherian or N-P tests. That many are humorous doesn’t mean disentangling their puzzles is straightforward; a medium to heavy shovel is recommended. We can switch to a light to medium shovel in Section 3.5: excavations of the evidential versus behavioral divide between Fisher and N-P turn out to be mostly built on sand. As David Cox observes, Fisher is often more performance-oriented in practice, but not in theory, while the reverse is true for Neyman and Pearson. At times, Neyman exaggerates the behavioristic conception just to accentuate how much Fisher’s tests need reining in. Likewise, Fisher can be spotted running away from his earlier behavioristic positions just to derogate the new N-P movement, whose popularity threatened to eclipse the statistics program that was, after all, his baby. Taking the polemics of Fisher and Neyman at face value, many are unaware how much they are based on personality and professional disputes. Hearing the actual voices of Fisher, Neyman, and Pearson (F and N-P), you don’ t have to accept the gospel of “what the founders really thought.” Still, there’ s an entrenched history and philosophy of F and N-P: A thick-skinned jacket is recommended. On our third stop (Section 3.6) we witness a bit of magic. The very concept of an error probability gets redefined and, hey presto!, a reconciliation between Jeff reys, Fisher, and Neyman is forged. Wear easily removed shoes and take a stiff walking stick. The Unificationist tribes tend to live near underground springs and lakeshore bounds; in the heady magic, visitors have been known to accidentally fall into a pool of quicksand.
3.4 Some Howlers and Chestnuts of Statistical Tests
The well-known definition of a statistician as someone whose aim in life is to be wrong in exactly 5 per cent of everything they do misses its target. (Sir David Cox 2006a, p. 197)
Showing that a method’s stipulations could countenance absurd or counterintuitive results is a perfectly legitimate mode of criticism. I reserve the term “howler” for common criticisms based on logical fallacies or conceptual misunderstandings. Other cases are better seen as chestnuts – puzzles that the founders of statistical tests never cleared up explicitly. Whether you choose to see my “howler” as a “chestnut” is up to you. Under each exhibit is the purported basis for the joke……
TO KEEP READING, SEE Mayo (2018, CUP): Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars.
Where are you in the journey? 
Error Statistics Philosophy 
There’s been quite a lot of discussion the past few days on twitter about Neyman vs Fisher, what Neyman meant in talking about conclusions, inference, decisions, etc. referring to some of the references in Excursion 3 Tour II. I’d like to bring some of them over here to the comments, but they won’t be complete.
Note the emphasis on clothing and tools here. Statistical explorations require the right shoes, jacket, shovel or stick. 3.4 is in a comedy club (medium to heavy shovel), 3.5 is in a stat museum gallery on F&N-P “inference vs behavior” (light shovel, thick-skinned jacket); 3.6 is in a jungle, where we witness some magic on defining error probability (easily removable shoes & stiff walking stick). Perhaps it’s owing to my background in fashion that this metaphor arises. But the main thing is not to take it too seriously.